How can I repair a hosed SSH configuration?

Background

Sometimes the /root/.ssh/authorized_keys and/or /etc/ssh/ssh_config files are hosed/corrupted by a user or possibly by software. This locks authorized users out of the instance.

Answer

Because RightLink does not use SSH, you can repair the SSH configuration by running a RightScript.

Note: Only EC2 is supported at this time, but you could use these scripts to adapt for other environments. Also, because the authorized_keys file is restored with the EC2 instance's key only, you will need to reboot the server to re-setup managed login for users with server_login.

1. Import the RightScripts from the marketplace:

• Print sshd configuration
• Repair sshd configuration
   

2. Run the "Print sshd configuration" as an Any Script. The output should look similar to:

********************************************************************************
*RS>                RightScript: 'Print sshd configuration'                 ****
05:08:55: Directory listing of of /root/.ssh:
05:08:55: --
05:08:55: /root/.ssh:
total 12K
drwx------ 2 root root 4.0K Jun  6 04:53 .
drwx------ 6 root root 4.0K Jun  6 05:05 ..
-rw------- 1 root root  388 Jun  6 04:53 authorized_keys
05:08:55: --
--
Directory listing of /etc/ssh:
05:08:55: /etc/ssh:
total 168K
drwxr-xr-x  2 root root 4.0K Jun  6 04:55 .
drwxr-xr-x 96 root root 4.0K Jun  6 05:08 ..
-rw-r--r--  1 root root 123K Apr  2 11:48 moduli
-rw-r--r--  1 root root 1.7K Apr  2 11:48 ssh_config
-rw-------  1 root root  672 Jun  6 04:55 ssh_host_dsa_key
-rw-r--r--  1 root root  611 Jun  6 04:55 ssh_host_dsa_key.pub
-rw-------  1 root root  227 Jun  6 04:53 ssh_host_ecdsa_key
-rw-r--r--  1 root root  183 Jun  6 04:53 ssh_host_ecdsa_key.pub
-rw-------  1 root root 1.7K Jun  6 04:55 ssh_host_rsa_key
-rw-r--r--  1 root root  403 Jun  6 04:55 ssh_host_rsa_key.pub
-rw-r--r--  1 root root  302 Jan 10  2011 ssh_import_id
-rw-r--r--  1 root root 2.5K Apr 24 00:38 sshd_config
05:08:55: --
Contents of /etc/ssh/ssh_config:
--
05:08:55: # This is the ssh client system-wide configuration file.  See
# ssh_config(5) for more information.  This file provides defaults for
# users, and the values can be changed in per-user configuration files
# or on the command line.

# Configuration data is parsed as follows:
#  1. command line options
#  2. user-specific file
#  3. system-wide file
# Any configuration value is only changed the first time it is set.
# Thus, host-specific definitions should be at the beginning of the
# configuration file, and defaults at the end.

# Site-wide defaults for some commonly used options.  For a comprehensive
# list of available options, their meanings and defaults, please see the
# ssh_config(5) man page.

Host *
#   ForwardAgent no
#   ForwardX11 no
#   ForwardX11Trusted yes
#   RhostsRSAAuthentication no
#   RSAAuthentication yes
#   PasswordAuthentication yes
#   HostbasedAuthentication no
#   GSSAPIAuthentication no
#   GSSAPIDelegateCredentials no
#   GSSAPIKeyExchange no
#   GSSAPITrustDNS no
#   BatchMode no
#   CheckHostIP yes
#   AddressFamily any
#   ConnectTimeout 0
#   StrictHostKeyChecking ask
#   IdentityFile ~/.ssh/identity
#   IdentityFile ~/.ssh/id_rsa
#   IdentityFile ~/.ssh/id_dsa
#   Port 22
#   Protocol 2,1
#   Cipher 3des
#   Ciphers aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc
#   MACs hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160
#   EscapeChar ~
#   Tunnel no
#   TunnelDevice any:any
#   PermitLocalCommand no
#   VisualHostKey no
#   ProxyCommand ssh -q -W %h:%p gateway.example.com
    SendEnv LANG LC_*
    HashKnownHosts yes
    GSSAPIAuthentication yes
    GSSAPIDelegateCredentials no
05:08:55: --
--
Contents of /var/spool/cloud/meta-data/public-keys-0-openssh-key:
05:08:55: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCzNbV8eLt8krYY2xoC1vbEpbW8zx1s4aRyJ0dOgT1AIlgLccE3uxpy1ec0x8csMIIT1tiDa5qNfNDCfQ27KSZQD0in5hz0x71EYGs3ofsUKsrAzQ2C81KHVJc7oX2RgCOVqHLJrT9jx7aDfoNgsHbs9vX9Yc/A8MIGTyZSCNiI36QVB97qZYTanrajKhtNnevKhYumuSWBcbYwAPW89nKCkJ/Lt5vQY2jCENqvAqDLziQ8CBV0E0mj3UHGABeAn8bFUSxFZ2hXV9X5HCxnb1bIH9MeIhWpS4z1MmWaMfOCf1me8UI7BwBBRZmRoGeCr6+yGE8f1WbQAkkKXCz4H1bF oss-ap
05:08:55: --
Contents of /root/.ssh/authorized_keys:
--
05:08:55: csh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCzNbV8eLt8krYY2xoC1vbEpbW8zx1s4aRyJ0dOgT1AIlgLccE3uxpy1ec0x8csMIIT1tiDa5qNfNDCfQ27KSZQD0in5hz0x71EYGs3ofsUKsrAzQ2C81KHVJc7oX2RgCOVqHLJrT9jx7aDfoNgsHbs9vX9Yc/A8MIGTyZSCNiI36QVB97qZYTanrajKhtNnevKhYumuSWBcbYwAPW89nKCkJ/Lt5vQY2jCENqvAqDLziQ8CBV0E0mj3UHGABeAn8bFUSxFZ2hXV9X5HCxnb1bIH9MeIhWpS4z1MmWaMfOCf1me8UI7BwBBRZmRoGeCr6+yGE8f1WbQAkkKXCz4H1bF oss-ap
05:08:55: --
05:08:55: Script exit status: 0
05:08:55: Script duration: 0.416071
05:08:55: Chef Run complete in 0.43177 seconds
*RS> Duration: 5.18 seconds
*RS> completed: Print sshd configuration

Notice how the public key does not much (I intentionally put a typo in the key type, "csh-rsa" instead of "ssh-rsa".

3. Now, run the "Repair sshd configuration" as an Any Script. This will replace the key from the meta-data cache and overwrite /etc/ssh/ssh_config with a stock setup for PKI; example output:

********************************************************************************
*RS>                RightScript: 'Repair sshd configuration'                ****
05:16:35: Restoring public key for instance
05:16:35: `/var/spool/ec2/meta-data/public-keys-0-openssh-key' -> `/root/.ssh/authorized_keys'
05:16:35: Restoring a stock-standard ssh_config
05:16:35: `/var/cache/rightscale/right_scripts_content/rs_attach70199601467640/ssh_config' -> `/etc/ssh/ssh_config'
05:16:35: Restarting sshd.
05:16:35: sshd: unrecognized service
05:16:35: ssh stop/waiting
05:16:35: ssh start/running, process 20717
05:16:35: Done.
05:16:35: Script exit status: 0
05:16:35: Script duration: 0.40479
05:16:35: Chef Run complete in 0.419857 seconds
*RS> Duration: 5.21 seconds
*RS> completed: Repair sshd configuration

If either of these files were indeed the issue with logging in via SSH, you should now be able to login again (reboot the server if you need to update managed login).