I enjoy trying to improve security in the environments I manage as a sysadmin. One of the hardest parts of improving security is not making it a hindrance to users. (There is a fine line between good security and hindrance, but that’s a rant for another time).
Recently I had been asked to improve the SSH security in an environment. SSH keys weren’t really being used, so that was my first suggestion, but I needed to make key management as simple as possible without forgoing any security benefits; so I decided to take a proper look at SSH Certificate Authorities (CA).
I’ve known about SSH CA for a while, though never played around with it. Before taking it into a production environment I decided to become familiar on my own turf.
There are two types of SSH CA, Host and User.
Lets look at the Host side first. When you install OpenSSH (or other SSH servers) on a system a number of host keys are generated
/etc/ssh/ssh_host_dsa_key
/etc/ssh/ssh_host_dsa_key.pub
/etc/ssh/ssh_host_ecdsa_key
/etc/ssh/ssh_host_ecdsa_key.pub
/etc/ssh/ssh_host_ed25519_key
/etc/ssh/ssh_host_ed25519_key.pub
/etc/ssh/ssh_host_rsa_key
/etc/ssh/ssh_host_rsa_key.pubWhen you first SSH to a server you are presented with a message that looks something like this
The authenticity of host '10.1.2.3 (10.1.2.3)' can't be established.
ECDSA key fingerprint is SHA256:CwrcHjdd9349u38rj392fr9j389rj3298rj23.
Are you sure you want to continue connecting (yes/no/[fingerprint])?Now what should happen, if it isn’t your server, is the sysadmin would provide you with the fingerprint of the host key(s) so you can confirm it is in fact the correct server you are connecting to.
But we all know that doesn’t happen. Everyone types yes and moves on with their life.
When you accept the fingerprint SSH adds the host public key to your known_hosts file, usually located under ~/.ssh.
If the host key ever changed you would want to know. It could be the sysadmin has rotated the keys (unlikely), the server has been rebuilt or another has taken that IP (likely), or someone is attempted a Man in the Middle attack (scary!).
Thankfully SSH does warn you, have you ever seen this?
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
SHA256:CwrcHjdd9347u38fj392fr9f389rj289snjd.
Please contact your system administrator.
Add correct host key in /home/pyratebeard/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /home/pyratebeard/.ssh/known_hosts:1
ECDSA host key for 10.1.2.3 has changed and you have requested strict checking.
Host key verification failed.Phew, now we can contact our sysadmin and make sure we’re not being intercepted.
Or, as is more common, we remove the “offending” line and reconnect, selecting to accept the new key
ssh-keygen -f ~/.ssh/known_hosts -R 10.1.2.3An SSH CA can help improve security by removing this need to confirm the key fingerprints, because as we know few people do it anyway.
By signing all your server’s host keys with a CA key your users only need to trust one entity, the CA.
For SSH, a CA is actually just another SSH key pair. You then sign a hosts public key using the CA private key to produce a certificate. To generate a CA it’s as easy as creating a new key pair
ssh-keygen -t ed25519 -C hostca@pyratebeard.net -f hostca-keyNow I have hostca-key and hostca-key.pub. I prefer using ED25519 instead of RSA or ECDSA.
The public key is the entity we want to trust, so we add that to our known_hosts file, prefixed with @cert-authority and the domain your servers are in
cat >> EOF << ~/.ssh/known_hosts
> @cert-authority *.pyratebeard.net [hostca public key]
> EOFIn my example I am using the domain pyratebeard.net and a wildcard for the subdomain. This means if I ssh to server.pyratebeard.net then this CA will be used. You could use a wildcard (*) with no domain if you use the same CA for every system you need to access.
Next we want to take a copy of the host public key
# method 1
cat >> EOF << server-ssh_host_ed25519_key.pub
> [public key]
> EOF
# method 2
rsync server.pyratebeard.net:/etc/ssh/ssh_host_ed25519_key.pub server-ssh_host_ed25519_key.pub
# method 3
ssh-keyscan server.pyratebeard.net | awk '/ssh-ed25519/ {print $2" "$3}' | tee server-ssh_host_ed25519_key.pubNow we can sign the host public key using our CA private key
ssh-keygen -s hostca-key -h -I server@pyratebeard.net -n server.pyratebeard.net,server -V +52w server-ssh_host_ed25519_key.pub-s sign - tells ssh-keygen which key to sign with-h host cert - indicates we’re creating a host cert (not a user cert)-I unique identifier - an identifier used for logging-n principals - hostname(s) certificate is valid for, good practice to use short hostname and FQDN-V validity - length of time the certificate is valid for, in this example it is 52 weeksThis command will produce a certificate ([key name]-cert.pub). To confirm the new server key cert details incant
ssh-keygen -L -f server-ssh_host_ed25519_key-cert.pubOnce the certificate has been generated it needs to be put on the server it is for, usually in the /etc/ssh directory
# method 1 (on server)
cat >> EOF << /etc/ssh/ssh_host_ed25519_key-cert.pub
> [server key cert]
> EOF
# method 2
rsync server-ssh_host_ed25519_key-cert.pub server:/etc/ssh/ssh_host_ed25519_key-cert.pubSo that SSH knows to use a certificate add the following to /etc/ssh/sshd_config and reload the SSH service
HostCertificate /etc/ssh/ssh_host_ed25519_key-cert.pubIf every server in the environment is signed with the same CA then only one line is needed in known_hosts, the @cert-authority entry. New servers can be signed and users won’t have to accept any fingerprints because they will already be trusted. Systems can be rebuilt, or keys regenerated, and it won’t cause any impact to users.
If you’re an admin of a shared Linux system, or are able to manage users’ workstations, then drop the @cert-authority line into /etc/ssh/ssh_known_hosts for it to work system-wide without bothering the users.
Traditionally, to set up SSH key authentication as a user the public key is added to the ~/.ssh/authorized_keys file
ssh-copy-id -f -i ~/.ssh/pyratebeard-ssh_ed25519_key.pub pyratebeard@serverUsing ssh-copy-id usually requires password authentication, or if you have console access, manually adding the public key to the authorized_keys file.
Depending on how you manage your SSH keys you may have a number of authorised keys available on the remote system. If your key was ever compromised, lost, or needed changing for some other reason, you would need to add a new public key to the remote system. Maybe you have multiple remote systems, and maybe you haven’t got an automated way to push out new keys. Or maybe your current key isn’t working and you can’t use password authentication as you have correctly locked down your SSH config.
By using a User CA the remote system(s) only have to trust the CA, then any user key that is signed will be granted access. This is especially useful in a production environment if you have multiple users across a magnitude of systems. When a new user joins the team they get their SSH key signed by the CA and they can instantly gain access to all the remote systems they require.
A User CA, the same Host, is an SSH keypair
ssh-keygen -t ed25519 -C userca@pyratebeard.net -f userca-keyThen we can sign a user public SSH key (note we don’t use the -h option)
ssh-keygen -s userca-key -I pyratebeard@pyratebeard.net -n pyratebeard -V +52w pyratebeard-ssh_ed25519_key.pub* -s sign - tells `ssh-keygen` which key to sign with
* -I unique identifier - an identifier used for logging
* -n principal - username the certificate is valid for, comma-delimit multiple usernames
* -V validity - length of time the certificate is valid for, omit for never expiringThe above command will produce a certificate ([key-name]-cert.pub). To confirm the new key cert details incant
ssh-keygen -L -f pyratebeard-ssh_ed25519_key-cert.pubLet SSH know which certificate to use by adding the following to ~/.ssh/config
Host *.pyratebeard.net
CertificateFile /path/to/pyratebeard-ssh_host_ed25519_key-cert.pubJust like we included the @cert-authority line in known_hosts, the remote system(s) need to know the User CA to trust. Copy the User CA public key to the remote system(s)
# method 1 (on server)
cat >> EOF << /etc/ssh/userca-key.pub
> [userca public key]
> EOF
# method 2
rsync userca-key.pub server:/etc/ssh/userca-key.pubIn the remote system(s) /etc/ssh/sshd_config add the following and reload sshd
TrustedUserCAKeys /etc/ssh/userca-key.pubSetting up a User CA means that the authorized_keys file is no longer required for signed keys. If the user key changes, as long as it is signed by the trusted CA it will be granted access.
All the benefits of SSH CA look great. Less messing around with known_hosts and authorized_keys. Quicker access for new user SSH keys and quicker access to new remote systems. But when it comes to security it can’t stay like that forever.
If a Host CA key is compromised a malicious system could be disguised as a trusted system, users wouldn’t be warned against a fingerprint change.
In order to revoke a Host CA change @cert-authority to @revoke in the relevant known_hosts file.
When this happens SSH will warn that the system authenticity can’t be established, as we saw before when logging in for the very first time.
If this is in a production environment the sysadmin would have to re-sign all the remote system keys and then inform users to update their known_hosts file.
A bit of overhead, but remedied easily enough.
What if the User CA key is compromised? A bad actor could gain access to all the remote systems by simply signing their SSH key (make sure you use passwords when generating CA keys!).
To revoke a User CA add the path to the CA in /etc/ssh/revoked_keys on the remote system(s)
cat << EOF >> /etc/ssh/revoked_keys
> /etc/ssh/userca-key.pub
> EOFLet SSH know about the revocation file in /etc/ssh/sshd_config and reload sshd
RevokedKeys /etc/ssh/revoked_keysThis will revoke access to all user keys which have been signed by the compromised CA. This could be a lot of users, and could potentially cause great impact. Ideally you would have multiple CA certs configured, maybe group teams to a CA so the impact is reduced to a small number of users.
This is where SSH CA can fall down. I know from my own experience that not many environments use CA. There have been attempts to make things more dynamic and transparent, especially on the user side of things. There is BLESS, developed by Netflix, which has received much praise. It is based on using AWS Lambda functions, not always available to environments.
There is also sshrimp by Jeremy Stott, who gave an interesting talk (Zero Trust SSH) at LinuxConfAU in 2020. sshrimp also works using Lambda functions.
In my client’s environment I can’t use Lambda at the moment. Potentially I could get something like BLESS configured but that would be an uphill battle of approvals with various stakeholders, something I’m not ready for right now. I tried looking at getting the CAs set up first, at least as a proof of concept to showcase the client. All the users are running Windows, with MobaXTerm being the connection tool of choice. For a few days I fought with MobaXTerm, trying to get it to pick up the trusted Host CA and the signed User cert. I have since drawn a line under it until I can come up with a new plan.
The upside to all of this is that I have learned a lot about SSH Certificate Authorities, and I (finally) have SSH CA configured for my own systems.