Using SSH certificates - revisited

A while back I wrote about some basic usage of SSH certificates as an authentication system. I only described the core, but the comments went into some further detail.

I thought it time to write a follow up post describing some of the more advanced features.

Quick recap

To handle cert based authentication you need a CA certificate. This is created with the ssh-keygen command.

e.g.

$ mkdir ssh-ca
$ cd ssh-ca
$ ssh-keygen -f server_ca

Server certificates are similarly signed with the same command.

e.g.

$ sudo ssh-keygen -s server_ca -I key_for_test1 -h -n test1.spuddy.org -V +52w /etc/ssh/ssh_host_rsa_key.pub

More details on this are available at the original post.

Checking the certificate details.

We can look at the certificate, again with ssh-keygen

$ ssh-keygen -L -f /etc/ssh/ssh_host_rsa_key-cert.pub
/etc/ssh/ssh_host_rsa_key-cert.pub:
        Type: ssh-rsa-cert-v01@openssh.com host certificate
        Public key: RSA-CERT SHA256:ggTP5Bg7d74kmK+g+g8KVNizhmEz7ilCmO41GPRRVag
        Signing CA: RSA SHA256:9Uu2dQc6N4hgwv5Yz/UVU+fJfUTJzShQ0i4LdU7owEU
        Key ID: "key_for_test1"
        Serial: 0
        Valid: from 2022-02-06T15:08:00 to 2023-02-05T15:09:50
        Principals: 
                test1.spuddy.org
        Critical Options: (none)
        Extensions: (none)

We can see the Key ID values matches the -I parameter used on the signing command, the principal name is the hostname, and the validity period is 52 weeks.

User certificates

This is where we start to go deeper.

Let’s sign a user key the same way as we did before, and then look at it.

$ ssh-keygen -s ssh-ca/server_ca -I user_sweh -n sweh -V +52w .ssh/id_rsa.pub
Signed user key .ssh/id_rsa-cert.pub: id "user_sweh" serial 0 for sweh valid from 2022-02-06T15:37:00 to 2023-02-05T15:38:31

$ ssh-keygen -L -f .ssh/id_rsa-cert.pub
.ssh/id_rsa-cert.pub:
        Type: ssh-rsa-cert-v01@openssh.com user certificate
        Public key: RSA-CERT SHA256:W+cq9v38nF6h1Wlse45h0qizmybkkj0J4Yh820ihG7A
        Signing CA: RSA SHA256:9Uu2dQc6N4hgwv5Yz/UVU+fJfUTJzShQ0i4LdU7owEU
        Key ID: "user_sweh"
        Serial: 0
        Valid: from 2022-02-06T15:37:00 to 2023-02-05T15:38:31
        Principals: 
                sweh
        Critical Options: (none)
        Extensions: 
                permit-X11-forwarding
                permit-agent-forwarding
                permit-port-forwarding
                permit-pty
                permit-user-rc

It’s interesting to note that “Principals” implies we can list more than one principal (the -n option). “Extensions” is also interesting.

Principals

Generating a certificate with multiple principals.

This is done by specifying each principal separated by a comma. In a general corporate environment, with Active Directory, it might be worth generating certificates with the sAMAccountName and UserPrincipalName values. Typically that’ll be a short name and an email address. So, for example,

$ ssh-keygen -s ssh-ca/server_ca -I user_sweh -n sweh,sweh@sweharris.org -V +52w .ssh/id_rsa.pub
Signed user key .ssh/id_rsa-cert.pub: id "user_sweh" serial 0 for sweh,sweh@sweharris.org valid from 2022-02-06T15:46:00 to 2023-02-05T15:47:10

$ ssh-keygen -L -f .ssh/id_rsa-cert.pub
.ssh/id_rsa-cert.pub:
        Type: ssh-rsa-cert-v01@openssh.com user certificate
        Public key: RSA-CERT SHA256:W+cq9v38nF6h1Wlse45h0qizmybkkj0J4Yh820ihG7A
        Signing CA: RSA SHA256:9Uu2dQc6N4hgwv5Yz/UVU+fJfUTJzShQ0i4LdU7owEU
        Key ID: "user_sweh"
        Serial: 0
        Valid: from 2022-02-06T15:46:00 to 2023-02-05T15:47:10
        Principals: 
                sweh
                sweh@sweharris.org
        Critical Options: (none)
        Extensions: 
                permit-X11-forwarding
                permit-agent-forwarding
                permit-port-forwarding
                permit-pty
                permit-user-rc

Using principals

In the default case, sshd tries to match the principal name to the username being logged into. This really works well most of the time; if I want to login as sweh and my certificate has sweh as the principal then it’ll let me in.

This is not so good if you’re trying to access a system, service, or shared account. e.g. root or (on AWS) ec2-user, or oracle or…

Side note: I don’t generally recommend logging in as these types of account directly. I prefer people to login as themselves and then use privilege escalation tools such as sudo to provide a stronger audit trail of who has done what. If 3 people have all logged in as root and the server gets rebooted, how do you know who did it?

Here is where we can use an option in sshd_config that’s normally disabled; AuthorizedPrincipalsFile

e.g.

AuthorizedPrincipalsFile /etc/ssh/auth_principals/%u

Here we can list the principals allow to access an account:

$ sudo cat /etc/ssh/auth_principals/ec2-user
testuser@sweharris.org
sweh@sweharris.org
$ ssh ec2-user@test1.spuddy.org id
uid=1000(ec2-user) gid=1000(ec2-user) groups=1000(ec2-user)

and looking at the logs…

Feb  6 16:09:23 test1 sshd[8160]: Accepted publickey for ec2-user from 10.0.0.158 port 34974 ssh2: RSA-CERT ID user_sweh (serial 0) CA RSA SHA256:9Uu2dQc6N4hgwv5Yz/UVU+fJfUTJzShQ0i4LdU7owEU

Roles as principals

This gets hard to manage at scale. If you have thousands of machines then you may not want to list every possible sysadmin on every machine. Instead you could create a role name, such as sysadmin and your signing tool can add this to the to user certificate. eg sweh,sweh@sweharris.org,sysadmin. Now the AuthorizedPrincipalsFile only needs to list sysadmin and anyone who has that listed will be able to have access.

It’s recommended that if you use role based principals then the validity is reduced (eg 90 days) to help enforce that when someone transfers to a new department then their roles will be removed within a reasonable period. Using @revoked and maintaining a CRL may also be useful to ensure certificates can no longer be used.

More dynamic

A larger enterprise may wish to use AuthorizedPrincipalsCommand instead of a fixed file. This could do a dynamic lookup (e.g. in LDAP or an API call) to generate a list of acceptable principals. This could allow for real-time evaluation (“Who is allowed to login as ec2-user on host test1?“). Such a process would handle leaver/transfer handling (“Stephen is no longer a sysadmin; Fred left the company; John has become a DBA”) without needing to revoke certificates or require re-signed certificates.

Extensions

Thus far we’ve just talked about who can login to a server as a specific user (“Stephen can login as ec2-user”). With extensions we can be a little more fine grained.

By default the following are permitted:

        Extensions: 
                permit-X11-forwarding
                permit-agent-forwarding
                permit-port-forwarding
                permit-pty
                permit-user-rc

This can be changed on the ssh-keygen line with the -O option. e.g. to remove all the “permit” options can we can use the clear value

$ ssh-keygen -s ssh-ca/server_ca -O clear -I user_sweh -n sweh,sweh@sweharris.org -V +52w .ssh/id_rsa.pub
Signed user key .ssh/id_rsa-cert.pub: id "user_sweh" serial 0 for sweh,sweh@sweharris.org valid from 2022-02-06T16:44:00 to 2023-02-05T16:45:52
$ ssh-keygen -L -f .ssh/id_rsa-cert.pub
.ssh/id_rsa-cert.pub:
        Type: ssh-rsa-cert-v01@openssh.com user certificate
        Public key: RSA-CERT SHA256:W+cq9v38nF6h1Wlse45h0qizmybkkj0J4Yh820ihG7A
        Signing CA: RSA SHA256:9Uu2dQc6N4hgwv5Yz/UVU+fJfUTJzShQ0i4LdU7owEU
        Key ID: "user_sweh"
        Serial: 0
        Valid: from 2022-02-06T16:44:00 to 2023-02-05T16:45:52
        Principals: 
                sweh
                sweh@sweharris.org
        Critical Options: (none)
        Extensions: (none)

We can also specify things like force-command (matching what we can put into authorized_keys).

$ ssh-keygen -s ssh-ca/server_ca -O clear -O force-command="/usr/bin/id" -I user_sweh -n sweh,sweh@sweharris.org -V +52w .ssh/id_rsa.pub
Signed user key .ssh/id_rsa-cert.pub: id "user_sweh" serial 0 for sweh,sweh@sweharris.org valid from 2022-02-06T16:47:00 to 2023-02-05T16:48:06
$ ssh-keygen -L -f .ssh/id_rsa-cert.pub                                       
.ssh/id_rsa-cert.pub:
        Type: ssh-rsa-cert-v01@openssh.com user certificate
        Public key: RSA-CERT SHA256:W+cq9v38nF6h1Wlse45h0qizmybkkj0J4Yh820ihG7A
        Signing CA: RSA SHA256:9Uu2dQc6N4hgwv5Yz/UVU+fJfUTJzShQ0i4LdU7owEU
        Key ID: "user_sweh"
        Serial: 0
        Valid: from 2022-02-06T16:47:00 to 2023-02-05T16:48:06
        Principals: 
                sweh
                sweh@sweharris.org
        Critical Options: 
                force-command /usr/bin/id
        Extensions: (none)

Now if I try to use this certificate I can only run the one command

$ ssh ec2-user@test1.spuddy.org cat /etc/passwd
uid=1000(ec2-user) gid=1000(ec2-user) groups=1000(ec2-user)

These extensions can let us lock down certificates pretty much as tightly as the older private keys (command and from restrictions in authorized_keys).

The force-command and source-address options are particularly useful when performing app-to-app type communication. These keys generally don’t have a passphrase on them (since it’s automated) so have the ability to let one server scp a file to another without granting full shell access and enforcing the server the connection came from is important.

In a firewalled environment it’s worth thinking about always using -O clear to start with to prevent tunneling through firewalls. I would also add -O no-user-rc since ~/.ssh/rc can be used to bypass restrictions.

Conclusion

SSH certificates have a lot of power to them.

However they do require a good admin tool to manage them; to do the signing, to add roles (if used), to add restrictions, remove permissions, revoke keys and so on. I try not to recommend tools in this blog (so don’t ask!) but if you search the internet you’ll find a number of them, both commercial and open-source.

There are limitations with certificates (as with keys). In particular, we can’t enforce passphrase complexity (or indeed any passphrase at all!) and we can’t stop users from sharing keys. Just because a key has an identity user_sweh doesn’t mean it was me who used it; I could have shared it with Fred and John! Some of this is education (we teach people not to share passwords; we need to start teaching them not to share keys). Some of this can also be mitigated by shorter durations for human owned keys (if I need to get it re-signed every 90 days then it’s less likely that a shared cert will be usable).

But the ability to centrally control the options on a key as part of the signature (as opposed to relying on the correct settings in authorized_keys) is big, and allows for better auditing.

Special Thanks

I want to thank everyone who commented or emailed on the original post. I learned from them and they got me digging deeper into certificate capabilities. Thank you!