A while back I wrote about some basic usage of SSH certificates as an authentication system. I only described the core, but the comments went into some further detail.
I thought it time to write a follow up post describing some of the more advanced features.
Quick recap
To handle cert based authentication you need a CA certificate. This is
created with the ssh-keygen
command.
e.g.
$ mkdir ssh-ca
$ cd ssh-ca
$ ssh-keygen -f server_ca
Server certificates are similarly signed with the same command.
e.g.
$ sudo ssh-keygen -s server_ca -I key_for_test1 -h -n test1.spuddy.org -V +52w /etc/ssh/ssh_host_rsa_key.pub
More details on this are available at the original post.
Checking the certificate details.
We can look at the certificate, again with ssh-keygen
$ ssh-keygen -L -f /etc/ssh/ssh_host_rsa_key-cert.pub
/etc/ssh/ssh_host_rsa_key-cert.pub:
Type: ssh-rsa-cert-v01@openssh.com host certificate
Public key: RSA-CERT SHA256:ggTP5Bg7d74kmK+g+g8KVNizhmEz7ilCmO41GPRRVag
Signing CA: RSA SHA256:9Uu2dQc6N4hgwv5Yz/UVU+fJfUTJzShQ0i4LdU7owEU
Key ID: "key_for_test1"
Serial: 0
Valid: from 2022-02-06T15:08:00 to 2023-02-05T15:09:50
Principals:
test1.spuddy.org
Critical Options: (none)
Extensions: (none)
We can see the Key ID
values matches the -I
parameter used on the
signing command, the principal name is the hostname, and the validity
period is 52 weeks.
User certificates
This is where we start to go deeper.
Let’s sign a user key the same way as we did before, and then look at it.
$ ssh-keygen -s ssh-ca/server_ca -I user_sweh -n sweh -V +52w .ssh/id_rsa.pub
Signed user key .ssh/id_rsa-cert.pub: id "user_sweh" serial 0 for sweh valid from 2022-02-06T15:37:00 to 2023-02-05T15:38:31
$ ssh-keygen -L -f .ssh/id_rsa-cert.pub
.ssh/id_rsa-cert.pub:
Type: ssh-rsa-cert-v01@openssh.com user certificate
Public key: RSA-CERT SHA256:W+cq9v38nF6h1Wlse45h0qizmybkkj0J4Yh820ihG7A
Signing CA: RSA SHA256:9Uu2dQc6N4hgwv5Yz/UVU+fJfUTJzShQ0i4LdU7owEU
Key ID: "user_sweh"
Serial: 0
Valid: from 2022-02-06T15:37:00 to 2023-02-05T15:38:31
Principals:
sweh
Critical Options: (none)
Extensions:
permit-X11-forwarding
permit-agent-forwarding
permit-port-forwarding
permit-pty
permit-user-rc
It’s interesting to note that “Principals” implies we can list more than
one principal (the -n
option). “Extensions” is also interesting.
Principals
Generating a certificate with multiple principals.
This is done by specifying each principal separated by a comma. In a general
corporate environment, with Active Directory, it might be worth generating
certificates with the sAMAccountName
and UserPrincipalName
values.
Typically that’ll be a short name and an email address. So, for example,
$ ssh-keygen -s ssh-ca/server_ca -I user_sweh -n sweh,sweh@sweharris.org -V +52w .ssh/id_rsa.pub
Signed user key .ssh/id_rsa-cert.pub: id "user_sweh" serial 0 for sweh,sweh@sweharris.org valid from 2022-02-06T15:46:00 to 2023-02-05T15:47:10
$ ssh-keygen -L -f .ssh/id_rsa-cert.pub
.ssh/id_rsa-cert.pub:
Type: ssh-rsa-cert-v01@openssh.com user certificate
Public key: RSA-CERT SHA256:W+cq9v38nF6h1Wlse45h0qizmybkkj0J4Yh820ihG7A
Signing CA: RSA SHA256:9Uu2dQc6N4hgwv5Yz/UVU+fJfUTJzShQ0i4LdU7owEU
Key ID: "user_sweh"
Serial: 0
Valid: from 2022-02-06T15:46:00 to 2023-02-05T15:47:10
Principals:
sweh
sweh@sweharris.org
Critical Options: (none)
Extensions:
permit-X11-forwarding
permit-agent-forwarding
permit-port-forwarding
permit-pty
permit-user-rc
Using principals
In the default case, sshd tries to match the principal name to the
username being logged into. This really works well most of the time;
if I want to login as sweh
and my certificate has sweh
as the
principal then it’ll let me in.
This is not so good if you’re trying to access a system, service, or
shared account. e.g. root
or (on AWS) ec2-user
, or oracle
or…
Side note: I don’t generally recommend logging in as these types of account directly. I prefer people to login as themselves and then use privilege escalation tools such as
sudo
to provide a stronger audit trail of who has done what. If 3 people have all logged in asroot
and the server gets rebooted, how do you know who did it?
Here is where we can use an option in sshd_config
that’s normally
disabled; AuthorizedPrincipalsFile
e.g.
AuthorizedPrincipalsFile /etc/ssh/auth_principals/%u
Here we can list the principals allow to access an account:
$ sudo cat /etc/ssh/auth_principals/ec2-user
testuser@sweharris.org
sweh@sweharris.org
$ ssh ec2-user@test1.spuddy.org id
uid=1000(ec2-user) gid=1000(ec2-user) groups=1000(ec2-user)
and looking at the logs…
Feb 6 16:09:23 test1 sshd[8160]: Accepted publickey for ec2-user from 10.0.0.158 port 34974 ssh2: RSA-CERT ID user_sweh (serial 0) CA RSA SHA256:9Uu2dQc6N4hgwv5Yz/UVU+fJfUTJzShQ0i4LdU7owEU
Roles as principals
This gets hard to manage at scale. If you have thousands of machines then
you may not want to list every possible sysadmin on every machine. Instead
you could create a role name, such as sysadmin
and your signing tool can
add this to the to user certificate. eg sweh,sweh@sweharris.org,sysadmin.
Now the AuthorizedPrincipalsFile
only needs to list sysadmin
and
anyone who has that listed will be able to have access.
It’s recommended that if you use role based principals then the validity
is reduced (eg 90 days) to help enforce that when someone transfers to
a new department then their roles will be removed within a reasonable
period. Using @revoked
and maintaining a CRL may also be useful to
ensure certificates can no longer be used.
More dynamic
A larger enterprise may wish to use AuthorizedPrincipalsCommand
instead
of a fixed file. This could do a dynamic lookup (e.g. in LDAP or an
API call) to generate a list of acceptable principals. This could allow
for real-time evaluation (“Who is allowed to login as ec2-user on host
test1?“). Such a process would handle leaver/transfer handling (“Stephen
is no longer a sysadmin; Fred left the company; John has become a DBA”)
without needing to revoke certificates or require re-signed certificates.
Extensions
Thus far we’ve just talked about who can login to a server as a specific user (“Stephen can login as ec2-user”). With extensions we can be a little more fine grained.
By default the following are permitted:
Extensions:
permit-X11-forwarding
permit-agent-forwarding
permit-port-forwarding
permit-pty
permit-user-rc
This can be changed on the ssh-keygen
line with the -O
option. e.g. to
remove all the “permit” options can we can use the clear
value
$ ssh-keygen -s ssh-ca/server_ca -O clear -I user_sweh -n sweh,sweh@sweharris.org -V +52w .ssh/id_rsa.pub
Signed user key .ssh/id_rsa-cert.pub: id "user_sweh" serial 0 for sweh,sweh@sweharris.org valid from 2022-02-06T16:44:00 to 2023-02-05T16:45:52
$ ssh-keygen -L -f .ssh/id_rsa-cert.pub
.ssh/id_rsa-cert.pub:
Type: ssh-rsa-cert-v01@openssh.com user certificate
Public key: RSA-CERT SHA256:W+cq9v38nF6h1Wlse45h0qizmybkkj0J4Yh820ihG7A
Signing CA: RSA SHA256:9Uu2dQc6N4hgwv5Yz/UVU+fJfUTJzShQ0i4LdU7owEU
Key ID: "user_sweh"
Serial: 0
Valid: from 2022-02-06T16:44:00 to 2023-02-05T16:45:52
Principals:
sweh
sweh@sweharris.org
Critical Options: (none)
Extensions: (none)
We can also specify things like force-command
(matching what we can put
into authorized_keys
).
$ ssh-keygen -s ssh-ca/server_ca -O clear -O force-command="/usr/bin/id" -I user_sweh -n sweh,sweh@sweharris.org -V +52w .ssh/id_rsa.pub
Signed user key .ssh/id_rsa-cert.pub: id "user_sweh" serial 0 for sweh,sweh@sweharris.org valid from 2022-02-06T16:47:00 to 2023-02-05T16:48:06
$ ssh-keygen -L -f .ssh/id_rsa-cert.pub
.ssh/id_rsa-cert.pub:
Type: ssh-rsa-cert-v01@openssh.com user certificate
Public key: RSA-CERT SHA256:W+cq9v38nF6h1Wlse45h0qizmybkkj0J4Yh820ihG7A
Signing CA: RSA SHA256:9Uu2dQc6N4hgwv5Yz/UVU+fJfUTJzShQ0i4LdU7owEU
Key ID: "user_sweh"
Serial: 0
Valid: from 2022-02-06T16:47:00 to 2023-02-05T16:48:06
Principals:
sweh
sweh@sweharris.org
Critical Options:
force-command /usr/bin/id
Extensions: (none)
Now if I try to use this certificate I can only run the one command
$ ssh ec2-user@test1.spuddy.org cat /etc/passwd
uid=1000(ec2-user) gid=1000(ec2-user) groups=1000(ec2-user)
These extensions can let us lock down certificates pretty much as
tightly as the older private keys (command
and from
restrictions in
authorized_keys
).
The force-command
and source-address
options are particularly useful
when performing app-to-app type communication. These keys generally don’t
have a passphrase on them (since it’s automated) so have the ability
to let one server scp
a file to another without granting full shell
access and enforcing the server the connection came from is important.
In a firewalled environment it’s worth thinking about always using
-O clear
to start with to prevent tunneling through firewalls. I would also add-O no-user-rc
since~/.ssh/rc
can be used to bypass restrictions.
Conclusion
SSH certificates have a lot of power to them.
However they do require a good admin tool to manage them; to do the signing, to add roles (if used), to add restrictions, remove permissions, revoke keys and so on. I try not to recommend tools in this blog (so don’t ask!) but if you search the internet you’ll find a number of them, both commercial and open-source.
There are limitations with certificates (as with keys). In particular,
we can’t enforce passphrase complexity (or indeed any passphrase at all!)
and we can’t stop users from sharing keys. Just because a key has an
identity user_sweh
doesn’t mean it was me who used it; I could have
shared it with Fred and John! Some of this is education (we teach people
not to share passwords; we need to start teaching them not to share keys).
Some of this can also be mitigated by shorter durations for human owned
keys (if I need to get it re-signed every 90 days then it’s less likely
that a shared cert will be usable).
But the ability to centrally control the options on a key as part
of the signature (as opposed to relying on the correct settings in
authorized_keys
) is big, and allows for better auditing.
Special Thanks
I want to thank everyone who commented or emailed on the original post. I learned from them and they got me digging deeper into certificate capabilities. Thank you!