Containers and other elastic compute structures are good ways of deploying applications, especially if you follow some of the guidelines I’ve made in other posts on this topic. However they don’t exist in a vacuum. They may need to call out to “external” services. For example, an Oracle database, or Amazon S3, or another API service provided by other containers. In order to do this it needs to authenticate to that service. That typically means requiring some form of authenticator (e.g a username and password, or a userkey and secret key).
Now in a traditional non-elastic environment this can be done at application deploy time; an operate person can login to the server and supply the necessary credentials. If you’re more advanced then you might use a password vault (e.g. CyberArk). These work because the operating environment is pretty static. In the case of CyberArk they require an agent to be installed on the client machine, registered, and authorised to the vault. Clearly neither “ops supplying credentials” nor “register agent” type solutions are very elastic friendly! So we need a different solution.
If you look around there are various products that try to solve this; a common one is Hashicorp Vault. This has some clever routines built into it, and was clearly built with an API-first model in mind. One clever feature is that it can create time-limited credentials, so if your app needs to login to MySQL then it can ask HashiVault for a username/password, the vault will create them in the database at that point in time, then delete them later. Clever!
Unfortunately it doesn’t address the question of how your application authenticates to the vault in the first place. We haven’t really solved the problem with this technology; we’ve just rewritten it.
Another solution I’ve heard a few places attempt is kerberos based authentication; however this doesn’t really solve the problem, either. How does your application get the initial TGT to be able to get the service keys it needs? If you use a keytab file, how does that get into the app? No, including that in your app source tree is not a suitable solution :-)
So what can we use as an identity anchor? When your app spins up and migrates around containers, when the network itself may be NAT’ed, when the contents of containers may vary as applications are upgraded… what can we use to anchor the identity so your application can authenticate to other services?
Viewpoints
I’ve found three ways of looking at this problem.
Seed credential
This the most obvious version; This may be your keytab file; if this can be provided securely to the app then the application can now authenticate to the Kerberos infrastructure and carry on. It may be an authentication token to a vault (eg HashiVault) from where it can then get the necessary credentials necessary for further services. How this is done will be technology specific. With something like docker you may be able to provide a persistent backend data store with that credential and this can be shared between the containers running the application. Not all container engines have this persistency layer so other solutions may be required.
Provided by the runtime environment
This is the solution provided by Cloud
Foundry. With this information can be
stored inside the “cloud controller” and passed onto the application
in the VCAP_SERVICES variable. It’s all handled by the orchestration
engine. Unfortunately the CF solution stores all this data in plain text
and passes it in environment variables. The concept seems reasonable
but the implementation is weak. Of course the values provided here may
be the actual credentials needed, or a seed credential for vault access.
Have a trusted identity assertion
Can we identify the app based on the execution environment? Perhaps the
orchestration layer could generate a “signed identity” (perhaps a
client side SSL cert) and supply that to the app when it is instantiated.
This could be used as a seed credential. Or maybe we can run a daemon
process in the parent OS that the client can call out to; we can make
use of the orchestration knowledge (e.g. the state maintained by docker,
or a Cloud Foundry DEA configuration file) to identify the calling app;
effectively this daemon would assert identity on behalf of the app.
Putting it together
These three viewpoints can overlap; for example the identity assertion
tool could talk to a backend datastore that can provide encryption keys
(eg an RSA key). The public key could be exposed to the world and so
the contents that cloud foundry normally stores in VCAP_SERVICES can
be encrypted; the identity assertion process would ensure that only
application A could get RSA keys for itself and not for any other app.
The credential then decrypted could provide access to a vault.
Of course there are implementation challenges in all these options; if you create an encryption store then how do you ensure that the keys remain secret, or have the ability to rotate keys? If you provide a persistent backend store then how do you ensure the contents won’t be exposed elsewhere? If you use a CF type solution then how do you solve the plain-text problem?
At the end of the day your orchestration layer becomes a critical component in allowing an application to assert identity. The solution you build needs to be resilient, reliable and strongly protected. Of course this should apply to the whole orchestration layer, anyway!