Security through clarity: keeping secrets in a multi-server Amazon EC2 setup

For one of our clients we’re deploying an application to multiple application instances on Amazon EC2. All the applications need access to three things: credentials, certificates and services.

To connect to the database server and Amazon S3 the application needs access to usernames and passwords. Server certificates are used in Apache and client certificates are used to communicate with external API’s. Finally we have a custom daemon that generates TLS client certificates called Certificate Depot.

We want to keep these three things as safe as possible without making frequent and automated deployment difficult. Our solution is to keep all the secret stuff in one location and only allow one point of entry.

We use a small EC2 instance as a little vault. Let’s call it the “secure instance”. The secure instance only runs an SSH daemon and Certificate Depot. A firewall blocks all connections, except on the SSH port. All communication with the secure instance goes over SSH. All data on the instance is stored on encrypted EBS volumes.

When a brand new application instance is booted it needs access to credentials and certificates so it can configure itself. This is done by temporarily uploading an SSH key to the application instance. The key allows a non-privileged user to connect to the secure instance. Once the app instance has everything it needs from the secure instances the key is deleted.

We wanted to keep the Certificate Depot code base small so we didn’t implement any authentication or authorization. We solve this at the network level. Certificate Depot only listens to localhost. Application instances access the Depot through an SSH tunnel to the secure instance.

By keeping the architecture simple we achieve what I like to call Security Through Clarity. The secure instance key is only needed when booting up new application or secure instances so only a few people need access to it. The single point of entry makes the secure instance less vulnerable to attacks. Credentials and certificates only need to be kept on the secure instance and a secure backup so they’re less likely to be leaked by sysadmins or developers.

When the secure instance key, credentials, or certificates are lost replacing them is straightforward because their owners and location are well defined.