Clients commonly ask us how to use our PKI and digital certificate operations management platform, CMS, to manage application certificates in a DevOps environment. They are “containerizing” their applications with Docker and want to manage certificates within Docker containers.
A bit of background on the question…
With DevOps (or the merging of software development and IT operations), the concept is to treat your infrastructure similarly to the way you [should] treat your application code – via formal specifications and automated build processes.
Prior to a DevOps world, software developers would commonly create applications without considering the details of the actual environment and infrastructure that the software would need to run in. Even when the target environment was considered, incorrect assumptions were frequently made or poorly communicated to the IT operations folks. When it was time to put the application into production, IT operations folks would have difficulty making applications run correctly. With DevOps, creation of a formal definition of the required infrastructure is part the software development process and automated tools are used to deploy the software within this infrastructure. The formal specification of the infrastructure is referred to as “Infrastructure as Code” or IaC.
The benefit with IaC is that the definition of infrastructure can be tracked in source control along with application source code, which helps keep the development side of the world in sync with the infrastructure side. With IaC, the deployment of your infrastructure becomes part of the application build process. This allows consistency in your deployment process between various deployment environments (test, QA, staging, production, etc.) and allows continuous integration to include infrastructure.
Docker is a common tool as part of a DevOps scenario. Docker allows you take a piece of software and place it into a “container” which abstracts out the infrastructure and presents a sandboxed operating system to the application. Orchestration engines such as Kubernetes or Docker Swarm leverage IaC to perform automated deployment and scaling of containers. Public repositories such as Docker Hub make it easy to share containers (and the related software) between applications and organizations.
This is all great, but adding certificates to the mix creates additional challenges:
- Certificate renewal happens on a different cadence than application updates. You may have monthly or quarterly releases of your software, but certificates need renewal annually. When application development ceases, certificate renewal still must occur for as long as the application is being used.
- Certificates are managed by different parties. Certificates may be purchased from public certificate authorities or may be issued from an internal private Public Key Infrastructure (PKI); in either case, the people responsible for acquisition or issuance are typically not the developers and may not be the IT operations folks.
While from a development point of view you can create a container that holds your web content or your web application, and your continuous integration test and build process can update and redeploy these containers with each software revision, you still need to consider how to update and maintain the certificates that your system presents to the outside world. This is the crux of the question.
So, what’s the answer…
As with everything in our field, the answer is generally, “it depends.” There are several models for managing application certificates with CMS and Docker. The good news is that in most cases the model (and answer) is simple.
If you do not want to write any code, the easiest way is to deploy our certificate orchestration agent container alongside your other application containers. Our agent then manages the certificates used by your application via shared storage natively available within Docker. Assuming your application can obtain its certificate from the file system, this does not require any changes to your application or containers and no coding or scripting is required. It is a completely out of the box solution.
The advantage of this model is that each instance of the container uses the same shared storage location which results in all of your containers using the same certificate. As your orchestration process scales up the number of nodes, you do not have to worry about keeping each web server behind your load balancer having the same certificate. When it is time to renew or change the certificate, all of the containers have access to the new certificate automatically.
Note that with this model, your certificates are held in shared storage, this means that the administrators of your shared storage have access to your certificate’s private key. Normally the administrators of your shared storage would also be Docker administrators who already have access to the containers (which also hold the private keys) so this is not an additional risk. If however, your administrative model separates duties between Docker administrators and the administrators of the shared storage, this may be an issue. In this case, Docker secrets may be used instead of shared storage. With Docker secrets, the private keys are encrypted while at rest and in transit and your storage administrators would not have access to them.
Using Docker secrets involves using a small amount of scripting within your orchestration system to retrieve the certificate from CMS and store it as a Docker secret.
If your application is not able to retrieve its certificates from the file system or uses some non-standard certificate storage format, a container startup script can be used to retrieve the necessary certificates from CMS and inject them into your application or store them in your proprietary format each time the container is deployed.
Finally, if you are okay with coupling the certificate renewal and application release processes together, or if you need to have containers that are deployed into environments that do not have access to CMS, you can use CMS to inject certificates during the container build process.
So in summary, when managing application certificates in a Docker environment with CMS, there are a variety of ways to accomplish the task, depending on the requirements of your application and its runtime environment. Most customers will be able to use a simple container to manage their Docker certificates.