When I worked in operations, I used to dread when the topic of managing SSL and TLS certificates came up. Creating certificates required re-reading large chunks of the OpenSSL documentation, getting them signed correctly was futile, at best, and remembering to rotate them at expiry usually fell through the cracks. For most enterprises certificate management is an arcane art, steeped in mystery and manual processes. Each new web service forces us to relive this nightmare.
As we refactor apps into microservices and move them into containers, the number of HTTPS endpoints tends to grow like weeds. Also, the number of development, test, QA, penetration testing, and compliance environments is exploding, and the number of iterations through development, testing, and deployment in each of those environments is skyrocketing.
In this blog series, I’ll highlight an easy path forward for operations teams that need to up their certificate-management game for Kubernetes. This first entry talks about the tools we can use for automated cert management. The second blog in the series, to be published later, will show the use of the tools to deploy the Harbor container registry with an SSL endpoint. The final blog post in the series closes with a summary of two alternatives that may work better with your existing certificate workflow and be valuable to improving developer velocity and production robustness.
Setting the level
A digital certificate (or cert) is a description of a public key, as part of a public-private key pair, a description of the owner of that key, and what the key can be used for. There are many kinds of certs, but I’ll limit the focus here to just TLS certs for HTTPS traffic and ignore things like code-signing certs, email certs, and so forth.
In the enterprise, certs are typically generated offline and signed by a Certificate Authority (CA) before they are shipped to where they will be used. This manual, and often slow, process tends to lead internal facing IT and developers to take shortcuts by using self-generated and self-signed certs. Self-signed certs are fine for some pre-production use cases, but they should never be used for production.
As we know from hundreds of long days and late nights troubleshooting failed deployments, doing things differently in pre-production and production causes problems. If you are going to need CA signed certs in production, you must issue them the same way for development and all other environments, or you will suffer the consequences.
Tools that make it easier
Docker makes containers simple, Kubernetes makes orchestrating them manageable, a tool called Let’s Encrypt makes creation and management of TLS certs for HTTPS easy to automate, and cert-manager makes using Let’s Encrypt for Kubernetes ingresses automatic. We’ll touch on Let’s Encrypt and cert-manager.
Let’s Encrypt
Let’s Encrypt is a CA that offers free signed certs via an API that allows automating the certificate-signing process. They have made managing HTTPS TLS certificates so easy that there is no excuse for using self-signed certs anymore. Let’s Encrypt launched in 2016, and it has many ways to provide certs for services in an automated fashion. If you are not using Kubernetes, you should still look into using Let’s Encrypt for automating certificate management.
The magic that makes Let’s Encrypt so easy is the Automatic Certificate Management Environment (ACME) API. ACME is described in this working draft.
It’s free and automated. Why wouldn’t you use it? First, they only provide certs for HTTPS traffic, but we’re talking about web services here, so ignore that.
Second, there are older browsers and some embedded devices that don’t support the Let’s Encrypt CA. They have a signing cert bundle that was created prior to the creation of Let’s Encrypt. There is a compatibility list that you should check if you have special device or browser requirements.
Finally, you may not want to use Let’s Encrypt if you must have the business’ identity confirmed by your certs. Let’s Encrypt only supports domain validation, which proves that the certificate owner also has control of the DNS for the domain using the certificate. There are two other levels of verification available for certificate signing, and if you have need for a high level of verification as provided in organization or extended-validation certs, you may not find Let’s Encrypt suitable.
cert-manager
cert-manager is an add-on for Kubernetes that enables certificate generation, signing, and renewal. The project, which has been around since 2017, automates the process of using the ACME API from Let’s Encrypt. In addition, cert-manager supports using other methods to automate certificate management, such as Venafi, HashiCorp Vault, and any other arbitrary CA, as long as you have the signing key-pair from a CA. The cert-manager documentation has a full list of supported issuer types.
cert-manager defines a set of custom resource definitions (CRDs) in Kubernetes that describe certificates and certificate issuers. You can create issuers per namespace or per cluster that handle talking to supported CAs mentioned above. These CRDs work with the Kubernetes ingress API object so that when an ingress specifies a need to use TLS, a new certificate API object is created and signed using the issuer. The final signed certificate and key are deposited into a secret for use by the ingress controller when terminating HTTPS connections.
The following diagram shows the relationships among API objects in Kubernetes when cert-manager is used to generate certificates:
Getting a Certificate from Let’s Encrypt
- Deploy an ingress controller like Contour.
- Get the ingress controller’s external address.
- Create a DNS CNAME record (service.example.com) that references the ingress controller’s address.
- Create the Kubernetes deployment and service for which you want to create a certificate.
- Create an Issuer or ClusterIssuer API object that specifies Let’s Encrypt or another issuer type.
- Create an ingress that requires TLS support and that a) lists the host names for which the certificate should be valid; b) specifies an annotation that signals this ingress should trigger cert-manager; and c) specifies an annotation that indicates which issuer object to use for signing the certificate
- The cert-manager certificate controller creates a certificate API object when the ingress annotation is detected.
- At this point, cert-manager does a few things behind the scenes that are different for each issuer type, but for Let’s Encrypt those are the following
- Create an order API object and contact the CA to request a signature
- Create a challenge API object and a host verification URL at a specific route in the ingress so that Let’s Encrypt can validate control of the DNS domain referenced in this ingress.
- Once the challenge has succeeded, send the certificate request to the ACME API.
- Receive a signed certificate and deposit it into the secret specified in the ingress definition.
- Remove the order object.
After this point, the cert-manager keeps an eye on the certificate expiry date and will create a new certificate as needed to start the cycle over again.
Summary
We’ve covered the tools used to automatically create and manage TLS certificates for Kubernetes HTTPS services. You should now understand the tools involved and the general process that cert-manager uses to ensure that the certificate lifecycle is managed.
Check back for the following two entries in this series to learn how to use those tools to secure a Harbor deployment and to learn about another strategy that may be more adaptable to your existing certificate management workflow.
