TLS and PKI in Kubernetes: a Certificate Is Just a Signed Claim

I ended Topic 6 by saying the whole cluster trusts one thing: the TLS mesh between its components. That felt like a cop-out to leave hanging, so this is me actually opening it up. Every piece of Kubernetes, the API server, etcd, the kubelet on each node, the admin running kubectl, proves who it is with a certificate. If you can't read those certificates, you can't really secure the cluster. So let's learn to read them.

Fair warning: certificates are the topic people quietly nod along to and never quite get. I was one of them. Here's the version I wish someone had handed me.

Keys first: why two are better than one

Encryption turns readable data into gibberish that's useless without a key. The simplest form, symmetric encryption, uses the same key to lock and unlock. That's fast, but it has an obvious problem: to talk to a server you'd have to send it the key first, and anyone sniffing the wire grabs it on the way past. Now they can read everything.

Asymmetric encryption fixes that with a pair of keys instead of one. A private key you never share, and a public key you can hand to the world. Anything locked with the public key can only be opened by the matching private key. So the public one being public doesn't matter, it's a padlock anyone can snap shut but only you hold the key to open.

The everyday example is SSH. You generate a pair, keep the private half, and drop the public half on any server you want in to:

ssh-keygen
# creates id_rsa (private) and id_rsa.pub (public)

cat ~/.ssh/authorized_keys   # public keys the server trusts
ssh -i id_rsa user1@server1

The server never sees your private key. It just checks that whoever's connecting can prove they hold the private half of a public key it already trusts. Same trick underpins HTTPS, and it's the trick the entire cluster runs on.

How HTTPS actually shakes hands

Here's the part that took me a while to picture. Symmetric encryption is fast, so we'd love to use it for the actual traffic. Asymmetric is the safe way to agree on that symmetric key without leaking it. So a TLS session uses both: asymmetric to do the handshake, symmetric for everything after.

You can make the keypair for a web server with OpenSSL in two lines:

openssl genrsa -out my-bank.key 1024
openssl rsa -in my-bank.key -pubout > my-bank.pem

But a bare public key isn't enough on its own, because nothing stops an attacker handing you their public key and pretending to be your bank. That's the gap a certificate fills.

Certificates and the authorities that vouch for them

A certificate is a public key plus a signed claim about who it belongs to. Crack one open and you'll see the subject (who it's for, as a Common Name), the issuer (who vouched for it), how long it's valid, and any Subject Alternative Names, the extra domains or IPs it covers.

Subject: CN=my-bank.com
Issuer: CN=kubernetes
Validity
    Not After : Feb  9 13:41:28 2020 GMT
X509v3 Subject Alternative Name:
    DNS:mybank.com, DNS:i-bank.com, DNS:we-bank.com

Anyone can generate a certificate, including a fraudster, so the signature is what matters. A Certificate Authority (a DigiCert, a GlobalSign, or a private CA you run yourself) checks your identity and signs your certificate with its own key. Browsers ship with the public keys of the well-known CAs baked in, so they can verify that signature and decide to trust you. The dance looks like this:

openssl req -new -key my-bank.key -out my-bank.csr \
  -subj "/C=US/ST=CA/O=MyOrg, Inc./CN=my-bank.com"
# send my-bank.csr to a CA -> they verify -> they sign -> you install the signed cert

You create a Certificate Signing Request (CSR) from your private key, the CA signs it, and you install what comes back. Public CAs cover public websites; a private CA does the same job for internal stuff nobody outside should trust. That whole system, keys, certificates, CAs and the process around them, is what people mean by PKI (public key infrastructure).

Every component in a cluster carries one

Now the payoff. A Kubernetes cluster is a pile of services all talking over TLS, which means two kinds of certificate. Servers (the API server, etcd, the kubelets) present a server certificate so clients know they're real. Clients (you via kubectl, the scheduler, the controller manager, kube-proxy) present a client certificate to prove who they are. And one CA signs the lot, which is the knot that ties it all together.

One thing that confused me at first: a server can also be a client. The API server serves HTTPS to everyone, but when it talks to etcd it's the one making the call, so it needs a client certificate for that too. Same box, two hats.

Creating the certificates with OpenSSL

You rarely do this by hand on a real cluster (kubeadm handles it), but doing it once is the fastest way to understand what kubeadm is quietly doing for you. First the CA itself: a key, a CSR with the common name KUBERNETES-CA, then self-signed because a root CA vouches for itself.

openssl genrsa -out ca.key 2048
openssl req -new -key ca.key -subj "/CN=KUBERNETES-CA" -out ca.csr
openssl x509 -req -in ca.csr -signkey ca.key -out ca.crt

Now an admin certificate, signed by that CA. The trick worth knowing: the group goes in the Organisation field, and system:masters is the built-in group that grants cluster-admin. The CN becomes your identity in the audit log, so name it sensibly.

openssl genrsa -out admin.key 2048
openssl req -new -key admin.key -subj "/CN=kube-admin/O=system:masters" -out admin.csr
openssl x509 -req -in admin.csr -CA ca.crt -CAkey ca.key -out admin.crt

With that you can hit the API directly, presenting your key, your cert and the CA cert so both sides trust each other:

curl https://kube-apiserver:6443/api/v1/pods \
  --key admin.key --cert admin.crt --cacert ca.crt

In practice you don't type all that, it lives in your kubeconfig. The API server certificate is the fiddly one, because the server answers to a bunch of names (the service IP, kubernetes.default, the node IP) and every one of them has to be listed as a SAN or clients reject it:

[alt_names]
DNS.1 = kubernetes
DNS.2 = kubernetes.default
DNS.3 = kubernetes.default.svc
DNS.4 = kubernetes.default.svc.cluster.local
IP.1  = 10.96.0.1
IP.2  = 172.17.0.87

Kubelets get their own pair per node, and their client certificate uses a very specific CN: system:node:<node-name>. That prefix is how the API server knows to hand out node-level permissions rather than treating the kubelet like a random user. Miss it and authorisation quietly does the wrong thing.

Reading certificates on a live cluster

This is the bit I'd actually be doing as a security person: walking onto a cluster and auditing what's there. Step one is working out how it was built. If it's a kubeadm cluster, the control plane runs as static pods with manifests under /etc/kubernetes/manifests and the certs sit under /etc/kubernetes/pki. A from-scratch systemd install hides the paths in unit files instead. Once you've found a cert, OpenSSL reads it back in plain English:

When I audit a cluster I keep a running table, one row per certificate, so nothing hides. Path, common name, the SANs, the issuer and, the one that bites people, the expiry date. Expired control plane certs are a classic 3am outage.

And when TLS breaks, the logs tell you. Under systemd it's journalctl -u etcd.service -l; on kubeadm it's kubectl logs <pod> -n kube-system; and if the control plane is down hard, drop to the container runtime with crictl logs (or docker logs). The phrase you're hunting for is tls: bad certificate, which almost always means a wrong CA, a missing SAN, or an expired cert.

Automating it: the Certificates API

Signing everything by hand doesn't scale past a tiny team. When a new admin joins, they shouldn't have to SSH to the master and touch the CA key, that key is the crown jewels, and every extra person near it is a risk. Kubernetes has a built-in answer: submit CSRs as objects and approve them with kubectl.

Jane creates her key and request the usual way:

openssl genrsa -out jane.key 2048
openssl req -new -key jane.key -subj "/CN=jane" -out jane.csr

Then the admin base64-encodes that CSR and drops it into an object. (Heads up: the old certificates.k8s.io/v1beta1 API is gone, use v1, which also wants a signerName.)

apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
  name: jane
spec:
  signerName: kubernetes.io/kube-apiserver-client
  groups:
    - system:authenticated
  usages:
    - digital signature
    - key encipherment
    - client auth
  request: <base64-encoded-CSR>

From there it's three commands: see it, approve it, pull the signed cert back out.

kubectl get csr
kubectl certificate approve jane
kubectl get csr jane -o yaml   # signed cert sits base64-encoded in .status.certificate

The signing itself is done by the controller manager, which is the component actually holding the CA key. You'll see it wired up in its manifest with --cluster-signing-cert-file and --cluster-signing-key-file both pointing at the CA. So the "who can mint certs" question really comes down to "who can approve CSRs and who can reach the controller manager's config", which is a lovely thing to check on an engagement.

When certificates stopped being scary

Certificates stopped being scary the moment I saw them as just "a public key with a signature saying who it belongs to". Everything else, the cluster CA, the server-and-client split, the SANs on the API server, the system:node: prefix, the CSR API, is that one idea repeated with different names. The security work is mostly boring diligence: right CN, right SANs, right issuer, not expired, and the CA key locked away from everyone who doesn't absolutely need it. I'm still shaky on the finer points of rotation and the front-proxy certs, so if you've automated cert rotation properly, I'd love to see how.

Next I want to turn all this into a working kubeconfig and then finally get into RBAC, since we've now got identities the cluster can actually authorise. If this helped, come say hi on LinkedIn or the contact page.

Further reading

• Kubernetes docs: PKI certificates and requirements
• kubeadm certs: management and renewal
• Kubernetes docs: Certificate Signing Requests

FAQ

Why does Kubernetes use so many certificates?

Because every component proves its identity with TLS. The API server, etcd, the kubelets and the admin all carry certificates so they can trust each other. It is one private PKI holding the cluster together.

What is a certificate authority (CA)?

A CA is the trusted signer. It issues and vouches for certificates, so if two parties trust the same CA they can trust each other's certificates. Kubernetes runs its own CA for the cluster.

How do I read a Kubernetes certificate?

With openssl. Pointing openssl x509 at a certificate file shows its subject, issuer, validity dates and usage, which is how you check who a certificate is really for and whether it has expired.

Related reading

• Topic 6: Security primitives and authentication (where authentication starts)
• Topic 8: Kubeconfig and API groups (using certs in a kubeconfig)
• Topic 9: Authorization and RBAC (then authorisation)
• Browse the whole Kubernetes Journey