Verify the Binary Before You Trust It, Then Upgrade in Order

Two questions that don't get asked enough. When you download a Kubernetes binary and run it as root on your cluster, do you actually know it's the exact file the maintainers shipped? And once it's running, when did you last upgrade it, or is it quietly sat on a version full of known holes? This one's about both: trust what you install, then keep it current. Neither is glamorous, both are the difference between a solid cluster and a soft target.

I've split it into two halves. First, verifying a binary before you trust it, which takes about thirty seconds. Then the bigger job: upgrading a cluster without knocking your apps offline. Usual format, what, how, why, takeaway.

Half one: verify the binary before you trust it

Verifying a binary means checking that the file you downloaded is byte-for-byte the one the Kubernetes team published, using a checksum. A checksum (or hash) is like a fingerprint for a file: run the file through a hashing function and you get a fixed string. Change a single byte and the fingerprint changes completely. Kubernetes publishes a SHA-512 checksum next to every release.

That download travels across the internet and through mirrors you don't control. Someone sitting on the network, or a compromised mirror, could hand you a tampered binary with a backdoor baked in. This is a supply-chain attack, and for a cluster component that runs as root, it's about as bad as it gets. The fingerprint is how you catch it: if the hash you compute doesn't match the official one, the file isn't the real thing.

The check itself is three moves: download the file, compute its SHA-512 locally, and compare it against the value on the official Kubernetes releases page. Here's the whole dance:

Half two starts here: why the versions don't all match

Before upgrading anything, you need to know a rule that confuses everyone at first: the components of a cluster don't all have to run the same version. There's an allowed gap, called version skew, and it exists precisely so you can upgrade one piece at a time instead of stopping the world.

The kube-apiserver is the anchor. Everything is measured against it, and crucially, nothing is allowed to run newer than it. Here's how far each piece is allowed to trail:

Upgrade the API server first and everything else can lag safely behind it for a while. What you must never do is let a component jump ahead of the API server.

When to upgrade, and by how much

Two rules keep you out of trouble. First, Kubernetes only supports the three newest minor versions at a time. If the latest is 1.32, then 1.32, 1.31 and 1.30 get patches; once 1.33 lands, 1.30 falls off and stops getting security fixes. So upgrade before your version ages out, an unsupported cluster is one with unpatched CVEs, which for a security person is the whole ballgame.

Second, you move one minor version at a time. You can't leap from 1.30 to 1.33. You step: 1.30 to 1.31, then 1.31 to 1.32, then 1.32 to 1.33. Patch releases (the third number) are fine to jump, but minors go one at a time. It's slower, but it's how you avoid a surprise mid-upgrade.

The upgrade order: control plane first, then workers

The sequence matters. You upgrade the control plane (the master nodes) first, then the worker nodes. While the control plane is upgrading, you lose the management layer for a short window, no kubectl, no new deployments, but the apps already running on your workers keep serving traffic the whole time. After the control plane is done, you sit in a supported mixed-version state (control plane new, workers still old) until you work through the nodes.

For the workers themselves, you've got three strategies:

Doing it with kubeadm: the control plane

kubeadm is the tool that makes this bearable. It plans the upgrade, swaps the static pod manifests, and renews certificates for you. I'll walk the control plane first. In these commands I'm using 1.30 to 1.31 as the example, swap in your own versions.

Step 1. Point your package source at the new minor and refresh. Kubernetes packages live at pkgs.k8s.io now (the old Google-hosted repos are dead), and each minor version has its own repo line, so you edit the version in the URL:

echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.31/deb/ /" \
  | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update

Step 2. Find the exact target version, then plan. List what's available, then let kubeadm tell you what it'll do:

sudo apt-cache madison kubeadm      # shows e.g. 1.31.0-1.1
sudo kubeadm upgrade plan

The plan is the bit I actually read twice. It shows your current version, the target, and which components kubeadm handles automatically versus what you upgrade by hand (the kubelet, always):

Step 3. Upgrade kubeadm itself, then apply. kubeadm has to match the target series before it can drive the upgrade. The unhold/hold pair just tells apt to stop pinning the package while you change it:

sudo apt-mark unhold kubeadm
sudo apt-get install -y kubeadm='1.31.0-1.1'
sudo apt-mark hold kubeadm

sudo kubeadm upgrade apply v1.31.0    # upgrades the control-plane components

Step 4. Drain the control-plane node, upgrade its kubelet and kubectl, bring it back. The kubelet runs outside the control-plane pods, so it's a separate package step. Draining first evicts workloads safely (more on drain below):

kubectl drain controlplane --ignore-daemonsets

sudo apt-mark unhold kubelet kubectl
sudo apt-get install -y kubelet='1.31.0-1.1' kubectl='1.31.0-1.1'
sudo apt-mark hold kubelet kubectl
sudo systemctl daemon-reload && sudo systemctl restart kubelet

kubectl uncordon controlplane          # allow scheduling again

Then the workers, one at a time

Now the same idea per worker, and this is where the "drain" concept earns its keep. Draining a node does two things: it marks the node unschedulable (cordon) so no new pods land on it, and it evicts the pods already there so they reschedule onto other nodes. That's what lets you take a node out for maintenance without dropping traffic.

For each worker, from the node upgrade its kubeadm, then tell it to upgrade, then from the control plane drain it, upgrade the kubelet, and uncordon:

# on the worker: update kubeadm and let it upgrade the node config
sudo apt-mark unhold kubeadm
sudo apt-get install -y kubeadm='1.31.0-1.1'
sudo apt-mark hold kubeadm
sudo kubeadm upgrade node

# from the control plane: move workloads off the node
kubectl drain worker-1 --ignore-daemonsets

# on the worker: upgrade kubelet + kubectl, restart
sudo apt-mark unhold kubelet kubectl
sudo apt-get install -y kubelet='1.31.0-1.1' kubectl='1.31.0-1.1'
sudo apt-mark hold kubelet kubectl
sudo systemctl daemon-reload && sudo systemctl restart kubelet

# back in service
kubectl uncordon worker-1

Verify it worked

The proof is kubectl get nodes. The VERSION column shows each node's kubelet version, so during the upgrade you'll literally watch it change node by node. That mixed state is normal and supported:

Two habits worth keeping

Two habits, and both are really about the same thing: not blindly trusting software. Verifying a binary is thirty seconds that stops you running a backdoored kubelet. Upgrading on a schedule is what keeps your cluster off the "unsupported and unpatched" list that attackers love. What clicked for me this time was version skew, once I understood the API server is the anchor and nothing goes ahead of it, the whole one-piece-at-a-time upgrade dance suddenly made sense instead of feeling fiddly. I've done this in a kubeadm lab but not yet on anything with real traffic, so I'm sure there are production gotchas (PodDisruptionBudgets, long-draining nodes) I've not felt yet, shout if you've hit them. Next I want to tie this back to hardening, keeping a cluster patched is step one, and the CIS benchmarks are step two.

If this helped, come say hi on LinkedIn or the contact page, and tell me what to break down next. More in the Kubernetes Journey.

Further reading

• Kubernetes docs: upgrading kubeadm clusters
• Kubernetes docs: version skew policy
• Kubernetes releases (binaries + SHA-512 checksums)

FAQ

Why should I verify Kubernetes binaries before installing?

Binaries travel over the internet, where a tampered mirror or a network attacker could swap the real file for a malicious one. Comparing the file's SHA-512 checksum against the official value proves it arrived unchanged. A mismatch means do not run it, it is a supply-chain red flag.

Can Kubernetes components run different versions?

Yes, within limits called version skew. The kube-apiserver is the anchor; the controller manager and scheduler may trail it by one minor version, and the kubelet and kube-proxy by up to two. Nothing should run newer than the API server. This skew is what lets you upgrade one component at a time.

Can I upgrade Kubernetes by more than one minor version at once?

No. Kubernetes only supports upgrading one minor version at a time, so from 1.30 you go to 1.31, then 1.32, never straight to 1.32. Kubernetes also supports only the three newest minor releases, so upgrade before your version drops out of support.

Related reading

• Topic 12: kubectl proxy & port-forward (reaching the API and services)
• Topic 11: Kubelet security (the component you'll upgrade most)
• Topic 5: CIS Benchmarks & kube-bench (hardening after patching)
• Browse the whole Kubernetes Journey