click the screen · press Enter
← back to blog
Kubernetes Journey · Topic 19

Least Privilege in Kubernetes: Start at Zero, Add Back Slowly

Kubernetes least privilege and system hardening - Kubernetes Journey

Early on I did the thing everyone does. A workload wouldn't come up, the errors were about permissions, and I was tired. So I bound the service account to cluster-admin, the app started, and I moved on. It worked. That's the problem. It worked, and now a pod running some random image had the keys to the entire cluster, all because I couldn't be bothered to write a five-line Role.

That little shortcut is the exact opposite of what this topic is about. Least privilege is the habit of giving anything, a person, an account, a process, only the access it needs and not a scrap more. It sounds obvious. Almost everyone still gets it wrong, because "just make it work" is faster than "work out what it actually needs". So let's slow down and do it properly, on the node and in the cluster. What it is, how to apply it, and a lab where you actually trim the fat off a box.

What least privilege actually means

Least privilege means every identity gets the smallest set of permissions that still lets it do its job. Read-only if it only reads. One namespace if it only touches one namespace. No login for accounts that never log in. Nothing "just in case".

In practice you start from zero and add back only what's needed, instead of starting from "everything" and trying to take things away. Default deny, then a short allow-list. That flip is the whole trick. It's much easier to grant a missing permission when something breaks than to notice a spare one that's quietly sitting there for months.

Attackers don't need every door open. They need one. Every extra permission, port, service or package is another thing that can be abused or go wrong. Least privilege shrinks that surface so that when someone slips (and they will), the damage is boxed in. A popped pod with a tiny Role is an annoyance. A popped pod with cluster-admin is a very bad day.

Think of a hotel keycard. It opens your room and the gym, it doesn't open every other room, the manager's office or the safe, and it stops working at checkout. Nobody argues that's unfair. That's least privilege, and it's the model we want everywhere.

Give it what it needs, when it needs it, and nothing else. The rest is just detail.
01
One pod popped

Attacker lands in a low-value container

02
Reads its token

Service account mounted inside

03
Token is cluster-admin

The shortcut I took

04
Whole cluster

Every namespace, every secret

Least privilege is the thing that breaks this chain at step 3. A narrow Role and step 4 never happens.

Two places it lives: the node and the cluster

People hear "least privilege" and jump straight to RBAC. That's half of it. A Kubernetes cluster runs on nodes, and a node is just a Linux machine. If that machine is bloated with services you never use, ports nobody listens on and packages left over from setup, then no amount of tidy RBAC saves you. Someone breaks the box, they're on the node, and now they're next to the kubelet and every pod on it.

So I split this into two layers, and the lab does both:

LEAST PRIVILEGE
node access
open ports
running services
kernel modules
stale packages
RBAC roles
Same principle, six places to apply it. Five live on the Linux node, one lives in the cluster.
Lab scope Everything below runs on machines you own. I'm using a throwaway Ubuntu VM for the node bits and a kind cluster for the RBAC bits. Don't run this on anything shared or production, and never point it at a box you don't control.

Layer 1: hardening the node

The goal here is boring and effective: turn off things that are on for no reason. Fewer moving parts, fewer ways in. I'll go through four checks, each is one command to see the problem and one to fix it.

Close ports nothing needs

What this does: ss -tulpn lists every port the machine is listening on, with the program behind each one. A listening port is an open door, so the first job is to see how many doors you've actually got.

ss -tulpn # what's listening, and who owns it
Netid State Local Address:Port Process tcp LISTEN 0.0.0.0:22 sshd # fine, you need this tcp LISTEN 127.0.0.1:6443 kube-apiserver # fine on a node tcp LISTEN 0.0.0.0:23 inetd (telnet) # why is this here? udp LISTEN 0.0.0.0:2049 rpc.nfsd # NFS nobody uses # two ports i didn't ask for, both worth killing
Read every line and ask "do I actually use this?". The honest answer is usually no for a couple of them.

Anything you don't recognise gets investigated, then stopped or firewalled. That telnet line is a classic, an ancient service listening in clear text that no one turned on deliberately. It shouldn't exist on a modern box.

Turn off services you don't run

What this does: a service can be enabled (starts on every boot) without you ever noticing. This lists the enabled ones so you can spot the freeloaders, then disables the ones you don't want.

# see everything set to start at boot
systemctl list-unit-files --type=service --state=enabled

# stop it now AND stop it starting again
sudo systemctl disable --now telnet.socket
sudo systemctl disable --now nfs-server

disable --now does both halves in one go: it stops the service this second and unhooks it from boot, so it doesn't quietly come back after a reboot. That "after a reboot" bit caught me out once, I disabled something, felt clever, rebooted, and there it was again because I'd only stopped it.

Blacklist kernel modules that never load

What this does: the kernel can load drivers for hardware and network protocols you'll never use. Each loaded module is extra code running with the highest privilege on the box, so unused ones are pure risk. lsmod shows what's loaded; a blacklist file stops the ones you don't want.

Take dccp and sctp, two obscure network protocols most servers never touch. If they're not loaded and can't be loaded, that's a whole class of bugs you've opted out of. Here's how to make sure they can't load:

# is it loaded right now?
lsmod | grep -E 'dccp|sctp'

# stop it loading, ever: point the module at /bin/true
cat <<'EOF' | sudo tee /etc/modprobe.d/hardening.conf
install dccp /bin/true
install sctp /bin/true
blacklist dccp
blacklist sctp
EOF

# unload it now if it happens to be loaded
sudo modprobe -r dccp 2>/dev/null; sudo modprobe -r sctp 2>/dev/null

The install <module> /bin/true line is the strong one. It tells the kernel that "loading this module" means "run /bin/true and do nothing", so even something asking for it by name gets a shrug instead of the driver. blacklist alone only stops automatic loading, so I use both.

Rip out software you don't need

What this does: every installed package is code that can have holes. If it's not used, it's just risk sitting on disk waiting for a CVE. Find the obvious offenders and remove them.

hunt down and remove clear-text / legacy tools
$ dpkg -l | grep -E 'telnet|ftp|rsh-client' ii telnet 0.17-42 amd64 The telnet client ii ftp 0.17-36 amd64 classic ftp client $ sudo apt purge -y telnet ftp Removing telnet (0.17-42) ... Removing ftp (0.17-36) ... # two fewer clear-text tools an attacker can grab
purge removes the package and its config. Legacy clear-text clients are the first things I strip.

Now here's a quick way to see the node's whole hardening state at a glance. This is the sort of table a benchmark tool like kube-bench or CIS-CAT spits out, and it maps neatly onto least privilege: every FAIL is something running that doesn't need to.

Node least-privilege check
CheckStatusNote
telnet listening on :23FAILClear-text service, disable
dccp / sctp loadableFAILUnused protocols, blacklist
NFS server enabledWARNOff unless you actually serve NFS
SSH key-only, root login offPASSRestricted node access, good
Only sshd + kube ports openPASSMinimal listening surface
Flip "Failures only" to see just what's left to fix. That's your to-do list.

Layer 2: least privilege in the cluster (RBAC)

Now the part I fumbled at the start. In the cluster, least privilege is RBAC done with restraint: a narrow Role scoped to one namespace and a handful of verbs, bound to the one account that needs it. Not cluster-admin. Never cluster-admin for a workload.

The best part is Kubernetes gives you a truth-teller for this: kubectl auth can-i. It answers "is this identity allowed to do this?" without you having to reason about the YAML in your head. Let's use it.

Build a reader that can only read

Step 1, spin up a cluster and a namespace. kind gives you a throwaway one in about a minute.

kind create cluster --name lp-lab
kubectl create namespace dev

Step 2, check the default first. Every namespace ships with a default service account. See what it can do before you touch anything.

what can the default service account do?
$ kubectl auth can-i --list --as=system:serviceaccount:dev:default -n dev Resources Non-Resource URLs Verbs selfsubjectreviews [] [create] # almost nothing. that's correct: a fresh SA should be near-powerless
A default service account starts with basically no rights. Good. The mistake is what we grant on top.

Step 3, make a Role that reads pods, and only that. One namespace, three read verbs, one resource. This is least privilege written down.

kubectl create serviceaccount reader -n dev

cat <<'EOF' | kubectl apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: dev
  name: pod-reader
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list", "watch"]   # read only, no create/delete
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: reader-binds-pod-reader
  namespace: dev
subjects:
- kind: ServiceAccount
  name: reader
  namespace: dev
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io
EOF

Step 4, prove it (the verify step). Ask can-i the yes questions and the no questions. Least privilege means the yeses are tiny and the noes are everything else.

test the reader account both ways
$ kubectl auth can-i get pods --as=system:serviceaccount:dev:reader -n dev yes $ kubectl auth can-i delete pods --as=system:serviceaccount:dev:reader -n dev no $ kubectl auth can-i get pods --as=system:serviceaccount:dev:reader -n kube-system no # scoped to dev only, can't peek at kube-system # reads its own namespace, nothing else. exactly what we asked for
Three questions, three correct answers. This is what "it does its job and no more" looks like.

Compare that to the shortcut version, and the difference is night and day:

Bind the service account to cluster-admin so the app "just works". Now that token can read every secret, delete any workload, and create new admin bindings in any namespace. One popped pod equals a full cluster takeover.
A Role with get/list/watch on pods, bound in one namespace. If that pod is popped, the attacker can list some pods in dev. That's the whole blast radius. They can't reach secrets, other namespaces, or the control plane.
Same app, same effort once you've done it twice. Wildly different worst case.
kind delete cluster --name lp-lab   # tidy up when you're done

What to look for as a defender

Least privilege isn't a one-off, it rots. People add "temporary" grants and never remove them. So the useful thing is to keep checking. On a node and in a cluster, these are the smells I'd hunt:

  • Any workload service account bound to cluster-admin or a wide ClusterRole. Pull the list of ClusterRoleBindings and read every subject. Nine times out of ten there's one that shouldn't be there.
  • Wildcards in Roles: verbs: ["*"] or resources: ["*"] is almost always someone who didn't want to think about it.
  • Listening ports with no owner you can name. Re-run ss -tulpn after every change and after upgrades, new services sneak in.
  • Nodes you can SSH into as root, or with a password. Node access should be keys only, named users, sudo logged.
  • Packages and modules that crept back after a distro upgrade. Hardening drifts, so re-check it on a schedule, not once.
The "just to get it working" trap Every over-privileged account I've ever seen started as a five-minute fix under pressure. The grant is temporary, the removal never happens, and a year later it's load-bearing and terrifying. If you must grant wide access to unblock yourself, write yourself a ticket to narrow it, then actually do it. Temporary means temporary.
Sensible defaults Start every account and node from deny. Give a workload its own service account with a namespaced Role, never cluster-admin. On the node: keys-only SSH, only the ports you use, only the services you run, unused modules blacklisted, legacy packages purged. Then re-check with kubectl auth can-i and ss -tulpn on a cadence, because it drifts.

What I found in my own homelab

Writing this made me go back and audit my own homelab, and yeah, I found a couple of service accounts with more than they needed and an old NFS service listening on a box that serves nothing. Slightly embarrassing for a post about least privilege, but that's kind of the point: it's not a thing you know, it's a thing you keep doing. The principle is dead simple. The discipline is the hard bit. The part I'm still chewing on is the node side at scale, doing this by hand on one VM is fine, but keeping fifty nodes trimmed without it drifting clearly needs config management (Ansible, or baked images), and I've not built that muscle yet. If you run this properly across a fleet, I'd genuinely like to know what you use. What's the one over-privileged thing you always find when you go looking? Mine's a leftover cluster-admin binding, every time.

If this helped least privilege click, come say hi on LinkedIn or the contact page, and tell me what to break down next. More in the Kubernetes Journey.

Further reading

FAQ

What is the principle of least privilege?

Least privilege means giving any person, account or process exactly the access it needs to do its job, and nothing more. No spare permissions, no extra ports, no unused software sitting around. It shrinks what an attacker can reach if they get in, so a small mistake stays a small mistake.

How does least privilege apply to a Kubernetes node?

A node is just a Linux box, so the same rules apply. Only run the software you need, keep services off unless they are used, blacklist kernel modules that never load, close ports nothing listens on, and restrict who can log in. Every extra thing running is one more door an attacker can try.

What is least privilege in RBAC?

In RBAC it means each user and service account gets a narrow Role scoped to one namespace and a few verbs, instead of cluster-admin. A reader account should read, not delete. You check it with kubectl auth can-i, which tells you exactly what an identity is allowed to do.

Why is cluster-admin dangerous?

cluster-admin can do anything to anything in the cluster. Bind it to a service account to save time and that token becomes a master key. If a pod holding it is popped, the attacker owns the whole cluster. It breaks least privilege completely, which is why loose cluster-admin bindings are a top finding.

How do I find unnecessary open ports on a node?

Run ss -tulpn to list every listening TCP and UDP socket with the process behind it. Anything you do not recognise or do not need gets stopped, or firewalled off. Fewer listening ports means a smaller attack surface, which is least privilege applied to the network.