click the screen · press Enter
← back to blog
Kubernetes Journey · Topic 27

RBAC Never Opens the Pod Spec. Admission Controllers Do.

Kubernetes admission controllers sitting between RBAC and etcd, validating and mutating API requests - Kubernetes Journey

A few weeks back, while I was still deep in the least privilege mindset, I set myself a small trap in my kind cluster. I built a Role that could only create pods, nothing else, textbook least privilege. Then I used that Role to create a pod running as root, with a hostPath mount pointed straight at the node's filesystem. RBAC didn't so much as pause. It let it straight through.

That annoyed me for about ten minutes before it clicked why. RBAC was never asked "is this pod dangerous". It was asked "can this identity create a pod", and the answer was yes. RBAC checks the verb, not the payload. There's a whole layer that reads the actual object you're sending and can say "hold on, not like that", and until this topic I'd been treating it as background noise. It's called admission control, and it turns out to be doing more work in every cluster I've touched than I'd given it credit for.

RBAC alone only checks if you can act, admission control inspects and can mutate or reject the actual pod object before it reaches etcd
RBAC answers "can you act". Admission control answers "what does that action produce".

Where admission control actually sits

Every kubectl command you run turns into an HTTP request against the API server, and that request walks a fixed path before anything gets saved. First it's authenticated: your kubeconfig hands over a client certificate or token and the API server confirms you are who you say you are (I covered the certificate side of this back in the RBAC post). Then it's authorized: RBAC checks whether that identity is allowed to do this verb on this resource. If both pass, the request reaches admission control, and only after that does it get written to etcd, the database that is Kubernetes' entire memory of your cluster.

01
Authentication

Who are you? Cert or token checked.

02
Authorization

RBAC: can this identity do this verb?

03
Admission control

Inspect, mutate or reject the object itself

04
etcd

Only now does it get persisted

Admission control is the last gate before the cluster remembers something forever.

Nothing before that third box knows or cares what's inside your YAML. RBAC would happily wave through a pod asking for privileged mode, an image tagged :latest from a registry nobody vetted, or runAsUser: 0, so long as the identity making the request has create on pods. Admission control is where the cluster finally opens the object and reads it.

Two kinds of admission controller, and an order that matters

Not every admission controller does the same job. There are two shapes, and knowing which is which explains a lot of otherwise odd behaviour:

  • Validating controllers can only say yes or no. They look at the object and either let it through unchanged or reject it with an error.
  • Mutating controllers can rewrite the object on the way past, adding a default value, injecting a label, attaching a sidecar container, before it's saved.

The order isn't arbitrary either: every mutating controller runs first, as a full pass, and only once all of them have had their turn do the validating controllers run against whatever the object looks like after mutation. That ordering is deliberate. It means a validator never rejects a field a mutator was about to fill in for you, like a missing storage class or a missing service account token.

Same admission controller, different pass A single plugin can register as both. PodSecurity, for example, is purely validating, but plenty of the built-ins in the table below quietly do both jobs in one plugin.

A handful worth knowing by name

The API server ships with a long list of built-in admission controllers, most of which you'll never think about because they're already doing sensible things by default. These are the ones I keep bumping into:

Admission controllers I actually reference
ControllerWhat it doesKind
NamespaceLifecycleRejects objects aimed at a namespace that doesn't exist; protects default/kube-system/kube-public from deletionValidating
PodSecurityEnforces a baseline/restricted security profile per namespace, blocking root, privileged mode, host mountsValidating
DefaultStorageClassAttaches the default StorageClass to a PVC that didn't name oneMutating
ServiceAccountAttaches the right service account and mounts its token into new podsMutating
NodeRestrictionStops a kubelet modifying node objects or pods outside its own nodeValidating
AlwaysPullImagesForces every pod to re-pull its image, so a cached local image can't sneak in unpatchedMutating
EventRateLimitCaps how many requests the API server processes from one source, an anti-DoS controlValidating
PodSecurity and NamespaceLifecycle are the two you'll meet the most; both are on in any current cluster.

I'm not going to pretend I have the full in-tree list memorised, I don't, and honestly most of it you'll never need to touch directly. The two that matter for day-to-day security work are PodSecurity and NamespaceLifecycle, so that's where the hands-on part is going to live.

Hands-on: see what's actually enabled first

Before changing anything, check what's already running. Spin up a disposable cluster, same as previous topics:

kind create cluster --name admission-lab

On a kubeadm-based cluster, and kind is one under the hood, the API server itself runs as a static pod in kube-system. You can exec straight into it and ask the binary what it was started with:

kubectl get pods -n kube-system | grep kube-apiserver
kubectl exec kube-apiserver-admission-lab-control-plane -n kube-system -- kube-apiserver -h | grep enable-admission-plugins
kube-apiserver -h | grep enable-admission-plugins
--enable-admission-plugins strings admission plugins that should be enabled in addition to default enabled ones (NamespaceLifecycle, LimitRanger, ServiceAccount, DefaultStorageClass, ResourceQuota, PodSecurity, MutatingAdmissionWebhook, ValidatingAdmissionWebhook, ...)
That's the flag's own help text listing the defaults, no cluster-hunting required.

Worth noticing: PodSecurity and NamespaceLifecycle are already in that default list. Nothing to enable, nothing to configure. They're just quietly doing their job on every request that reaches this cluster, which is exactly why the trap I set myself earlier only worked because my Role's namespace had no PodSecurity label on it yet, more on that in a second.

Hands-on: watch admission control overrule what RBAC allowed

Two small experiments. The first shows a controller you get for free; the second recreates the exact trap from the top of this post, on purpose, so you can see the fix land.

Experiment 1: a namespace that doesn't exist yet

Try to put a pod somewhere that isn't there:

kubectl run nginx --image nginx --namespace blue
Error from server (NotFound): namespaces "blue" not found

Authentication passed, RBAC (assuming you're cluster-admin locally) passed, and the request still died, because NamespaceLifecycle checked whether "blue" is a real namespace and it isn't. This is a nice, honest demo because there's genuinely nothing to configure, it's just there.

A bit of history worth knowing Older Kubernetes split this into two separate controllers: NamespaceExists, which rejected requests to missing namespaces, and NamespaceAutoProvision, which could auto-create the namespace instead of rejecting. Both are deprecated now, merged into the single NamespaceLifecycle you just saw. If you go looking for NamespaceAutoProvision on a modern cluster expecting to flip it on, you'll be looking for a while, I was.

Experiment 2: RBAC says yes, PodSecurity says no

Now the one that actually matters. Make a namespace and tell Kubernetes to enforce the restricted profile on it, which is the strict end of PodSecurity, no root, no privilege escalation, no unnecessary capabilities:

kubectl create namespace secure-ns
kubectl label namespace secure-ns pod-security.kubernetes.io/enforce=restricted

Now try to create a pod that explicitly asks to run as root, the same shape of pod that sailed through in my Role-only test:

kubectl run root-pod --image nginx --namespace secure-ns \
  --overrides '{"spec":{"securityContext":{"runAsUser":0},"containers":[{"name":"root-pod","image":"nginx","securityContext":{"runAsUser":0}}]}}'
kubectl run root-pod --namespace secure-ns ...
Error from server (Forbidden): pods "root-pod" is forbidden: violates PodSecurity "restricted:latest": > allowPrivilegeEscalation != false > unrestricted capabilities (must set capabilities.drop=["ALL"]) > runAsNonRoot != true > seccompProfile (must set type to RuntimeDefault or Localhost)
RBAC let the request through the door. PodSecurity read the pod and stopped it at the desk.

That's the exact moment I was missing three weeks ago. Same RBAC, same create pods permission, completely different outcome, because this namespace has a validating admission controller actually looking at the pod spec. Fix the container so it satisfies the restricted profile and the identical Role now succeeds:

kubectl run safe-pod --image nginx --namespace secure-ns --overrides '{
  "spec": {
    "securityContext": {"runAsNonRoot": true, "seccompProfile": {"type": "RuntimeDefault"}},
    "containers": [{"name":"safe-pod","image":"nginx","securityContext":{
      "allowPrivilegeEscalation": false,
      "capabilities": {"drop": ["ALL"]}
    }}]
  }
}'
# pod/safe-pod created
This is the whole point of the post The Role never changed. What changed is that the namespace now has a validating admission controller reading the object, not just the API server checking a verb. RBAC and admission control aren't competing controls, they're stacked ones, and skipping the second one is how a "least privilege" Role still lets someone create a dangerous pod.

Beyond the built-ins: webhooks, OPA, Kyverno

Everything above ships in the API server binary. But the two most useful built-ins, in a very literal sense, are MutatingAdmissionWebhook and ValidatingAdmissionWebhook, because they don't enforce a policy themselves, they call out to a webhook server you run and let it decide. That's the extension point tools like OPA Gatekeeper and Kyverno plug into: instead of the fixed rules built into PodSecurity, you write your own policy ("every image must come from our internal registry", "every deployment must set resource limits") and the webhook enforces it cluster-wide. I haven't run either of these in anger yet, my homelab hasn't needed custom policy beyond what PodSecurity gives me, but it's clearly the next layer once "no root, no privileged" stops being enough for what you're trying to enforce.

What this looks like from the defending side

An admission denial isn't quiet. It fails the kubectl command with the message you saw above, and it also shows up as a Kubernetes Event against the object that got rejected:

kubectl get events -n secure-ns --field-selector reason=FailedCreate

If API audit logging is turned on for your cluster, the denial is captured there too, at whatever audit level you've set, which is the more durable record if you actually want to alert on "someone keeps trying to run privileged pods in a restricted namespace" rather than relying on events, which age out. Repeated PodSecurity denials from the same identity is a genuinely useful signal: either a misconfigured deployment pipeline, or someone actively probing what they can get away with.

Where this leaves things

The framing that finally stuck for me: RBAC is the door policy, it decides who's allowed to walk in and what they're allowed to carry. Admission control is what happens once you're inside with your bag on the table, someone actually looks at what's in it. You need both. A tight RBAC Role with no PodSecurity enforcement on the namespace is a door policy with nobody checking bags, which is precisely the gap my little experiment fell into.

The bit I'm still not confident on: how far to push custom policy with Gatekeeper or Kyverno before it turns into its own maintenance burden, rules drifting out of date, false positives blocking a legitimate deploy at 5pm on a Friday. For now my rule is simple, PodSecurity: restricted on anything that isn't explicitly a platform namespace, and I'll reach for a policy engine only when a rule genuinely can't be expressed by the built-in profiles. If you're running Gatekeeper or Kyverno at scale and have a sane way to keep the policies from rotting, that's the thing I'd like to learn next.

FAQ

What is an admission controller in Kubernetes?

A plugin inside the API server that runs after authentication and authorization but before an object is saved to etcd. It can inspect the object you're sending (a pod, a namespace, anything) and either reject it outright or change it on the way through. RBAC decides if you may act; admission controllers decide what that action is allowed to produce.

What's the difference between a validating and a mutating admission controller?

A mutating controller can edit the object before it's saved, like injecting a default storage class or a sidecar container. A validating controller can only accept or reject it, no edits. Mutating controllers always run first, as a full group, then validating controllers run against the result, so nothing gets rejected based on a field a mutator was about to fix.

Why isn't RBAC enough to stop someone creating an insecure pod?

RBAC only checks verbs and resource types, like whether you can create pods at all. It never opens the pod spec, so it can't tell a pod running as root from a harmless one, or catch a hostPath mount to the filesystem. That's a different job, and it belongs to admission controllers like PodSecurity.

How do I check which admission controllers are enabled on my cluster?

On a kubeadm-based cluster (kind included), exec into the kube-apiserver static pod and grep its own help output: kubectl exec kube-apiserver-<node> -n kube-system -- kube-apiserver -h | grep enable-admission-plugins. That prints the exact flag value the API server was started with, defaults and all.

What happened to the NamespaceExists and NamespaceAutoProvision admission controllers?

Both are deprecated. Modern Kubernetes merged their jobs into a single controller, NamespaceLifecycle, which is on by default in any real cluster. It rejects objects aimed at a namespace that doesn't exist and blocks deletion of default, kube-system and kube-public, without you configuring anything.