readOnlyRootFilesystem Isn't an Immutable Container

Here's the claim I'll defend for the next few minutes: readOnlyRootFilesystem: true is a good setting that gets sold as the wrong thing. Guides hand it to you as the way to make a container "immutable", as if flipping it means the container can no longer be tampered with while it runs. It can. A read-only root filesystem stops writes to one place. An immutable container is a much bigger promise, and this one setting doesn't keep it.

I still think you should set it. That's the odd part of this post. I'm arguing against the marketing, not the control. If you run workloads you didn't write, on a cluster where "immutable" is a box someone ticks on a compliance sheet, this is the gap between what that box says and what it does. Quick definition first, because the word does a lot of lifting: an immutable container is one whose running state can't be changed after it starts. No files edited, no packages added, no config swapped, no shell fiddling with it. Read-only root filesystem is just the "no files edited on the root disk" slice of that.

Why the read-only equals immutable idea sticks

It sticks because the setting does something you can see immediately. Turn it on, try to write a file, get told no. That feels total. And the framing in most tutorials reinforces it: they show the flag, show a failed write, and move on with "there, now it's immutable". Nobody's lying, they're just describing the smallest part of the job and letting the word "immutable" carry the rest. So people set the one flag, tick the box, and believe the container can't be messed with. Three things say otherwise.

It breaks real apps, so you carve the read-only back open

The first crack shows up before any attacker does. Take the standard Nginx image and set the flag, nothing else:

apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
  - name: nginx
    image: nginx
    securityContext:
      readOnlyRootFilesystem: true

It won't come up. Nginx needs to write its PID and its cache while it runs, so the moment the disk is read-only those writes fail and the container errors out:

The fix is to mount small writable volumes on exactly the paths it needs. An emptyDir is the usual choice: it's a scratch directory that lives with the pod and gets wiped when the pod dies, so it's writable storage that doesn't persist.

    securityContext:
      readOnlyRootFilesystem: true
    volumeMounts:
    - name: cache
      mountPath: /var/cache/nginx
    - name: run
      mountPath: /var/run
  volumes:
  - name: cache
    emptyDir: {}
  - name: run
    emptyDir: {}

Now it starts. But look at what you just did. "Read-only" already has two writable exceptions you punched into it by hand, and on a busier app that list grows to /tmp, a logs path, a work directory. Each hole is a place a process, or something that became that process, can still write. The setting is real, but it was never absolute, and the exceptions are yours to keep small.

It does nothing about a shell or memory

This is the big one, and it's where "immutable" falls over. A read-only root filesystem is about writing to disk. Most of what worries me at runtime doesn't touch disk.

Someone with the rights to kubectl exec into the pod gets an interactive shell inside a supposedly immutable container, and the read-only flag doesn't even slow that down. They can read your mounted Secrets, hit the service account token, poke the network, and run tools straight from memory without ever writing a file. Modern tradecraft is mostly fileless anyway, precisely to dodge disk-based detection, so a control that only watches the disk is watching the wrong door. The container's files never changed, and yet it's thoroughly not in its trusted state.

On a privileged container it's close to decoration

Here's the one that surprised me when I thought it through. The read-only flag keeps working even with privileged: true. Set both and a package install still fails on the read-only disk:

It looks like the control is holding the line. It isn't. A privileged container shares the host kernel with the doors wide open: it can reach host devices, write into /proc, and generally step out onto the node. The read-only root filesystem stops apt, and meanwhile the container has a path to the host that has nothing to do with its own disk. Read-only root is not isolation, and pairing it with privileged gives you a container that can't update itself but can own the machine. That's the wrong thing to feel reassured by.

The honest counter: so why set it at all?

Because raising the bar is worth doing even when it isn't a wall, and this raises it in a way I actually rate. A read-only root filesystem kills the laziest attacks outright: the web shell dropped into the web root, the cron file written for persistence, the modified binary left behind for next time. It forces an attacker off disk and into memory, and memory is where a runtime tool like Falco can actually notice them. It turns "quietly edited a file" into "had to do something noisier". None of that is nothing. It's a genuinely good control that I'd set on every workload that tolerates it.

The mistake isn't setting it. The mistake is stopping there and calling the container immutable. On its own it's one layer. It earns its keep next to the ones that cover what it can't: runAsNonRoot so the shell someone lands isn't root, dropped Linux capabilities so that shell can do less, no privileged so it can't reach the host, and tight RBAC so far fewer people can exec in at all.

Where the old advice actively misleads: PodSecurityPolicy

One correction, because the guides I learned this from all end the same way and it's now wrong. They tell you to enforce read-only roots with a PodSecurityPolicy. Don't reach for it: PodSecurityPolicy was removed in Kubernetes 1.25 and doesn't exist anymore. A policy/v1beta1 PodSecurityPolicy manifest applied to a current cluster does nothing useful.

What replaced it is built-in Pod Security Admission, which enforces the baseline and restricted profiles at the namespace level with a label. The catch worth knowing: the restricted profile requires runAsNonRoot, no privileged and dropped capabilities, but it does not force readOnlyRootFilesystem. So if you want a read-only root required across the cluster rather than left to whoever wrote the manifest, you need a policy engine like OPA Gatekeeper or Kyverno to demand it at admission. Pod Security Admission gets you most of the row above, just not that one tile.

I'll be honest about the edge of my own claim here. There's a reading where "immutable" is fine shorthand if everyone in the room already means "read-only root plus non-root plus no privileged, enforced at admission". If your team genuinely means all that when they say it, we're arguing about a word. My worry is that most places don't, and the single flag is doing the talking. If you've got a cluster where "immutable containers" is written down as a control, the thing worth checking tonight is simple: pull one running pod's spec and see whether readOnlyRootFilesystem is the only line there, or whether runAsNonRoot and privileged: false are sitting next to it. If it's on its own, that's the gap this whole post is about, and it's a five-minute fix per workload. If that's your cluster and you want a second pair of eyes on the securityContext, that's the kind of thing I'm happy to compare notes on, though I'll say plainly I've read far more of this than I've broken in a lab, so push back if your experience says different.

Check it yourself in one line

You don't need a lab to see where you stand. This lists every container that has no read-only root set, across the whole cluster:

kubectl get pods -A -o jsonpath='{range .items[*]}{range .spec.containers[*]}{.name}{"\t"}{.securityContext.readOnlyRootFilesystem}{"\n"}{end}{end}' | grep -vE 'true$'

Anything that comes back with a blank or false in the second column is a container whose root disk is writable at runtime. That's your real starting list, and it's usually longer than people expect.

References

• Kubernetes docs: configure a security context for a pod or container
• Kubernetes docs: Pod Security Admission
• Kubernetes docs: Pod Security Standards (baseline / restricted)
• Kubernetes blog: Pod Security Admission and the removal of PodSecurityPolicy
• Kubernetes docs: the emptyDir volume

FAQ

Does readOnlyRootFilesystem make a container immutable?

Not on its own. It blocks writes to the container's root filesystem, which stops an attacker dropping a file to disk, but it does nothing about someone running an interactive shell, executing code in memory, or writing to the emptyDir volumes you had to mount for the app to start. It is one control, not immutability.

Why does my Nginx pod fail to start with readOnlyRootFilesystem true?

The stock Nginx image needs to write its PID and cache at runtime, usually under /var/run and /var/cache/nginx. With a read-only root filesystem those writes fail and the container errors out. Mount a small writable emptyDir volume on each path it needs and it will start.

Does readOnlyRootFilesystem still work on a privileged container?

Yes, the root filesystem stays read-only even with privileged true, so package installs and file edits still fail. But a privileged container can reach the host through things like /proc and host devices, so the read-only setting does not restore the isolation that privileged threw away. Drop privileged instead of relying on this.

Can I still use PodSecurityPolicy to enforce a read-only filesystem?

No. PodSecurityPolicy was removed in Kubernetes 1.25 and no longer exists. Use built-in Pod Security Admission for baseline and restricted profiles, or a policy engine like OPA Gatekeeper or Kyverno to require readOnlyRootFilesystem, runAsNonRoot and no privileged across the cluster at admission.

Is a read-only root filesystem still worth setting?

Yes, definitely. It kills the easiest persistence and web-shell-to-disk tricks and forces an attacker into memory, where a runtime tool can see them. Just set it alongside runAsNonRoot, dropped capabilities and no privileged, and enforce the whole set at admission rather than trusting each author.

Related reading

• Topic 40: Falco in One Page (catching the in-memory activity read-only can't stop)
• Topic 29: Pod Security Policies Are Gone (what enforces this now)
• Topic 26: AppArmor and Linux Capabilities (shrinking what a landed shell can do)
• Topic 32: Containers Share One Kernel (why privileged reaches the host)
• My Kubernetes pentest notes (the securityContext checks without the argument)
• Browse the whole Kubernetes Journey