click the screen · press Enter
Inside//UnixSingh
← back to blog
Kubernetes Journey · Topic 35

Kubernetes Isolation Is a Stack, and Most Clusters Build Half of It

By UnixSingh·7 Aug 2026·15 min read
The layers of Kubernetes tenant isolation from namespace and RBAC on the control plane down to network policy, storage and dedicated nodes on the data plane - Kubernetes Journey

Last topic I made the case that a namespace isn't a wall, and that real isolation is a handful of controls stacked on top of it. This one goes down the stack and looks at each layer as its own thing, because they aren't interchangeable. A network policy and a dedicated node both "isolate", but they stop completely different attacks, and I keep seeing clusters that nailed the top of the stack and left the bottom wide open.

There's one framing that made all of this click for me, so I'll lead with it: isolation splits cleanly into two halves. What a tenant can ask the API to do, and what their running workloads can do to each other. Get that split and the rest falls into place.

Kubernetes control plane isolation with namespaces, RBAC and quotas compared with data plane isolation using network policy, storage classes and dedicated nodes, each stopping a different attack
Two halves. The control plane guards the API, the data plane guards the machines. Most clusters do the left column and forget the right.

The split that makes it make sense

Every isolation control in Kubernetes lives in one of two planes. The control plane is the Kubernetes API, the brain that decides what gets created. The data plane is the nodes, the network and the disks, where your workloads actually run.

Control plane isolation stops a tenant asking the API for things that aren't theirs, using namespaces, RBAC and quotas. Data plane isolation stops one tenant's running pods reaching another's, using network policy, storage separation and where pods get scheduled.

The two failure modes are completely different. A control plane gap means a tenant can kubectl their way into your stuff. A data plane gap means a compromised pod, one that got in through a bug in the app, not the API, can pivot to another tenant over the network or the shared kernel. You can lock down one plane completely and still be wide open on the other.

Which means "is my cluster isolated" is really two questions. Who can the API be talked into helping, and what can a running workload touch once it's alive.

The isolation stack, and what each layer actually stops
LayerPlaneStops
Namespace + RBACControlA tenant reading or changing another's resources via the API
ResourceQuotaControlA tenant requesting more compute than its share
NetworkPolicyDataA pod opening a connection to another tenant's pod
Storage (Classes + PVCs)DataA tenant reaching another's volumes, or starving their disk
Node isolationDataA container escape landing on another tenant's workloads
Read the "stops" column as five different attacks. Skipping a row doesn't weaken the others, it just leaves that one attack open.

I covered the top two rows properly in the multi-tenancy topic, so I'll keep the control plane brief here and spend the real time on the data plane, which is where the gaps usually are.

Control plane, in one breath

A namespace carves the cluster into logical sections and is where nearly every other security feature attaches. RBAC, scoped with a Role and RoleBinding inside that namespace, decides who can do what. A ResourceQuota caps how much the namespace can consume. That's the control plane: three objects that between them answer "who's allowed to ask for what".

# the whole control-plane boundary for one tenant, roughly
kubectl create namespace development
# Role: what actions are allowed in here
# RoleBinding: which user/SA gets that Role
# ResourceQuota: the ceiling on cpu/memory/pods

The one thing I'll re-flag because it's the classic mistake: bind with a RoleBinding inside the namespace, never a ClusterRoleBinding, or the tenant escapes the namespace entirely. Everything else about RBAC is in Topic 34. Now the interesting half.

Data plane, layer one: the network

Quick recap because it's the foundation of data plane isolation, then the tenancy-specific bit. By default every pod can reach every other pod, across namespaces. A NetworkPolicy is how you change that, and the useful pattern for tenancy is allowing traffic from a whole namespace, not just a pod:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: backend-from-tenant-b-only
  namespace: tenant-a
spec:
  podSelector:
    matchLabels:
      app: backend
  policyTypes: ["Ingress"]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              tenant: tenant-b       # only pods from tenant-b's namespace
      ports:
        - protocol: TCP
          port: 8080

That says: tenant-a's backend accepts TCP on 8080, and only from pods living in a namespace labelled tenant: tenant-b. Everything else is dropped. The namespaceSelector is the tenancy tool, it lets you write cross-tenant rules by naming the other tenant's namespace instead of chasing individual pod labels.

The label the selector depends on A namespaceSelector matches labels on the namespace object, and namespaces don't get useful labels automatically. If the rule blocks everything including the traffic you meant to allow, the first thing to check is whether tenant-b's namespace actually carries tenant: tenant-b: kubectl label ns tenant-b tenant=tenant-b. And, as ever, none of this does anything unless your CNI enforces NetworkPolicy at all.

Data plane, layer two: storage

This is the layer I see skipped most, and it's a real one. Two tenants sharing a cluster are, by default, drawing volumes from the same pool. That's two problems in one: a tenant could reach another's data if the RBAC on claims is sloppy, and a tenant hammering the disk can slow everyone sharing that storage backend.

The first line of defence is free: a PersistentVolumeClaim is a namespaced object, so RBAC already stops tenant-a from listing or mounting tenant-b's claims, as long as you didn't hand out cluster-wide storage permissions. Verify it the usual way:

kubectl auth can-i get pvc -n tenant-b \
  --as=system:serviceaccount:tenant-a:default
# no  <- what you want to see

The second is a StorageClass per tenant tier. A StorageClass describes how volumes get provisioned, what kind of disk, how fast, from where. Giving tenants different classes separates where their storage comes from and lets you match performance to who's paying for it:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: tenant-critical
provisioner: kubernetes.io/aws-ebs
parameters:
  type: io2                 # provisioned-IOPS SSD for the critical tenant
  iopsPerGB: "50"
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
Two storage settings with security weight, not just performance reclaimPolicy: Delete means the underlying disk is wiped when the claim goes away, which you want for a departing tenant. Set it to Retain and old volumes linger with the previous tenant's data on them, waiting to be re-bound by accident. And volumeBindingMode: WaitForFirstConsumer holds off creating the volume until a pod is actually scheduled, so a volume for a node-pinned tenant gets provisioned in the right place instead of somewhere the pod can't reach it. The defaults are about performance. These two are about not leaking data.
What actually separates tenant storage
ControlWhat it isolates
PVCs are namespaced + RBACWho can see and mount a claim (free, already there)
StorageClass per tenantWhere volumes come from, and their performance tier
reclaimPolicy: DeleteWhether a departed tenant's data is wiped or lingers
ResourceQuota on storageHow much a tenant can provision before being cut off
Namespaced PVCs are the isolation. StorageClasses are the tiering and the provenance. You want both.

Data plane, layer three: the node

The bottom of the stack, and the one that matters most the moment you stop trusting your tenants. Everything above still leaves every tenant's pods mixed together on the same physical machines, sharing the same kernel. Node isolation says: give this tenant its own nodes, so a container escape lands somewhere that only hurts them.

The mechanism is taints and tolerations, and the mental model is a bouncer. A taint on a node is a "you're not welcome here" sign that repels every pod by default. A toleration on a pod is the wristband that lets it past that specific sign. Taint a node for a customer, give only that customer's pods the matching toleration, and general workloads stay off.

1. Taint the node so ordinary pods won't schedule onto it:

kubectl taint nodes node-01 tenant=customer-a:NoSchedule

2. Give the tenant's pods a matching toleration so they're allowed on:

apiVersion: v1
kind: Pod
metadata:
  name: customer-a-web
  namespace: customer-a
spec:
  tolerations:
    - key: "tenant"
      operator: "Equal"
      value: "customer-a"
      effect: "NoSchedule"
  nodeSelector:
    tenant: customer-a          # see the gotcha below, this line matters
  containers:
    - name: web
      image: nginx
The trap that catches everyone: a toleration permits, it does not pin This is the one thing to take away from the whole node section. A toleration only says the pod is allowed onto the tainted node. It does not say the pod must go there. Without more, the scheduler is free to place that pod on any untainted node instead, so your "dedicated" tenant pod happily lands on shared infrastructure and you never notice. To actually pin it, you need the other half: label the node (kubectl label node node-01 tenant=customer-a) and add a nodeSelector or node affinity so the pod is required there. Taint keeps others off; nodeSelector keeps yours on. You need both, and almost every tutorial shows only the taint.

3. Verify the pod actually landed where you meant:

kubectl get pod customer-a-web -n customer-a -o wide
NAME READY STATUS NODE customer-a-web 1/1 Running node-01 $ kubectl describe node node-01 | grep -A1 Taints Taints: tenant=customer-a:NoSchedule
On node-01, the tainted one, and nothing without the toleration can join it. That's a dedicated node pool in three commands.

Combine dedicated nodes with a sandboxed runtime from the sandboxing topic and you've built most of what "hard multi-tenancy" actually means: even if a workload breaks out of its container, it's on hardware running only its own tenant, and it has a smaller kernel surface to break out through in the first place.

The honest trade: this costs utilisation

Node isolation isn't free, and it's worth being clear-eyed about why. The entire economic argument for containers is packing lots of workloads densely onto shared machines. Dedicating nodes per tenant throws some of that away, you'll have a customer's nodes sitting half-idle because you can't fill them with anyone else's pods. That's the deal: you're buying a stronger blast-radius boundary with lower utilisation and a bigger bill.

So spend it where the threat is Pin the tenants that need pinning, the ones running untrusted code or under a compliance rule, and let everyone else share. This is the same soft-vs-hard judgement from Topic 34, just applied at the node layer: match the isolation to the threat model, and don't buy dedicated nodes for internal teams who trust each other.

What I check, layer by layer

On an authorised review of a shared cluster, I walk the stack top to bottom, because a gap at any layer undoes the ones above it:

# control plane: can a tenant identity act outside its namespace?
kubectl auth can-i --list --as=system:serviceaccount:tenant-a:default
kubectl get clusterrolebindings -o wide | grep -v '^system:'

# network: is there any policy at all, or is it flat?
kubectl get networkpolicy -A

# storage: are PVCs actually walled by namespace, or is there a cluster-wide grant?
kubectl auth can-i get pvc -n tenant-b --as=system:serviceaccount:tenant-a:default

# node: are 'dedicated' tenants actually pinned, or just tolerated?
kubectl get nodes -o custom-columns=NODE:.metadata.name,TAINTS:.spec.taints
kubectl get pods -A -o custom-columns=NS:.metadata.namespace,POD:.metadata.name,NODE:.spec.nodeName

The one that pays surprisingly often is the last pair. A cluster will have taints on its nodes and tolerations on its pods, everyone assumes tenants are pinned, and then get pods -o wide shows a "dedicated" customer's workloads scattered across shared nodes because nobody added the nodeSelector. The taint was doing half a job and looked like it was doing the whole one.

The split I wish I'd had a year ago

The control-plane-versus-data-plane split is the bit I wish I'd had a year ago. Before it, "Kubernetes isolation" was a bag of features I half-remembered. After it, it's two questions with a clear division of labour, and I can look at any cluster and ask them in order. RBAC gaps and network gaps aren't the same finding, and treating them as one blurred my thinking for longer than it should have.

My honest opinion on the stack: the top is easy and the bottom is where the real security lives, which is exactly backwards from where most effort goes. Namespaces and RBAC are the first thing everyone sets up and the first thing every tutorial covers. Storage and node isolation are the layers that actually contain a compromised workload, and they're the ones I most often find missing or half-built. The taint-without-nodeSelector thing isn't a rare mistake, it's close to the default outcome of following the docs literally.

Where I'm still unsure: I've done all of this in a homelab with a couple of nodes, so I've never felt the utilisation pain of node-per-tenant at real scale, where "half-idle dedicated nodes" turns into a serious cloud bill. Everyone who runs hard multi-tenancy tells me that trade-off is the whole game, and I believe them, but I can't yet tell you first-hand where the line sits between "worth it" and "just use separate clusters". If you run this in production, I'd like to know how you decided.

Next I want to switch tracks and get properly offensive again: what's actually inside a container image, and how much attack surface you inherit the moment you type FROM.

References

FAQ

What is the difference between control plane and data plane isolation in Kubernetes?

Control plane isolation limits what a tenant can do through the Kubernetes API, using namespaces, RBAC and resource quotas. Data plane isolation limits what running workloads can do to each other, using network policies, storage separation and node placement. One protects the API, the other protects the machines.

Do taints and tolerations provide security isolation?

Partly. A taint keeps other pods off a node and a matching toleration lets a tenant's pods on, which stops the noisy-neighbour problem and reduces blast radius. But a toleration only permits scheduling, it does not force it, so you pair taints with nodeSelector or affinity to actually pin a tenant to its nodes.

How does storage isolation work in Kubernetes?

Each tenant's PersistentVolumeClaims live in their own namespace, so RBAC already stops one tenant reading another's claims. StorageClasses add per-tenant provisioning, giving critical tenants faster disks and separating where volumes come from, so one tenant's storage load or a leftover volume cannot affect another.

Is a dedicated node per tenant the same as hard multi-tenancy?

It is a big part of it. Dedicated nodes mean a container escape lands on hardware running only that tenant's workloads, so it cannot reach others through the shared kernel. Combined with a sandboxed runtime it is most of what hard multi-tenancy means, at the cost of lower utilisation.

Which isolation layer should I add first?

Namespaces and RBAC first, because without them there is no boundary at all. Then network policy, because pods talk to each other by default. Storage and node isolation come next as the threat model hardens. The order matters: each layer assumes the ones below it are in place.