Kubernetes Fairness: a Quota Is a Ceiling, Not a Referee

Two topics ago I set a ResourceQuota on a tenant and moved on feeling like I'd solved the noisy-neighbour problem. I hadn't. A quota caps what a tenant can ask for. It says nothing about who wins when the cluster is actually short on something, and that's a completely separate set of controls I'd never really looked at.

Here's the gap in one line: a quota is a ceiling, not a referee. It stops a tenant requesting a hundred cores. It does not decide whose pod keeps running when the node runs out of memory, or whose API calls get answered when the API server is drowning. This topic is the referee, three mechanisms that only matter when things get scarce, plus one leaky default that has nothing to do with resources at all.

Ceiling vs referee

These are two different jobs that get muddled together constantly. A ResourceQuota is admission-time accounting: it refuses a request that would push a namespace over its cap, and its work is finished the moment the object is accepted. Fairness is runtime arbitration. It decides who actually gets served when demand exceeds supply right now, long after admission stopped caring.

That arbitration doesn't live in one place either, which is part of why it's easy to miss. It happens in three, and each one owns a different chokepoint. The API server runs API Priority and Fairness, deciding whose requests get processed at all. The scheduler and kubelet run Pod Priority and Preemption, deciding whose pods get placed and whose get kicked off to make room. And the kubelet runs QoS classes, deciding whose pods die first when a node runs out of memory.

So you can set perfect quotas and still have one tenant take the cluster down. A runaway controller hammering the API, or a tenant whose pods declare no memory limits and get OOM-killed in a way that drags neighbours with them, are both comfortably within quota the entire time. The quota was never the thing standing between them and the outage. It was never asked to be.

Quotas answer "how much can you have". Fairness answers "who wins when there isn't enough". Multi-tenancy needs both, and the second half is the one that gets skipped, because it only shows up on a bad day.

API Priority and Fairness: protecting the front door

The API server is a single shared resource every tenant talks to for everything, creating pods, listing services, watching for changes. One tenant with a buggy controller retrying in a tight loop can flood it and slow the API down for everyone, and no ResourceQuota touches API traffic.

API Priority and Fairness (APF) is the built-in defence. It queues incoming requests into priority levels and hands out the server's limited concurrency fairly, so a flood in one bucket can't starve the others. It's on by default and went stable in Kubernetes 1.29, so on any current cluster it's already protecting you, quietly.

Two objects drive it. A PriorityLevelConfiguration defines a bucket and how much concurrency it's guaranteed. A FlowSchema decides which requests land in which bucket:

apiVersion: flowcontrol.apiserver.k8s.io/v1
kind: PriorityLevelConfiguration
metadata:
  name: tenant-critical
spec:
  type: Limited
  limited:
    nominalConcurrencyShares: 30    # a big slice of the server's concurrency
    limitResponse:
      type: Queue
---
apiVersion: flowcontrol.apiserver.k8s.io/v1
kind: FlowSchema
metadata:
  name: tenant-critical-traffic
spec:
  priorityLevelConfiguration:
    name: tenant-critical
  matchingPrecedence: 1000          # lower number = checked first
  rules:
    - subjects:
        - kind: ServiceAccount
          serviceAccount:
            name: default
            namespace: tenant-critical
      resourceRules:
        - verbs: ["*"]
          apiGroups: ["*"]
          resources: ["*"]

See what's already there and, more usefully, whether anything is getting queued or dropped:

Honestly, most people never need to write a custom FlowSchema. The defaults already isolate system traffic from workload traffic. It's worth knowing APF exists mostly so that when the API server feels slow, you check the rejected-requests metric instead of blaming the network, and so that on a genuinely busy multi-tenant cluster you can give a critical tenant its own guaranteed slice.

Pod Priority and Preemption: who gets on the node

Now down to the nodes. When the cluster is full and a new pod can't be scheduled, Kubernetes doesn't just leave it pending forever. If that pod is high priority, it can preempt, evict a lower-priority pod to free up room and take its place. Priority is set with a PriorityClass:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: tenant-critical
value: 1000000
globalDefault: false
description: "Production workloads for the critical tenant."
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: tenant-batch
value: 1000
globalDefault: false
description: "Batch and dev workloads, safe to evict."

Attach it with one field on the pod, priorityClassName: tenant-critical, and that pod now jumps the scheduling queue and can push batch pods off a full node to get scheduled.

QoS classes: who dies first when memory runs out

Priority is about getting onto a node. QoS is about staying on it. When a node runs out of memory, the kubelet has to kill something, and the order it picks isn't random, it's driven by the pod's Quality of Service class. And here's the part that surprises people: you don't set the QoS class. Kubernetes infers it from how you wrote your requests and limits.

Guaranteed means you pinned requests and limits to the same value, so the kubelet knows exactly what the pod needs and reserves it. These die last:

resources:
  requests: { cpu: "500m", memory: "512Mi" }
  limits:   { cpu: "500m", memory: "512Mi" }   # equal => Guaranteed

Burstable means you set a floor but allowed a higher ceiling, fine for variable workloads, evicted before Guaranteed. BestEffort means you set nothing, and that's the trap:

# no resources block at all => BestEffort => first to be killed

Check what class a running pod actually landed in, because it's not always what you'd guess:

How the three fit together

A default that leaks: cross-namespace DNS

Switching gears, because not every multi-tenancy gap is about resources. By default, DNS in a cluster is completely open across namespaces. A pod in tenant-a can resolve tenant-b's services by their fully qualified name and, out of the box, connect to them:

# from a pod in tenant-a, this resolves tenant-b's service just fine
nslookup backend.tenant-b.svc.cluster.local

That's handy for legitimate cross-service traffic and unhelpful for tenant isolation, because one tenant can enumerate another's services just by guessing names.

The control that actually works is the one from earlier in the series: a NetworkPolicy. A DNS name that resolves but that your NetworkPolicy won't let you connect to is mostly harmless, the attacker learns a service exists and can't reach it. So don't chase a magic CoreDNS setting. Default-deny cross-namespace traffic with NetworkPolicy, and the DNS "leak" stops mattering, because discovery without reachability isn't much of a leak.

And one more: traffic between pods isn't encrypted

The other default worth naming: pod-to-pod traffic is plain text unless you do something about it. On a shared multi-tenant network, anyone who can sniff the node's traffic can read it. The fix is mutual TLS between workloads, which I covered end to end in the mTLS topic, so I won't repeat it. The short version: a service mesh (Istio, Linkerd) gives every pod a sidecar that encrypts traffic transparently, and a CNI like Cilium can do the same at the kernel level with eBPF and WireGuard, no sidecar. Either way the application doesn't change. The one thing to carry over from that topic: if you use Istio, make sure it's in strict mode, not permissive, or the encryption you think is mandatory is optional.

What I check

On a shared cluster, once the walls from the last two topics are covered, these are the fairness and leak checks I'd add:

# APF: is anything being throttled, and who has custom priority?
kubectl get --raw /metrics | grep apiserver_flowcontrol_rejected_requests_total
kubectl get flowschema -o custom-columns=NAME:.metadata.name,PL:.spec.priorityLevelConfiguration.name

# priority: can a tenant hand itself a high PriorityClass and preempt neighbours?
kubectl get priorityclass -o custom-columns=NAME:.metadata.name,VALUE:.value
kubectl auth can-i use priorityclass/system-cluster-critical \
  --as=system:serviceaccount:tenant-a:default

# QoS: which pods are one memory spike from eviction?
kubectl get pods -A -o custom-columns=NS:.metadata.namespace,POD:.metadata.name,QOS:.status.qosClass | grep BestEffort

# DNS + encryption: can a tenant reach across, and is traffic in the clear?
kubectl run t --rm -it --image=busybox --restart=Never -n tenant-a \
  -- nslookup backend.tenant-b.svc.cluster.local
kubectl get peerauthentication -A -o yaml | grep -i mode

The two that pay: a tenant that can use a high-value PriorityClass has an eviction weapon, and a pile of BestEffort pods on shared nodes is a stability finding waiting for the first memory spike. Both look completely fine right up until the cluster is under load, which is exactly when you don't want to be discovering them.

Where the docs and I part ways

Pod Priority is documented as a scheduling feature. Read the upstream page and it's all about making sure your important workloads get placed, with preemption presented as a helpful side effect. I think that framing is wrong, or at least dangerously incomplete, because preemption is the only mechanism here that lets one tenant reach across and terminate another tenant's running pods. That is a denial-of-service primitive. It doesn't stop being one because the field that triggers it is called priorityClassName and lives in a scheduling doc. If a tenant can choose their own PriorityClass, you have handed them a lever to starve their neighbours, and nowhere in the official material does that get called what it is.

The other disagreement is about emphasis. Given three mechanisms, the docs give APF the most careful treatment, and it's the one I'd tell you to leave alone. The defaults isolate system traffic from workload traffic perfectly well and hand-tuning FlowSchemas is a genuinely niche need. QoS gets a fraction of the attention and it's the one that matters to almost everybody, because "no resources block" silently means "first to die" and that footgun is sitting in half the manifests I've ever read. The thing you're most likely to get bitten by is the thing you're least likely to have read about.

I'll be honest about the limit of my own view here, because it's a real one. All of this came from reading and from a two-node homelab. I've set up the RBAC to stop preemption abuse, but I have never watched a production cluster under genuine memory pressure decide who lives, and I'd guess the theory and the 3am reality diverge in ways I can't predict from my setup, particularly around how brutal BestEffort eviction feels when it's your service. If you've been on-call for that, I'd like to know what you wish you'd set beforehand.

That wraps the multi-tenancy and isolation run. Next I'm finally switching tracks to something properly offensive: what's actually inside a container image, and how much attack surface you inherit the moment you type FROM.

References

• Kubernetes docs: API Priority and Fairness
• Kubernetes docs: pod priority and preemption
• Kubernetes docs: pod quality of service classes
• Kubernetes docs: resource quotas and limits
• CoreDNS: the kubernetes plugin (the real directives)

FAQ

What is API Priority and Fairness in Kubernetes?

API Priority and Fairness (APF) is a built-in kube-apiserver feature that queues and prioritises incoming API requests so no single client can flood the API server. It went stable in Kubernetes 1.29. FlowSchemas match requests to a priority level, and each level gets a share of the server's concurrency.

What is the difference between Pod Priority and QoS class?

Pod priority, set with a PriorityClass, mainly affects scheduling order and which pods get preempted to make room. QoS class, derived from a pod's requests and limits, mainly affects which pods the kubelet evicts first when a node runs out of memory. One is about getting on, the other about staying on.

What are the three Kubernetes QoS classes?

Guaranteed, where every container sets equal requests and limits, so it is evicted last. Burstable, where requests are set but lower than limits, evicted in the middle. BestEffort, where no requests or limits are set at all, evicted first when a node is under memory pressure. You do not set the class directly, it is inferred.

Does a ResourceQuota stop the noisy neighbour problem completely?

No. A quota caps how much a namespace can request, but it does not decide who wins when the cluster is actually contended. API flooding, node memory pressure and eviction order are governed by APF, pod priority and QoS class, which are separate mechanisms from the quota.

Can one tenant discover another tenant's services via DNS?

By default, yes. Kubernetes DNS resolves any service in any namespace by its fully qualified name, so a pod can look up another tenant's service. The real control is a NetworkPolicy that blocks the actual connection, since a name that resolves but cannot be reached is mostly harmless.

Related reading

• Topic 34: Multi-Tenancy, Soft vs Hard (quotas and LimitRange, the ceiling half)
• Topic 35: Levels of Isolation (the full isolation stack)
• Topic 33: One-Way vs Mutual TLS (pod-to-pod encryption in full)
• Topic 14: Network Policies (the real fix for the DNS leak)
• Browse the whole Kubernetes Journey