PodSecurity Can't Check Your Registry, So I Built a Webhook That Can

Last topic, PodSecurity stopped a root pod that my RBAC Role happily allowed. That felt like a proper win, right up until I asked myself a follow-up question: could PodSecurity also stop someone pulling an image from a registry nobody on my team has ever heard of? It can't. It doesn't look at image names at all. That gap is what actually pushed me to stop reading about admission webhooks and go build one.

This topic is that webhook, start to finish. Not the theory, the actual Flask app, the actual YAML, and the actual TLS mistake that cost me most of an evening.

First, a mutating built-in I skipped last time: DefaultStorageClass

Quick recap since it's the cleanest example of a mutating controller doing its job invisibly. Kind ships a default StorageClass out of the box:

kubectl get storageclass
# NAME                 PROVISIONER             AGE
# standard (default)   rancher.io/local-path   4h

Create a PVC that doesn't name a storage class at all:

kubectl apply -f - <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: myclaim
spec:
  accessModes: ["ReadWriteOnce"]
  resources:
    requests:
      storage: 500Mi
EOF

That's the whole shape of a mutating controller: it saw a gap in the object, filled it, and moved on. Fine for something as safe as a storage default. Not fine for a rule you actually care about enforcing, which is where webhooks come in.

What the API server actually sends your webhook

Once a request survives every built-in controller, and you've registered a webhook for that resource, the API server POSTs a JSON object called an AdmissionReview to your webhook over HTTPS. It's not complicated once you strip the ceremony out:

Your webhook reads that, decides, and answers with its own JSON: at minimum the same uid and an allowed boolean. Deny it, and the API server rejects the original request with whatever message you give it. That's the whole contract.

Hands-on: a validating webhook that checks image registries

Here's the rule PodSecurity genuinely can't enforce: every image must come from an approved registry. A tiny Flask app is enough to try this in a lab:

from flask import Flask, request, jsonify

ALLOWED_PREFIXES = ("ghcr.io/unixsingh-lab/", "docker.io/library/")
app = Flask(__name__)

@app.route("/validate", methods=["POST"])
def validate():
    review = request.get_json()
    uid = review["request"]["uid"]
    pod = review["request"]["object"]
    bad = [c["image"] for c in pod["spec"]["containers"]
           if not c["image"].startswith(ALLOWED_PREFIXES)]
    if bad:
        return jsonify({"apiVersion": "admission.k8s.io/v1", "kind": "AdmissionReview",
            "response": {"uid": uid, "allowed": False,
                "status": {"message": f"image(s) not from an approved registry: {bad}"}}})
    return jsonify({"apiVersion": "admission.k8s.io/v1", "kind": "AdmissionReview",
        "response": {"uid": uid, "allowed": True}})

Register it with a ValidatingWebhookConfiguration. Two fields here trip people up because a lot of examples floating around online predate them: admissionReviewVersions and sideEffects are both required on a current cluster, leave either out and the API server rejects the configuration itself before your webhook ever gets a chance to run:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: image-registry-check.unixsingh.lab
webhooks:
  - name: image-registry-check.unixsingh.lab
    admissionReviewVersions: ["v1"]
    sideEffects: None
    clientConfig:
      service:
        namespace: webhook-lab
        name: image-check-svc
        path: /validate
      caBundle: <base64 CA certificate>
    rules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["CREATE"]
        resources: ["pods"]
        scope: "Namespaced"
    failurePolicy: Fail

Deploy the Flask app behind that image-check-svc Service, and test both directions:

Hands-on: a mutating webhook that patches in a missing seccomp profile

Now the other direction. Instead of blocking, quietly fix. I want every pod in this cluster to get seccompProfile: RuntimeDefault even in namespaces that aren't running PodSecurity: restricted (restricted enforces it for you; baseline and unlabelled namespaces don't). A mutating webhook can backfill that for the whole cluster in one place:

import base64, json
from flask import Flask, request, jsonify

app = Flask(__name__)

@app.route("/mutate", methods=["POST"])
def mutate():
    review = request.get_json()
    req = review["request"]
    pod = req["object"]
    sec_ctx = pod["spec"].get("securityContext")
    if sec_ctx is None:
        patch = [{"op": "add", "path": "/spec/securityContext",
                  "value": {"seccompProfile": {"type": "RuntimeDefault"}}}]
    elif "seccompProfile" not in sec_ctx:
        patch = [{"op": "add", "path": "/spec/securityContext/seccompProfile",
                  "value": {"type": "RuntimeDefault"}}]
    else:
        patch = []
    encoded = base64.b64encode(json.dumps(patch).encode()).decode()
    return jsonify({"apiVersion": "admission.k8s.io/v1", "kind": "AdmissionReview",
        "response": {"uid": req["uid"], "allowed": True,
                     "patch": encoded, "patchType": "JSONPatch"}})

The response can't just say "here's the new object", it has to be a JSON Patch, base64-encoded, sat inside the JSON response. Genuinely a bit ugly to hand-roll, and exactly the kind of plumbing that makes reaching for Kyverno or OPA Gatekeeper look a lot more appealing than babysitting my own Flask app.

kubectl run plain --image nginx
kubectl get pod plain -o jsonpath='{.spec.securityContext.seccompProfile.type}'
# RuntimeDefault

I never asked for that field. The mutating webhook added it before the pod was ever persisted, same mechanism as DefaultStorageClass at the top of this post, just running code I wrote instead of code Kubernetes shipped.

The TLS gotcha that ate my evening

The API server flatly refuses to call a webhook over plain HTTP. Every call carries a live admission decision, so it insists on HTTPS, and it verifies your certificate against the caBundle you put in the webhook configuration. In a lab that means hand-rolling a self-signed CA:

openssl req -x509 -newkey rsa:2048 -days 365 -nodes \
  -keyout ca.key -out ca.crt -subj "/CN=webhook-lab-ca"
openssl req -newkey rsa:2048 -nodes -keyout tls.key \
  -out tls.csr -subj "/CN=image-check-svc.webhook-lab.svc"
# sign tls.csr with ca.key/ca.crt, mount tls.crt + tls.key into the pod,
# base64 ca.crt into the ValidatingWebhookConfiguration's caBundle

The mistake I actually made: the certificate's CN didn't exactly match the Service's internal DNS name, image-check-svc.webhook-lab.svc, cluster suffix and all. Close enough for a human, not close enough for TLS. Every request died with an x509: certificate is valid for ..., not ... error that told me precisely nothing was wrong with my webhook logic and everything was wrong with my cert. I'm not pretending this is solved properly, I hand-rolled the cert for the lab; a real cluster wants cert-manager or similar doing the rotation, and that's still on my list to actually wire up.

failurePolicy: the choice that decides what happens when your webhook falls over

One field in that YAML deserves its own section, because getting it backwards is how a webhook takes down a cluster or quietly does nothing:

Neither is universally correct. A security control you'd genuinely rather block on should fail closed (Fail). A convenience mutation that isn't load-bearing shouldn't be allowed to take pods down with it (Ignore). Pick per webhook, deliberately, not as a global default you forgot you set.

Where this leaves things

The built-ins from last topic, PodSecurity, NamespaceLifecycle, DefaultStorageClass, cover the rules Kubernetes' own authors anticipated. A webhook covers the rule that's specific to your organisation, your registries, your naming conventions, the stuff no generic control could ever know to check. The cost is real: you're now running a service that TLS, cert rotation and failurePolicy all depend on getting right, and a bug in your Flask app is now a bug in your cluster's admission path.

Honest bit: hand-writing these two webhooks was a genuinely useful way to understand the mechanics, AdmissionReview, JSON Patch, the TLS handshake, but I wouldn't run my own Flask app as a production gate. That's precisely the job Kyverno and OPA Gatekeeper exist to do properly, policy as YAML or Rego instead of a webhook you maintain yourself. Next thing on my list is actually standing one of those up and seeing how much of this plumbing it takes off my hands.

FAQ

What's the difference between a built-in admission controller and a custom admission webhook?

A built-in controller like PodSecurity ships inside the API server binary and enforces a fixed set of rules. A webhook is a server you write and run yourself; the API server calls out to it over HTTPS for every matching request, so it can enforce any rule you can code, like which registry an image must come from.

What does the AdmissionReview object contain?

It's the JSON the API server POSTs to your webhook: a uid to match the response to, which operation is happening (create, update, delete), the full object being submitted, and the userInfo of whoever sent the request. Your webhook reads this and decides allow, deny, or patch.

How does a mutating webhook actually change a Kubernetes object?

It returns a JSON Patch, a small list of add/replace/remove operations, base64-encoded inside its response. The API server applies that patch to the object before saving it. It can't just send back a modified copy, the patch format is mandatory.

Why does an admission webhook need TLS?

The API server refuses to call a webhook over plain HTTP, full stop. Every call carries a live admission decision, so Kubernetes insists on HTTPS with a certificate the caBundle in your webhook configuration can verify. No valid TLS, no calls, your rule silently never runs.

What does failurePolicy: Fail vs Ignore actually control?

It decides what happens if your webhook is unreachable or times out. Fail blocks the request, safe but it can take your cluster down with it. Ignore lets the request through unchecked, keeping things running but silently skipping your rule. Pick per webhook, not as a blanket default.

Related reading

• Topic 27: Admission Controllers (RBAC's Blind Spot) (the built-ins this post extends)
• Topic 25: Seccomp in Kubernetes (what the mutating webhook is actually injecting)
• Topic 16: Securing the Docker Daemon (more on why image provenance matters)
• Browse the whole Kubernetes Journey