
In Topic 28 I built an admission webhook to enforce one rule: images must come from an approved registry. To answer that single yes/no question I wrote a Go server, generated a TLS certificate, stuffed the CA bundle into a webhook config and shipped a Deployment to run it. It worked. It also felt like a lot of scaffolding for what is, underneath, an if statement.
Turns out there's a whole project for exactly this, and I'd been quietly rebuilding a worse version of it. That's OPA, and its Kubernetes wrapper, Gatekeeper. Here's what they are, how you'd actually run them, and the handful of things that tripped me up.
What OPA actually is
OPA is a small program whose only job is answering "is this allowed?". You send it a chunk of JSON describing a request, it checks that against rules you've written, and it sends back a decision.
The rules are written in a language called Rego (rhymes with "Lego", and honestly that's the friendliest thing about it at first). You load your Rego into OPA, then you ask OPA questions over an HTTP API. OPA never talks to your database, never sees your users, never enforces anything itself. It just decides. Your application is still the thing that returns a 403.
The moment a rule lives outside your code, you can test it on its own, read it without a debugger, change it without a redeploy, and reuse the same rule from a Python service and a Go service without writing it twice.
So OPA doesn't do authorisation for you. It gives you one place to keep the answer.
The version where the rule lives in your code
Start with something with no checks at all. A tiny Flask app that serves reports:
from flask import Flask, request
app = Flask(__name__)
@app.get("/reports/<name>")
def get_report(name):
return f"report: {name}", 200
Anyone who can reach it gets everything. So you add a check, and the obvious first version is a list of people you trust:
ANALYSTS = {"alice", "priya"}
@app.get("/reports/<name>")
def get_report(name):
user = request.args.get("user", "")
if user not in ANALYSTS:
return "not allowed", 403
return f"report: {name}", 200
To be clear, taking the username from a query string is not authentication, it's a stand-in so we can keep looking at the authorisation bit. In anything real that identity comes from a verified token or a client certificate.
This is fine. It's fine right up until it isn't, and the way it stops being fine is boringly predictable:
- The set grows. Now some people can read their own reports too, and finance can read everything, and contractors can read nothing after 6pm.
- A second service appears, in Go. The rule gets rewritten. It's now subtly different in two places and nobody notices for four months.
- Someone asks "who can read the finance reports?" and the only honest answer is "let me grep three repos".
- You can't test the rule without booting the whole app.
None of that is a bug. It's just what happens when a decision is scattered through the code that acts on it.
Moving the decision out
OPA's pitch is one small change of shape. Instead of the app deciding, the app asks:
Your app already knows who's calling and what they want
Plain JSON: user, action, the thing being touched
Rules you wrote, versioned in git, tested on their own
true or false. Your app still does the enforcing
Hands-on: running OPA on its own
No Kubernetes here yet, deliberately. Everything below runs on one Linux box, and it's the fastest way to get Rego to click before the cluster plumbing gets in the way.
1. Grab the binary and check it runs.
curl -L -o opa https://github.com/open-policy-agent/opa/releases/download/v1.17.0/opa_linux_amd64_static
chmod +x ./opa
./opa version
2. Start it as a server, bound to localhost. The --addr flag matters more than it looks. OPA's API has no authentication or authorisation of its own out of the box, so anything that can reach port 8181 can read your policies and, depending on how you've configured it, write new ones. Bind it to loopback, or put it behind something.
./opa run --server --addr localhost:8181
3. Write the policy. Save this as authz.rego:
package reports.authz
analysts := {"alice", "priya"}
default allow := false
# analysts can read anything
allow if {
input.action == "read"
input.user in analysts
}
# anyone can read a report they own
allow if {
input.action == "read"
input.owner == input.user
}
Reading that top to bottom:
package reports.authzis the namespace. It becomes part of the URL you query, which is a neat trick once you've seen it.default allow := falseis the whole security posture in one line. If no rule below matches, the answer is no. Deny by default, for free.- Each
allow if { ... }block is a separate way to say yes. Every line inside a block must hold, and the blocks are ORed together. So it reads as "allow if all of these, OR allow if all of those". - Nothing says "deny". You describe the yeses and let the default handle everything else. That took me a couple of goes to stop fighting.
if and contains keywords mandatory. Nearly every OPA blog post older than that writes rules as allow { ... } with no if, and that simply won't parse on a current binary. Same story with default allow = false, which is now :=. If you're following an old walkthrough and getting parse errors, that's usually why. opa fmt --write will rewrite most of it for you.4. Load it and ask a question. Policies go in over the API with a PUT:
curl -X PUT --data-binary @authz.rego \
http://localhost:8181/v1/policies/reports
curl -s http://localhost:8181/v1/policies | head -c 200
Now query the allow rule directly. Note how the package path turns straight into the URL:
curl -s http://localhost:8181/v1/data/reports/authz/allow \
-H 'Content-Type: application/json' \
-d '{"input":{"user":"alice","action":"read","owner":"dev-team"}}'
5. Point the app at it. The Flask route loses its rule and gains a phone call:
import requests
OPA = "http://localhost:8181/v1/data/reports/authz/allow"
@app.get("/reports/<name>")
def get_report(name):
payload = {"input": {
"user": request.args.get("user", ""),
"action": "read",
"owner": OWNERS.get(name),
}}
decision = requests.post(OPA, json=payload, timeout=1).json()
if decision.get("result") is not True:
return "not allowed", 403
return f"report: {name}", 200
false, it returns {} with no result key at all. So if not decision["result"] throws a KeyError, and a lazier decision.get("result") == False quietly lets the request through. Check for is not True, and treat a timeout or a connection error as a deny too. An authorisation service that fails open is worse than no authorisation service, because you think you've got one.The bit that sold me: policies you can unit-test
This is the part I didn't expect and now can't unsee. OPA ships a test runner, so your access control rules get tests like any other code. Save this as authz_test.rego:
package reports.authz_test
import data.reports.authz
test_analyst_can_read if {
authz.allow with input as {"user": "alice", "action": "read", "owner": "dev-team"}
}
test_owner_can_read_own if {
authz.allow with input as {"user": "sam", "action": "read", "owner": "sam"}
}
test_stranger_denied if {
not authz.allow with input as {"user": "mallory", "action": "read", "owner": "dev-team"}
}
test_analyst_cannot_delete if {
not authz.allow with input as {"user": "alice", "action": "delete", "owner": "dev-team"}
}
with input as swaps in a fake request for that one test. Any rule starting with test_ gets picked up automatically.
That last test is the one worth copying the habit from. test_analyst_cannot_delete doesn't check that something works, it checks that something still doesn't. Negative tests are how you catch the day someone widens a rule by accident. If you write nothing else, write those.
Same engine, different front door: Gatekeeper
Gatekeeper is OPA packaged as a Kubernetes admission controller. It runs the same Rego engine, but instead of your app asking the questions, the API server does, for every resource anyone creates or updates.
Underneath, it's a validating admission webhook. Exactly the mechanism from Topic 28, except the server, the certificates and the webhook registration are all built and maintained for you. And rather than making you PUT raw Rego at an API, Gatekeeper gives you two custom resources so policy behaves like everything else in the cluster: kubectl apply it, kubectl get it, keep it in git.
Here's the gap it fills. Pod Security Admission ships three fixed profiles and no way to add a fourth. It cannot express "every namespace must have an owner label" or "images only from our registry", because those aren't pod hardening, they're your organisation's rules. That gap is the entire reason Gatekeeper exists.
PSA covers the standard baseline. Gatekeeper covers the rules only you have.
Installing it is one manifest. Check you've got cluster-admin first, because it creates CRDs and a webhook config:
kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper/v3.23.0/deploy/gatekeeper.yaml
kubectl get pods -n gatekeeper-system
ConstraintTemplate: the rule
Gatekeeper splits policy in two, and the split is the good idea in the whole project.
A ConstraintTemplate is the reusable logic, written once. It carries the Rego, plus a schema describing what settings it accepts. Applying one creates a brand new CRD, so your rule becomes a resource type in the cluster.
apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
name: k8srequiredlabels
spec:
crd:
spec:
names:
kind: K8sRequiredLabels
validation:
openAPIV3Schema:
type: object
properties:
labels:
type: array
items:
type: string
targets:
- target: admission.k8s.gatekeeper.sh
rego: |
package k8srequiredlabels
violation[{"msg": msg, "details": {"missing_labels": missing}}] {
provided := {label | input.review.object.metadata.labels[label]}
required := {label | label := input.parameters.labels[_]}
missing := required - provided
count(missing) > 0
msg := sprintf("you must provide labels: %v", [missing])
}
The Rego is doing straightforward set arithmetic, and once you see it that way it stops looking cryptic:
providedcollects the label keys actually on the incoming object. The{x | ...}shape is a set comprehension, "build me a set of every label key that exists here".requiredcollects the labels the constraint asked for, read frominput.parameters.missing := required - providedis set subtraction. What was asked for and isn't there.- If that set has anything in it, the rule produces a
violation, and Gatekeeper turns violations into a rejection.
Notice the flip from the earlier example. In the standalone app policy I wrote allow rules. Gatekeeper wants violation rules instead. It's the same logic pointed the other way: describe what's wrong, and silence means it's fine.
Constraint: where the rule applies
A Constraint is an instance of that template. This is where you say which resources it hits and what the parameters are. Same template, as many constraints as you like:
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredLabels
metadata:
name: ns-must-have-owner
spec:
match:
kinds:
- apiGroups: [""]
kinds: ["Namespace"]
parameters:
labels: ["owner"]
Apply both, then try to create a namespace the lazy way:
msg string straight out of the Rego, which is a good reason to write a helpful one.Now the payoff. Second constraint, same template, completely different rule, and not a line of Rego written:
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredLabels
metadata:
name: prod-pods-need-oncall
spec:
enforcementAction: warn
match:
scope: Namespaced
kinds:
- apiGroups: [""]
kinds: ["Pod"]
namespaces: ["production"]
parameters:
labels: ["oncall"]
Different kind, different namespace, different label, softer enforcement. That's the whole reason the template and the constraint are separate objects: the person who writes Rego and the person who decides where a rule applies don't have to be the same person, or even the same team.
Rolling a policy out without breaking everyone
By default a constraint denies. You almost never want that on day one, because you don't yet know how much of the cluster is already non-compliant. enforcementAction is the dial:
| Value | What happens | When I'd use it |
|---|---|---|
| deny (default) | Request is rejected | Once the warnings have gone quiet |
| warn | Allowed, user sees a warning at apply time | Migration. People find out before you enforce |
| dryrun | Allowed silently, violation recorded in status only | Day one. Measure the blast radius first |
And this is where that audit pod earns its keep. An admission webhook only ever sees new writes, so it has no opinion on the thousand resources already sitting in the cluster. Gatekeeper's audit controller periodically re-checks everything against every constraint and writes what it finds into the constraint's own status:
kubectl get constraints
kubectl get k8srequiredlabels ns-must-have-owner \
-o jsonpath='{.status.totalViolations}{"\n"}'
kubectl get k8srequiredlabels ns-must-have-owner \
-o jsonpath='{.status.violations[*].name}{"\n"}'
Apply a constraint as dryrun, wait for an audit cycle, read totalViolations, and you know exactly what would have broken before anything did. That's the workflow I'd actually follow, and it's the bit most tutorials skip straight past.
The things that cost me time
None of these are in the happy-path docs, which is precisely why they're worth writing down.
1. It's input.review.object, not input.request.object. This is the big one, and it's wrong in a lot of copy-pasted snippets floating about. Kubernetes calls the payload an AdmissionRequest, so input.request feels right, but Gatekeeper wraps it and hands your Rego input.review.
provided comes back empty, every required label looks missing, and the constraint rejects everything that matches it, including resources that are perfectly labelled. You don't get "policy did nothing". You get a policy that blocks correct objects, and an error message insisting a label you can see with your own eyes isn't there.2. Your editor will format the Rego into a version Gatekeeper refuses. Run opa fmt on that ConstraintTemplate and it'll happily rewrite violation[...] { ... } into the modern violation contains ... if { ... } form. Paste that back under spec.targets[].rego and Gatekeeper rejects it, because that field is still Rego v0 only. Gatekeeper 3.19 and later do support v1, but you have to ask for it explicitly with a different block:
targets:
- target: admission.k8s.gatekeeper.sh
rego: |
package k8srequiredlabels
violation[{"msg": msg}] {
# ... v0 syntax only
}targets:
- target: admission.k8s.gatekeeper.sh
code:
- engine: Rego
source:
version: "v1"
rego: |
package k8srequiredlabels
violation contains {"msg": msg} if {
# ... v1 syntax
}3. type: object is not optional. Leave it off openAPIV3Schema and a v1 ConstraintTemplate will be rejected. Plenty of older examples use v1beta1, where it was allowed, so this bites when you copy an old template and bump the apiVersion.
4. namespaces also matches cluster-scoped things. If you write a constraint scoped to namespaces: ["production"] and expect it to only touch namespaced resources, add scope: Namespaced. Otherwise cluster-scoped objects can fall into the match too. Every matcher you add is ANDed with the others, so a constraint doing nothing is usually a match block that's narrower than you think.
5. Deleting a ConstraintTemplate deletes every constraint built from it. The template owns the CRD, and removing a CRD takes its custom resources with it. So one kubectl delete constrainttemplate can silently switch off a dozen rules across the cluster, with no error and nothing obvious in the logs.
What this looks like from the other side
If I've landed a foothold in a cluster and I can list resources, the constraints are one of the first things I'd read, because they're a written confession of what the defenders check for.
kubectl get constrainttemplates
kubectl get constraints
kubectl get constraints -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.enforcementAction}{"\n"}{end}'
Two things jump out of that last command. Anything sitting on dryrun or warn is a policy that looks enforced on a dashboard and stops nothing. And the gaps matter more than the entries: a cluster with a careful registry constraint and nothing about hostPath has told you where to push.
Worth knowing on the defending side too: gatekeeper-system is exempted from its own policies by default, and the main validation webhook ships with a failure policy that lets requests through if Gatekeeper isn't answering. That's a sensible default, since a policy engine that can take the API server down with it is its own kind of outage, but it does mean "Gatekeeper is unhealthy" and "no policy is being enforced" can be the same thirty seconds. The project's Failing Closed page covers the trade-off properly if you want the other behaviour.
The price of admission
Rego is the price of admission, and it's a real price. It's a declarative language that thinks in sets and undefined values, and for the first hour it felt like it was being awkward on purpose. It isn't, and it clicked eventually, but I'd be lying if I called it easy. If a team doesn't want to carry that, Kyverno does much the same job in plain YAML, and Gatekeeper's own policy library means you can get a long way applying templates other people already wrote.
What I'd genuinely take from this one, though, isn't the Kubernetes half. It's opa test. I've never once written a unit test for an authorisation rule in an application, and I've reviewed plenty of apps where nobody else had either. Watching four access control tests go green in a few milliseconds, with no app running, reframed the whole thing for me. The Kubernetes integration is the well-known bit. The testable-rules bit is the one I'll actually use.
Where I'm still unsure: I've only run this in a homelab cluster where nothing is under load. Gatekeeper sits in the write path for every resource, and I have no feel for what that costs on a busy cluster with a few hundred constraints. I also haven't touched the CEL engine at all, which on paper avoids Rego entirely. If you've run either in anger, I'd like to hear how it went.
Next topic, the one I promised last time and got distracted from: how Kubernetes Secrets actually leak in a cluster that's running perfectly well, and why RBAC usually isn't the thing that failed.
References
- Open Policy Agent documentation
- Announcing OPA 1.0 (the Rego v1 syntax change)
- The Rego Playground
- Gatekeeper: installation
- Gatekeeper: constraint templates and constraints
- Gatekeeper: enabling Rego v1 in a ConstraintTemplate
- Gatekeeper: audit
- Gatekeeper policy library
FAQ
What is Open Policy Agent (OPA)?
OPA is a small open-source engine whose only job is answering yes or no. Your application sends it a JSON description of a request, OPA checks that against policies written in a language called Rego, and returns a decision. The rules live outside your application code.
What is the difference between OPA and OPA Gatekeeper?
OPA is the general-purpose engine you can call over HTTP from any application. Gatekeeper is a Kubernetes-specific project that runs that engine as a validating admission webhook, and wraps policies in two custom resources, ConstraintTemplate and Constraint, so you manage them with kubectl.
Do I still need Pod Security Admission if I run Gatekeeper?
Yes, and they work well together. Pod Security Admission is built in, costs nothing to run and covers the standard pod hardening baseline with one namespace label. Gatekeeper covers the organisation-specific rules PSA has no way to express, like required labels or an approved registry list.
Why is my Gatekeeper constraint not blocking anything?
Check three things. The Rego must read input.review.object, not input.request.object. The constraint's enforcementAction must be deny rather than warn or dryrun. And the match block must actually select the resource, since kinds, namespaces and scope are all ANDed together.
Can I write Gatekeeper policies without learning Rego?
Partly. The Gatekeeper policy library ships ready-made ConstraintTemplates for common rules, so you only write the small Constraint that sets the parameters. Recent Gatekeeper versions also support CEL as an alternative engine, and Kyverno is a separate tool that expresses policy as plain YAML.
Related reading
- Topic 28: Building a Custom Admission Webhook (the hand-rolled version of what Gatekeeper does for you)
- Topic 29: Pod Security Policies, and Why They're Gone (the three fixed profiles Gatekeeper fills the gaps around)
- Topic 27: Admission Controllers, RBAC's Blind Spot (why anything checks a spec at all)
- Topic 9: Kubernetes Authorization and RBAC (the cluster's own answer to "who can do what")
- Browse the whole Kubernetes Journey