click the screen · press Enter
← back to blog
Kubernetes Journey · Topic 30

Policy as Code: Rego Is the Price of Admission

Open Policy Agent and OPA Gatekeeper enforcing policy as code on a Kubernetes cluster, with Rego rules deciding whether an admission request is allowed - Kubernetes Journey

In Topic 28 I built an admission webhook to enforce one rule: images must come from an approved registry. To answer that single yes/no question I wrote a Go server, generated a TLS certificate, stuffed the CA bundle into a webhook config and shipped a Deployment to run it. It worked. It also felt like a lot of scaffolding for what is, underneath, an if statement.

Turns out there's a whole project for exactly this, and I'd been quietly rebuilding a worse version of it. That's OPA, and its Kubernetes wrapper, Gatekeeper. Here's what they are, how you'd actually run them, and the handful of things that tripped me up.

Authorisation rules buried in application code compared with policy as code in Open Policy Agent, where one engine answers every service and the rules can be unit-tested and changed without a redeploy
Same decision either way. The difference is where it lives and whether you can test it on its own.

What OPA actually is

OPA is a small program whose only job is answering "is this allowed?". You send it a chunk of JSON describing a request, it checks that against rules you've written, and it sends back a decision.

The rules are written in a language called Rego (rhymes with "Lego", and honestly that's the friendliest thing about it at first). You load your Rego into OPA, then you ask OPA questions over an HTTP API. OPA never talks to your database, never sees your users, never enforces anything itself. It just decides. Your application is still the thing that returns a 403.

The moment a rule lives outside your code, you can test it on its own, read it without a debugger, change it without a redeploy, and reuse the same rule from a Python service and a Go service without writing it twice.

So OPA doesn't do authorisation for you. It gives you one place to keep the answer.

Quick terminology check Authentication is proving who you are: a password, a token, a client certificate. Authorisation is what you're allowed to do once you're through the door. OPA only ever handles the second one. If you're fuzzy on where Kubernetes draws that line, Topic 9 on RBAC covers the cluster's own version.

The version where the rule lives in your code

Start with something with no checks at all. A tiny Flask app that serves reports:

from flask import Flask, request

app = Flask(__name__)

@app.get("/reports/<name>")
def get_report(name):
    return f"report: {name}", 200

Anyone who can reach it gets everything. So you add a check, and the obvious first version is a list of people you trust:

ANALYSTS = {"alice", "priya"}

@app.get("/reports/<name>")
def get_report(name):
    user = request.args.get("user", "")
    if user not in ANALYSTS:
        return "not allowed", 403
    return f"report: {name}", 200

To be clear, taking the username from a query string is not authentication, it's a stand-in so we can keep looking at the authorisation bit. In anything real that identity comes from a verified token or a client certificate.

This is fine. It's fine right up until it isn't, and the way it stops being fine is boringly predictable:

  • The set grows. Now some people can read their own reports too, and finance can read everything, and contractors can read nothing after 6pm.
  • A second service appears, in Go. The rule gets rewritten. It's now subtly different in two places and nobody notices for four months.
  • Someone asks "who can read the finance reports?" and the only honest answer is "let me grep three repos".
  • You can't test the rule without booting the whole app.

None of that is a bug. It's just what happens when a decision is scattered through the code that acts on it.

Moving the decision out

OPA's pitch is one small change of shape. Instead of the app deciding, the app asks:

01
Request arrives

Your app already knows who's calling and what they want

02
App builds an input

Plain JSON: user, action, the thing being touched

03
OPA evaluates Rego

Rules you wrote, versioned in git, tested on their own

04
Decision comes back

true or false. Your app still does the enforcing

The app keeps the enforcement. It just stops owning the reasoning.

Hands-on: running OPA on its own

No Kubernetes here yet, deliberately. Everything below runs on one Linux box, and it's the fastest way to get Rego to click before the cluster plumbing gets in the way.

1. Grab the binary and check it runs.

curl -L -o opa https://github.com/open-policy-agent/opa/releases/download/v1.17.0/opa_linux_amd64_static
chmod +x ./opa
./opa version

2. Start it as a server, bound to localhost. The --addr flag matters more than it looks. OPA's API has no authentication or authorisation of its own out of the box, so anything that can reach port 8181 can read your policies and, depending on how you've configured it, write new ones. Bind it to loopback, or put it behind something.

./opa run --server --addr localhost:8181
./opa run --server --addr localhost:8181
{"addrs":["localhost:8181"],"level":"info","msg":"Initializing server."} waiting for policies...
An empty OPA. It's running, it just has nothing to say yet.

3. Write the policy. Save this as authz.rego:

package reports.authz

analysts := {"alice", "priya"}

default allow := false

# analysts can read anything
allow if {
    input.action == "read"
    input.user in analysts
}

# anyone can read a report they own
allow if {
    input.action == "read"
    input.owner == input.user
}

Reading that top to bottom:

  • package reports.authz is the namespace. It becomes part of the URL you query, which is a neat trick once you've seen it.
  • default allow := false is the whole security posture in one line. If no rule below matches, the answer is no. Deny by default, for free.
  • Each allow if { ... } block is a separate way to say yes. Every line inside a block must hold, and the blocks are ORed together. So it reads as "allow if all of these, OR allow if all of those".
  • Nothing says "deny". You describe the yeses and let the default handle everything else. That took me a couple of goes to stop fighting.
Rego changed, and old tutorials didn't OPA 1.0 landed at the end of 2024 and made the if and contains keywords mandatory. Nearly every OPA blog post older than that writes rules as allow { ... } with no if, and that simply won't parse on a current binary. Same story with default allow = false, which is now :=. If you're following an old walkthrough and getting parse errors, that's usually why. opa fmt --write will rewrite most of it for you.

4. Load it and ask a question. Policies go in over the API with a PUT:

curl -X PUT --data-binary @authz.rego \
  http://localhost:8181/v1/policies/reports

curl -s http://localhost:8181/v1/policies | head -c 200

Now query the allow rule directly. Note how the package path turns straight into the URL:

curl -s http://localhost:8181/v1/data/reports/authz/allow \
  -H 'Content-Type: application/json' \
  -d '{"input":{"user":"alice","action":"read","owner":"dev-team"}}'
three queries, three answers
# alice is an analyst {"result":true} # sam is not, but sam owns this one {"result":true} # mallory is neither {"result":false} == policy is live, nothing was redeployed ==
Three different users, one unchanged policy file, no application involved yet.

5. Point the app at it. The Flask route loses its rule and gains a phone call:

import requests

OPA = "http://localhost:8181/v1/data/reports/authz/allow"

@app.get("/reports/<name>")
def get_report(name):
    payload = {"input": {
        "user": request.args.get("user", ""),
        "action": "read",
        "owner": OWNERS.get(name),
    }}
    decision = requests.post(OPA, json=payload, timeout=1).json()
    if decision.get("result") is not True:
        return "not allowed", 403
    return f"report: {name}", 200
Fail closed, and be fussy about it If a rule is undefined, OPA doesn't return false, it returns {} with no result key at all. So if not decision["result"] throws a KeyError, and a lazier decision.get("result") == False quietly lets the request through. Check for is not True, and treat a timeout or a connection error as a deny too. An authorisation service that fails open is worse than no authorisation service, because you think you've got one.

The bit that sold me: policies you can unit-test

This is the part I didn't expect and now can't unsee. OPA ships a test runner, so your access control rules get tests like any other code. Save this as authz_test.rego:

package reports.authz_test

import data.reports.authz

test_analyst_can_read if {
    authz.allow with input as {"user": "alice", "action": "read", "owner": "dev-team"}
}

test_owner_can_read_own if {
    authz.allow with input as {"user": "sam", "action": "read", "owner": "sam"}
}

test_stranger_denied if {
    not authz.allow with input as {"user": "mallory", "action": "read", "owner": "dev-team"}
}

test_analyst_cannot_delete if {
    not authz.allow with input as {"user": "alice", "action": "delete", "owner": "dev-team"}
}

with input as swaps in a fake request for that one test. Any rule starting with test_ gets picked up automatically.

./opa test . -v
PASS data.reports.authz_test.test_analyst_can_read PASS data.reports.authz_test.test_owner_can_read_own PASS data.reports.authz_test.test_stranger_denied PASS data.reports.authz_test.test_analyst_cannot_delete -------------------------------------------------- PASS: 4/4
Four tests, no app, no database, no HTTP. Stick this in CI and a broken rule stops being a production discovery.

That last test is the one worth copying the habit from. test_analyst_cannot_delete doesn't check that something works, it checks that something still doesn't. Negative tests are how you catch the day someone widens a rule by accident. If you write nothing else, write those.

Try it without installing anything The Rego Playground gives you a policy pane, an input pane and live output in the browser. When a rule isn't matching and I can't see why, I paste it in there and start deleting lines until it goes green. Faster than staring.

Same engine, different front door: Gatekeeper

Gatekeeper is OPA packaged as a Kubernetes admission controller. It runs the same Rego engine, but instead of your app asking the questions, the API server does, for every resource anyone creates or updates.

Underneath, it's a validating admission webhook. Exactly the mechanism from Topic 28, except the server, the certificates and the webhook registration are all built and maintained for you. And rather than making you PUT raw Rego at an API, Gatekeeper gives you two custom resources so policy behaves like everything else in the cluster: kubectl apply it, kubectl get it, keep it in git.

Here's the gap it fills. Pod Security Admission ships three fixed profiles and no way to add a fourth. It cannot express "every namespace must have an owner label" or "images only from our registry", because those aren't pod hardening, they're your organisation's rules. That gap is the entire reason Gatekeeper exists.

PSA covers the standard baseline. Gatekeeper covers the rules only you have.

Installing it is one manifest. Check you've got cluster-admin first, because it creates CRDs and a webhook config:

kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper/v3.23.0/deploy/gatekeeper.yaml

kubectl get pods -n gatekeeper-system
kubectl get pods -n gatekeeper-system
NAME READY STATUS AGE gatekeeper-audit-6d9c7f4b58-4kt2m 1/1 Running 47s gatekeeper-controller-manager-7bb9c5d6f9-h8xqd 1/1 Running 47s gatekeeper-controller-manager-7bb9c5d6f9-m2vrk 1/1 Running 47s gatekeeper-controller-manager-7bb9c5d6f9-zp4nl 1/1 Running 47s
Three controller-manager replicas sit in the admission path. The audit pod is the interesting one, and I'll come back to it.

ConstraintTemplate: the rule

Gatekeeper splits policy in two, and the split is the good idea in the whole project.

A ConstraintTemplate is the reusable logic, written once. It carries the Rego, plus a schema describing what settings it accepts. Applying one creates a brand new CRD, so your rule becomes a resource type in the cluster.

apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: k8srequiredlabels
spec:
  crd:
    spec:
      names:
        kind: K8sRequiredLabels
      validation:
        openAPIV3Schema:
          type: object
          properties:
            labels:
              type: array
              items:
                type: string
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8srequiredlabels

        violation[{"msg": msg, "details": {"missing_labels": missing}}] {
          provided := {label | input.review.object.metadata.labels[label]}
          required := {label | label := input.parameters.labels[_]}
          missing := required - provided
          count(missing) > 0
          msg := sprintf("you must provide labels: %v", [missing])
        }

The Rego is doing straightforward set arithmetic, and once you see it that way it stops looking cryptic:

  • provided collects the label keys actually on the incoming object. The {x | ...} shape is a set comprehension, "build me a set of every label key that exists here".
  • required collects the labels the constraint asked for, read from input.parameters.
  • missing := required - provided is set subtraction. What was asked for and isn't there.
  • If that set has anything in it, the rule produces a violation, and Gatekeeper turns violations into a rejection.

Notice the flip from the earlier example. In the standalone app policy I wrote allow rules. Gatekeeper wants violation rules instead. It's the same logic pointed the other way: describe what's wrong, and silence means it's fine.

Constraint: where the rule applies

A Constraint is an instance of that template. This is where you say which resources it hits and what the parameters are. Same template, as many constraints as you like:

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredLabels
metadata:
  name: ns-must-have-owner
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Namespace"]
  parameters:
    labels: ["owner"]

Apply both, then try to create a namespace the lazy way:

kubectl create ns finance
Error from server ([ns-must-have-owner] you must provide labels: {"owner"}): admission webhook "validation.gatekeeper.sh" denied the request: [ns-must-have-owner] you must provide labels: {"owner"}
kubectl apply -f finance-ns.yaml
# metadata.labels.owner: platform namespace/finance created
The rejection message is the msg string straight out of the Rego, which is a good reason to write a helpful one.

Now the payoff. Second constraint, same template, completely different rule, and not a line of Rego written:

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredLabels
metadata:
  name: prod-pods-need-oncall
spec:
  enforcementAction: warn
  match:
    scope: Namespaced
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
    namespaces: ["production"]
  parameters:
    labels: ["oncall"]

Different kind, different namespace, different label, softer enforcement. That's the whole reason the template and the constraint are separate objects: the person who writes Rego and the person who decides where a rule applies don't have to be the same person, or even the same team.

Rolling a policy out without breaking everyone

By default a constraint denies. You almost never want that on day one, because you don't yet know how much of the cluster is already non-compliant. enforcementAction is the dial:

enforcementAction
ValueWhat happensWhen I'd use it
deny (default)Request is rejectedOnce the warnings have gone quiet
warnAllowed, user sees a warning at apply timeMigration. People find out before you enforce
dryrunAllowed silently, violation recorded in status onlyDay one. Measure the blast radius first
Same rule, three different amounts of pain. Start at the bottom row and work up.

And this is where that audit pod earns its keep. An admission webhook only ever sees new writes, so it has no opinion on the thousand resources already sitting in the cluster. Gatekeeper's audit controller periodically re-checks everything against every constraint and writes what it finds into the constraint's own status:

kubectl get constraints

kubectl get k8srequiredlabels ns-must-have-owner \
  -o jsonpath='{.status.totalViolations}{"\n"}'

kubectl get k8srequiredlabels ns-must-have-owner \
  -o jsonpath='{.status.violations[*].name}{"\n"}'

Apply a constraint as dryrun, wait for an audit cycle, read totalViolations, and you know exactly what would have broken before anything did. That's the workflow I'd actually follow, and it's the bit most tutorials skip straight past.

The things that cost me time

None of these are in the happy-path docs, which is precisely why they're worth writing down.

1. It's input.review.object, not input.request.object. This is the big one, and it's wrong in a lot of copy-pasted snippets floating about. Kubernetes calls the payload an AdmissionRequest, so input.request feels right, but Gatekeeper wraps it and hands your Rego input.review.

Why this failure is nastier than a normal typo A set comprehension over a path that doesn't exist doesn't error, it just produces an empty set. So provided comes back empty, every required label looks missing, and the constraint rejects everything that matches it, including resources that are perfectly labelled. You don't get "policy did nothing". You get a policy that blocks correct objects, and an error message insisting a label you can see with your own eyes isn't there.

2. Your editor will format the Rego into a version Gatekeeper refuses. Run opa fmt on that ConstraintTemplate and it'll happily rewrite violation[...] { ... } into the modern violation contains ... if { ... } form. Paste that back under spec.targets[].rego and Gatekeeper rejects it, because that field is still Rego v0 only. Gatekeeper 3.19 and later do support v1, but you have to ask for it explicitly with a different block:

targets:
  - target: admission.k8s.gatekeeper.sh
    rego: |
      package k8srequiredlabels

      violation[{"msg": msg}] {
        # ... v0 syntax only
      }
targets:
  - target: admission.k8s.gatekeeper.sh
    code:
      - engine: Rego
        source:
          version: "v1"
          rego: |
            package k8srequiredlabels

            violation contains {"msg": msg} if {
              # ... v1 syntax
            }
Two different fields, two different Rego dialects. Mixing them up gives you a template that won't apply and an error that doesn't obviously say why.

3. type: object is not optional. Leave it off openAPIV3Schema and a v1 ConstraintTemplate will be rejected. Plenty of older examples use v1beta1, where it was allowed, so this bites when you copy an old template and bump the apiVersion.

4. namespaces also matches cluster-scoped things. If you write a constraint scoped to namespaces: ["production"] and expect it to only touch namespaced resources, add scope: Namespaced. Otherwise cluster-scoped objects can fall into the match too. Every matcher you add is ANDed with the others, so a constraint doing nothing is usually a match block that's narrower than you think.

5. Deleting a ConstraintTemplate deletes every constraint built from it. The template owns the CRD, and removing a CRD takes its custom resources with it. So one kubectl delete constrainttemplate can silently switch off a dozen rules across the cluster, with no error and nothing obvious in the logs.

What this looks like from the other side

If I've landed a foothold in a cluster and I can list resources, the constraints are one of the first things I'd read, because they're a written confession of what the defenders check for.

kubectl get constrainttemplates
kubectl get constraints
kubectl get constraints -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.enforcementAction}{"\n"}{end}'

Two things jump out of that last command. Anything sitting on dryrun or warn is a policy that looks enforced on a dashboard and stops nothing. And the gaps matter more than the entries: a cluster with a careful registry constraint and nothing about hostPath has told you where to push.

Worth knowing on the defending side too: gatekeeper-system is exempted from its own policies by default, and the main validation webhook ships with a failure policy that lets requests through if Gatekeeper isn't answering. That's a sensible default, since a policy engine that can take the API server down with it is its own kind of outage, but it does mean "Gatekeeper is unhealthy" and "no policy is being enforced" can be the same thirty seconds. The project's Failing Closed page covers the trade-off properly if you want the other behaviour.

The price of admission

Rego is the price of admission, and it's a real price. It's a declarative language that thinks in sets and undefined values, and for the first hour it felt like it was being awkward on purpose. It isn't, and it clicked eventually, but I'd be lying if I called it easy. If a team doesn't want to carry that, Kyverno does much the same job in plain YAML, and Gatekeeper's own policy library means you can get a long way applying templates other people already wrote.

What I'd genuinely take from this one, though, isn't the Kubernetes half. It's opa test. I've never once written a unit test for an authorisation rule in an application, and I've reviewed plenty of apps where nobody else had either. Watching four access control tests go green in a few milliseconds, with no app running, reframed the whole thing for me. The Kubernetes integration is the well-known bit. The testable-rules bit is the one I'll actually use.

Where I'm still unsure: I've only run this in a homelab cluster where nothing is under load. Gatekeeper sits in the write path for every resource, and I have no feel for what that costs on a busy cluster with a few hundred constraints. I also haven't touched the CEL engine at all, which on paper avoids Rego entirely. If you've run either in anger, I'd like to hear how it went.

Next topic, the one I promised last time and got distracted from: how Kubernetes Secrets actually leak in a cluster that's running perfectly well, and why RBAC usually isn't the thing that failed.

References

FAQ

What is Open Policy Agent (OPA)?

OPA is a small open-source engine whose only job is answering yes or no. Your application sends it a JSON description of a request, OPA checks that against policies written in a language called Rego, and returns a decision. The rules live outside your application code.

What is the difference between OPA and OPA Gatekeeper?

OPA is the general-purpose engine you can call over HTTP from any application. Gatekeeper is a Kubernetes-specific project that runs that engine as a validating admission webhook, and wraps policies in two custom resources, ConstraintTemplate and Constraint, so you manage them with kubectl.

Do I still need Pod Security Admission if I run Gatekeeper?

Yes, and they work well together. Pod Security Admission is built in, costs nothing to run and covers the standard pod hardening baseline with one namespace label. Gatekeeper covers the organisation-specific rules PSA has no way to express, like required labels or an approved registry list.

Why is my Gatekeeper constraint not blocking anything?

Check three things. The Rego must read input.review.object, not input.request.object. The constraint's enforcementAction must be deny rather than warn or dryrun. And the match block must actually select the resource, since kinds, namespaces and scope are all ANDed together.

Can I write Gatekeeper policies without learning Rego?

Partly. The Gatekeeper policy library ships ready-made ConstraintTemplates for common rules, so you only write the small Constraint that sets the parameters. Recent Gatekeeper versions also support CEL as an alternative engine, and Kyverno is a separate tool that expresses policy as plain YAML.