click the screen · press Enter
Inside//UnixSingh
← back to blog
Kubernetes Journey · Topic 18

Without Audit Logging, Nobody Can Say Who Deleted That Deployment

By UnixSingh·19 Jul 2026·12 min read
Kubernetes audit logging explained - Kubernetes Journey

Here's a question that quietly ruins someone's week: a production deployment vanishes, and nobody can say who deleted it. Was it a person? A bad script? A compromised token? On a default cluster you genuinely can't tell, because Kubernetes doesn't remember. It does the work and moves on.

Auditing is how you get that memory back. In the last topic we looked at what a node leaks about itself. This one is the opposite direction: how you make the cluster keep a record of every single thing anyone asks it to do. What it is, how to switch it on, and which events I'd actually watch. What, how, why, takeaway.

What auditing actually is

Auditing is the API server writing down every request it receives. Every kubectl command, every controller action, every token that talks to the cluster, becomes a small JSON record called an audit event.

Everything in Kubernetes goes through one front door, the kube-apiserver. Because every request passes through that one place, it's the perfect spot to record them. When auditing is on, the API server checks each request against a policy you wrote, decides how much detail to keep, and appends an event to a log. Think CCTV on the only entrance to a building. You don't need a camera in every room if there's one door and everyone has to use it.

Without it you're blind. With it you can answer "who did what, when, and were they allowed?" after the fact, which is exactly what you need for chasing an incident, passing a compliance check (GDPR, HIPAA, PCI DSS all want this), or just working out why something broke.

What's in one audit event

Every event is just a labelled record of one request. Here's a trimmed one for someone creating a deployment, with the fields that matter:

{
  "kind": "Event",
  "apiVersion": "audit.k8s.io/v1",
  "level": "Metadata",
  "verb": "create",                         // WHAT they did
  "user": { "username": "admin",
            "groups": ["system:masters"] },  // WHO did it
  "sourceIPs": ["192.168.1.10"],            // WHERE from
  "userAgent": "kubectl/v1.30.0",
  "objectRef": { "resource": "deployments",  // WHAT they touched
                 "namespace": "default",
                 "name": "nginx" },
  "responseStatus": { "code": 201 },         // did it work? (201 = created)
  "requestReceivedTimestamp": "2026-07-19T07:50:04Z",   // WHEN
  "annotations": {
    "authorization.k8s.io/decision": "allow",           // were they allowed?
    "authorization.k8s.io/reason": "RBAC: ClusterRoleBinding cluster-admin"
  }
}

Read it left to right and it's a plain English sentence: admin, from 192.168.1.10, created a deployment called nginx in default, it succeeded, and RBAC allowed it. That authorization.k8s.io/reason line is my favourite bit, it tells you which RBAC rule opened the door, which is gold when you're hunting an over-broad binding.

ONE AUDIT EVENT
user (who)
verb (what)
objectRef (which resource)
sourceIPs (where)
timestamps (when)
authz decision (allowed?)
Six questions, answered for every single request. That's the whole value of auditing.

The four levels: how much to write down

You don't record everything at full detail, that would be a firehose and it'd capture things you don't want on disk. So each rule picks a level, which is just how much of the request to keep. There are four, from silent to everything:

None

Log nothing. Used to deliberately drop noisy requests.

Metadata

Who, what, when, which resource. No bodies. The sensible default.

Request

Metadata plus the request body (the object you sent).

RequestResponse

Everything: metadata, request body and the full response.

Detail goes up left to right. So does log volume, and the risk of writing secrets to disk.

More detail isn't better, it's just more. Metadata answers "who touched what" for almost nothing. Save Request and RequestResponse for the handful of actions where you truly need the payload, and never point them at Secrets (more on that trap later).

The audit policy: your rulebook

The policy is a YAML file that lists rules. When a request comes in, the API server walks the rules top to bottom and uses the first one that matches, so order matters. Here's a small, honest starter policy:

apiVersion: audit.k8s.io/v1
kind: Policy
rules:
# 1. don't bother logging noisy read-only system chatter
- level: None
  users: ["system:kube-scheduler", "system:kube-controller-manager"]

# 2. secrets: record THAT they were touched, never the contents
- level: Metadata
  resources:
  - group: ""
    resources: ["secrets", "configmaps"]

# 3. for pods, keep the full request body on writes
- level: Request
  verbs: ["create", "update", "delete"]
  resources:
  - group: ""
    resources: ["pods"]

# 4. everything else: metadata is plenty
- level: Metadata

Notice the shape: drop the noise first, pin secrets to Metadata so their contents never land in the log, get richer detail only where it earns its keep, then a catch-all at the bottom. That last rule matters, without it, anything that didn't match above gets logged at the default and you can end up noisier than you meant.

A quick word on stages Each request passes through stages as it's handled: RequestReceived when it arrives, then ResponseComplete when it's done (plus ResponseStarted for long-running watches, and Panic if it blows up). Most events you care about are ResponseComplete, that's the "it finished, here's what happened" record. You can tell a policy to skip noisy stages like RequestReceived if you want half the volume.

Hands-on: turn it on and catch yourself in the log

Reading about this only gets you so far. Let's stand up a cluster with auditing switched on, do something, and go find it in the log. I'm using kind (Kubernetes in Docker) because it's throwaway, and it lets us pass audit flags to the API server cleanly. Your own lab only.

Step 1: write the policy and a place for logs

mkdir -p audit
cat > audit/policy.yaml <<'EOF'
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: Metadata
  resources:
  - group: ""
    resources: ["secrets"]
- level: RequestResponse
  verbs: ["create", "delete"]
  resources:
  - group: ""
    resources: ["pods"]
- level: Metadata
EOF

Step 2: create a cluster that mounts the policy into the API server

kind runs the control plane inside a container, so we mount our audit folder into the node, then tell the kube-apiserver where the policy and log live. This config does both:

cat > kind-audit.yaml <<'EOF'
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  extraMounts:
  - hostPath: ./audit
    containerPath: /etc/kubernetes/audit
  kubeadmConfigPatches:
  - |
    kind: ClusterConfiguration
    apiServer:
      extraArgs:
        audit-policy-file: /etc/kubernetes/audit/policy.yaml
        audit-log-path: /etc/kubernetes/audit/audit.log
      extraVolumes:
      - name: audit
        hostPath: /etc/kubernetes/audit
        mountPath: /etc/kubernetes/audit
        readOnly: false
        pathType: DirectoryOrCreate
EOF

kind create cluster --name audit-lab --config kind-audit.yaml

Step 3: do something worth logging

kubectl run nginx --image=nginx
kubectl delete pod nginx

Step 4: find yourself in the log (the verify step)

The log is a stream of JSON, one event per line. Pull out the pod create you just did. If a matching line comes back, auditing works:

grep the audit log for your own action
$ grep '"name":"nginx"' audit/audit.log | grep '"verb":"create"' | head -1 | jq '{user:.user.username, verb, res:.objectRef.resource, code:.responseStatus.code}' { "user": "kubernetes-admin", # WHO "verb": "create", # WHAT "res": "pods", "code": 201 # it worked } # there you are - caught in your own CCTV
A returned line is proof the policy loaded and the API server is writing events.
kind delete cluster --name audit-lab   # tidy up

On a real kubeadm cluster the idea is the same, you just add those two flags to the kube-apiserver static pod manifest at /etc/kubernetes/manifests/kube-apiserver.yaml and mount the files in. The API server restarts itself when that file changes, so a typo there takes your control plane down for a minute. Ask me how I know.

Reading the logs like a defender

A log you never look at is just disk usage. The point is to ask it questions. Once events are flowing (ideally into a search tool, not a file on the box), these are the ones I'd hunt for:

  • Anonymous or unexpected users: anything where the username is system:anonymous, or a service account acting well outside its lane.
  • Privilege changes: create/update on clusterrolebindings and rolebindings. A new ClusterRoleBinding to cluster-admin is either a deploy or an attacker, and you want to know which.
  • Secrets access: a subject reading Secrets it's never read before.
  • Exec into pods: the pods/exec subresource is someone getting a shell inside a container. Rare in normal ops, loud when abused.
  • Denials: a burst of authorization.k8s.io/decision: "forbidden" from one identity looks a lot like someone probing what they can reach.
Audit events worth an alert
What you seeVerdictWhy
New binding to cluster-adminALERTPrivilege grant, confirm it was a deploy
pods/exec on a prod podALERTInteractive shell in a container
Spike of forbidden from one userWATCHLooks like access probing
Controller reading its own configmapNORMALRoutine system chatter
Flip "Failures only" to see just the two I'd page on. The rest is context.
The secrets trap This is the mistake I see most: someone sets RequestResponse broadly "to be thorough" and now every Secret and ConfigMap's plaintext is sitting in the audit log. That log is often less protected than the Secrets themselves, so you've just made things worse. Keep Secrets at Metadata. Record that they were touched, never what's inside.
Sensible defaults Start with Metadata as the catch-all so you capture who-did-what cheaply. Drop known-noisy system loops with level: None. Raise to Request/RequestResponse only on the specific writes you care about, and never on Secrets. Ship the log off the node to a store an attacker on that node can't reach. Rotate it (--audit-log-maxage, --audit-log-maxbackup, --audit-log-maxsize) so it doesn't fill the disk.

Not a switch, a decision

What clicked for me writing this is that auditing isn't a security feature you switch on and forget, it's a diary. The API server will happily write it, but it's on you to decide what's worth remembering and then actually read it back. An audit log nobody queries is theatre. The bit I keep going back and forth on is volume, on a busy cluster even Metadata everywhere is a lot, and getting the policy to be quiet about the boring stuff without going quiet about the dangerous stuff is more of an art than I expected. I've got the kind lab solid, but I've not yet wired the log into a proper search stack to write real alerts on it, so that's the next evening. If you run auditing in anger, I'd love to know: what's the one event you always alert on? Mine's a fresh cluster-admin binding.

If this made auditing make sense, come say hi on LinkedIn or the contact page, and tell me what to break down next. More in the Kubernetes Journey.

Further reading

FAQ

What is auditing in Kubernetes?

Auditing is the API server writing a record of every request it handles. Each record, called an audit event, captures who made the call, what they did, which resource they touched, when, from where and whether it was allowed. It is the cluster's tamper-evident history for security, compliance and troubleshooting.

What are the Kubernetes audit policy levels?

There are four: None logs nothing, Metadata logs the who, what and when but no bodies, Request adds the request body, and RequestResponse adds the response body too. You raise the level only for the requests you truly care about, because higher levels mean far more data and can capture secrets.

How do I enable audit logging in Kubernetes?

You write an audit policy file, then point the API server at it with two flags, audit-policy-file and audit-log-path. On a kubeadm cluster you add those flags to the kube-apiserver static pod manifest and mount both the policy file and the log directory into the pod, then let the API server restart.

Does Kubernetes audit logging capture secrets?

It can, and that is a trap. If you set Request or RequestResponse on Secrets or ConfigMaps, the plaintext ends up in the audit log. Keep sensitive resources at Metadata level so you record that they were accessed without writing their contents to disk.

Where are Kubernetes audit logs stored?

By default in a file on the control-plane node at the path you set with audit-log-path. That is fine for a lab, but on a real cluster you ship those logs off the node to a SIEM or log store, because anyone who can wipe the node can wipe your only record of what they did.