Kubernetes Secrets Are Not Secret

I promised this one two topics ago and then got distracted by policy engines. Here it is, and it starts with the least surprising demo in Kubernetes: I created a Secret, ran two commands, and had the password back on screen in about a second.

That's not a bug and it isn't news to anyone who has read the docs. It's just that the word "Secret" does a lot of work it hasn't earned, and I've watched people (me included, early on) treat one like a vault. So let's be precise about what a Secret actually buys you, what it definitely doesn't, and the four gaps worth closing.

What a Secret really is

A Secret is an object in the Kubernetes API that holds small key-value pairs, base64 encoded, stored in etcd, and handed to a pod by the kubelet on the node where that pod runs.

You create it, you reference it from a pod, and Kubernetes delivers the values into the container as environment variables or as files. The application reads them at runtime. It never sees them at build time, so the credential is not baked into the image.

Here's the real win, and it's worth saying plainly. The credential leaves your source code, leaves your container image, and leaves your git history. Instead it becomes a separate object with its own name, its own namespace, and its own line in RBAC. Someone allowed to deploy your app is not automatically allowed to read your database password. That separation is the point.

A Secret's job is to move the credential out of the code and into something you can put access controls on. Confidentiality is a thing you add on top, not a thing you get for free.

base64 is a wrapper, not a lock

Here's the demo. A tiny Secret for an imaginary payments service:

kubectl create secret generic payments-db \
  --from-literal=DB_HOST=postgres.data.svc \
  --from-literal=DB_USER=payments_ro \
  --from-literal=DB_PASSWORD='Tr1cky-N0t-S3cret'

Now look at it two ways. describe is coy about the values, which is where a lot of the false comfort comes from:

And getting from that string to the password is one pipe:

kubectl get secret payments-db -o jsonpath='{.data.DB_PASSWORD}' | base64 -d
# Tr1cky-N0t-S3cret

base64 exists so binary data can travel through things that only handle text. It's the same encoding that carries attachments through email. There is no key, no password, nothing to crack. Calling it obfuscation is generous.

Three ways to make one, and the two that bite

The quick way is straight from the command line, which is what I used above. It's fine for a lab and it has one obvious problem: that password is now sitting in your shell history in plain text.

# reads the value from a file, so nothing lands in your shell history
kubectl create secret generic payments-db --from-file=./db_password

# name the key yourself instead of using the filename
kubectl create secret generic ssh-creds \
  --from-file=ssh-privatekey=./id_ed25519 \
  --from-file=ssh-publickey=./id_ed25519.pub

# a whole directory: every file in it becomes a key
kubectl create secret generic bundle --from-file=./creds-dir/

# an env file (KEY=value lines), handy for porting a .env across
kubectl create secret generic payments-db --from-env-file=./payments.env

# mix and match
kubectl create secret generic ssh-creds \
  --from-file=ssh-privatekey=./id_ed25519 \
  --from-literal=passphrase='Tr1cky-N0t-S3cret'

# generate the YAML without creating anything, so you can review it first
kubectl create secret generic payments-db \
  --from-literal=DB_USER=payments_ro \
  --dry-run=client -o yaml

The --from-file variants are worth knowing beyond the tidiness. Anything you pass with --from-literal is in your history, in your terminal scrollback, and quite possibly in a shell session recording somewhere. The file forms keep the value off the command line entirely.

For anything you keep, write the YAML. And here's the bit a lot of older tutorials get wrong: you do not have to base64 encode by hand. The stringData field takes plain text and Kubernetes does the encoding when it stores the object:

apiVersion: v1
kind: Secret
metadata:
  name: payments-db
type: Opaque
stringData:
  DB_HOST: postgres.data.svc
  DB_USER: payments_ro
  DB_PASSWORD: Tr1cky-N0t-S3cret

apiVersion: v1
kind: Secret
metadata:
  name: payments-db
type: Opaque
data:
  DB_HOST: cG9zdGdyZXMuZGF0YS5zdmM=
  DB_USER: cGF5bWVudHNfcm8=
  DB_PASSWORD: VHIxY2t5LU4wdC1TM2NyZXQ=

Neither of those files is safe to commit. Encoding the password changes nothing about that, which is exactly why I'd rather write stringData: it looks as dangerous as it is. A wall of base64 in a repo has fooled more than one reviewer into scrolling past.

Environment variable or mounted file? Not a coin flip

Every tutorial presents these as two equal options. They aren't, and the difference is a security one.

# as environment variables: every key becomes an env var
envFrom:
  - secretRef:
      name: payments-db

# as files: every key becomes a file in the mounted directory
volumes:
  - name: db-creds
    secret:
      secretName: payments-db
      defaultMode: 0400
containers:
  - name: api
    volumeMounts:
      - name: db-creds
        mountPath: /etc/payments/creds
        readOnly: true

I default to volume mounts now, and the reason isn't really the theory. It's that "the environment gets collected" is exactly how credentials end up in a third-party error tracker, sitting in a SaaS dashboard nobody thinks of as part of the cluster. Sentry, a stack-trace logger, a debug endpoint that dumps os.environ when someone is troubleshooting at 2am. None of those read files off disk. All of them read the environment.

Check what actually landed:

Proof: reading a Secret straight out of etcd

Every object in Kubernetes ends up in etcd, the cluster's database. By default a Secret lands there as plain text. Not base64, plain text, because the API server decodes it on the way in. Talking about that is one thing, so let's go and look.

Everything from here runs on a control plane node of a single-node lab cluster. Don't do it to anything you don't own.

Step 1: get etcdctl. It usually isn't installed:

Step 2: read the key. Secrets live under /registry/secrets/<namespace>/<name>. etcd speaks mutual TLS, so you need the certificates kubeadm already put on the node:

ETCDCTL_API=3 etcdctl \
  --cacert /etc/kubernetes/pki/etcd/ca.crt \
  --cert   /etc/kubernetes/pki/etcd/server.crt \
  --key    /etc/kubernetes/pki/etcd/server.key \
  get /registry/secrets/default/payments-db | hexdump -C | head -20

That's the point of the whole exercise. Anyone holding an etcd snapshot, a backup tarball, or a disk image of a control plane node holds every credential in the cluster. No API server involved, no RBAC involved, nothing to bypass.

Turning encryption at rest on, start to finish

Step 3: check whether it's already on. Two ways, and I'd do both because one reads the intended config and one reads what's actually running:

grep -n 'encryption-provider-config' /etc/kubernetes/manifests/kube-apiserver.yaml

ps aux | grep [k]ube-apiserver | tr ' ' '\n' | grep encryption

No output from either means Secrets are going into etcd unencrypted, which is the default on a plain kubeadm cluster.

Step 4: generate a key. The local providers want 32 random bytes, base64 encoded:

head -c 32 /dev/urandom | base64
# 8Qn2rC0mKcYqvVv1zJ4pR7Xb9tLdWfH3eA6sN0uGiPk=

Step 5: write enc.yaml. This is where I have to flag something, because the broken version of this file gets copied around a lot:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - secretbox:
          keys:
            - name: key1
              secret: 8Qn2rC0mKcYqvVv1zJ4pR7Xb9tLdWfH3eA6sN0uGiPk=
      - identity: {}

On the algorithm, since the choice is not obvious and the docs are blunter about it than most tutorials are:

And the obvious weakness of every row except the last: the key sits in a file on the control plane, next to the data it protects. Encryption at rest with a local key defends against a stolen backup. It does not defend against someone who already has the node.

Step 6: put the file somewhere the API server can reach it. The API server is a static pod, so it only sees what's mounted into it:

mkdir -p /etc/kubernetes/enc
mv enc.yaml /etc/kubernetes/enc/
chmod 600 /etc/kubernetes/enc/enc.yaml

# back this up before you touch it, seriously
cp /etc/kubernetes/manifests/kube-apiserver.yaml ~/kube-apiserver.yaml.bak

Step 7: edit the manifest. Three separate additions to /etc/kubernetes/manifests/kube-apiserver.yaml, and missing any one of them leaves you with an API server that won't start:

spec:
  containers:
    - command:
        - kube-apiserver
        # ... existing flags ...
        - --encryption-provider-config=/etc/kubernetes/enc/enc.yaml   # 1
      volumeMounts:
        # ... existing mounts ...
        - name: enc                                                   # 2
          mountPath: /etc/kubernetes/enc
          readOnly: true
  volumes:
    # ... existing volumes ...
    - name: enc                                                       # 3
      hostPath:
        path: /etc/kubernetes/enc
        type: DirectoryOrCreate

Step 8: verify the flag is actually live. Not that you edited the file, that the running process has it:

Step 9: prove it works on something new. Create a second Secret and look at it in etcd:

kubectl create secret generic payments-db-2 --from-literal=key2=topsecret

ETCDCTL_API=3 etcdctl \
  --cacert /etc/kubernetes/pki/etcd/ca.crt \
  --cert   /etc/kubernetes/pki/etcd/server.crt \
  --key    /etc/kubernetes/pki/etcd/server.key \
  get /registry/secrets/default/payments-db-2 | hexdump -C | head

Step 10: re-encrypt everything that already existed. A Secret is only encrypted when it's next written, so read them all and write them straight back:

kubectl get secrets --all-namespaces -o json | kubectl replace -f -

Then run the etcdctl check on the original Secret one more time. If it still comes back readable, the rewrite didn't cover it, and you'd rather find that out now than during an incident. Job's not done until the old key looks like the new one.

The gap encryption doesn't close

Encryption at rest protects the database file. It does nothing at all against someone talking to the API server, because the API server decrypts on the way out. So the actual control is least privilege, and there are two doors into a Secret, not one.

Worth sitting with, that one. If you grant a CI service account create on Deployments in a namespace so it can ship the app, you have also granted it read access to every credential in that namespace. Not through a bug, through the normal, documented mechanism. So when I'm assessing a cluster, "who can create workloads here" is the same question as "who can read the secrets here".

kubectl auth can-i get secrets --namespace payments
kubectl auth can-i create pods --namespace payments

# the fuller picture, as a specific service account
kubectl auth can-i --list \
  --as=system:serviceaccount:payments:deployer -n payments

The RBAC simulator in Topic 10 is handy for reasoning about a Role before you apply it, and the two-doors idea is the thing I'd check it against.

Service account tokens: the correction worth knowing

A lot of study material still shows a Secret of type kubernetes.io/service-account-token appearing automatically whenever you create a ServiceAccount, holding ca.crt, namespace and a long JWT. That behaviour is gone. Kubernetes stopped auto-creating those in version 1.24.

What happens now: a pod gets its token through a projected volume. The kubelet asks the API server for a token that's bound to that pod, tied to a specific audience, and given an expiry. It refreshes it before it runs out. The credential is short-lived and dies with the pod.

kubectl get secrets -A --field-selector type=kubernetes.io/service-account-token

If that command returns anything on a modern cluster, look closely. A token in a Secret is a static credential that does not expire, is not bound to any pod, and works from anywhere in the world that can reach your API server. Sometimes there's a real reason for one. Often it's a leftover from a cluster that was upgraded through 1.24 and nobody swept up. Kubernetes will eventually label unused ones as invalid on its own, but "eventually" defaults to a year.

A projected token expires by itself. A token in a Secret expires when someone remembers. Treat every one you find as a finding until you know why it's there.

What I actually check

Lab clusters and authorised engagements only, obviously. But this is the short list, roughly in the order I'd run it after landing any level of API access.

# 1. what can I read at all
kubectl auth can-i --list

# 2. everything, everywhere, decoded
kubectl get secrets -A -o json | jq -r '
  .items[] | .metadata.namespace + "/" + .metadata.name as $n |
  .data // {} | to_entries[] | "\($n)  \(.key) = \(.value|@base64d)"'

# 3. credentials people typed straight into the pod spec
kubectl get pods -A -o yaml | grep -iE 'password|passwd|token|api[_-]?key' | head -40

# 4. credentials hiding in ConfigMaps, where nothing protects them
kubectl get configmaps -A -o yaml | grep -iE 'password|secret|token' | head -40

# 5. from inside a pod, the token I was given for free
cat /var/run/secrets/kubernetes.io/serviceaccount/token

Number three is the one that keeps paying. People go to the trouble of creating a Secret for the database, then put the third-party API key in a plain env: value in the Deployment, where it sits in the manifest, in git, and in kubectl get deploy -o yaml for anyone with read access. Number four is the same instinct one step worse, because a ConfigMap doesn't even get the separate RBAC treatment.

What I'd actually do about it

1. Fix RBAC first, both doors. Scope get and list on secrets to the service accounts that genuinely need them, and treat pod-creation rights in a namespace as equivalent to read access on that namespace's Secrets. Everything below matters less than this.
2. Turn on encryption at rest, with the providers in the right order. Then force a rewrite so the existing objects are actually covered, and check with etcdctl rather than assuming.
3. Mount as files, not environment variables, with defaultMode: 0400 and readOnly: true. You get rotation without a restart for free.
4. Never commit the manifest. Use stringData so the file looks as sensitive as it is, and keep it out of the repo. If Secrets have to live in git, they need to be sealed or referenced, not encoded.
5. Mark stable Secrets immutable: true. It blocks accidental edits and takes load off the kubelet, since it stops watching for changes. You delete and recreate to rotate, which is a fine trade for something like a TLS certificate.
6. Move the real ones out of the cluster. HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, Google Secret Manager. Something like the External Secrets Operator syncs from those into Kubernetes so your manifests reference a name rather than a value, and you inherit proper rotation and audit logging.

The name is the problem

My honest opinion: the name is the problem. If the resource had been called CredentialRef or something equally boring, nobody would assume it was encrypted, and half the bad practice around it would never have started. It does one useful job well, keeping credentials out of images and code and behind their own RBAC rule, and the whole industry read the label and assumed a vault.

The thing that changed how I look at clusters, though, is the two-doors problem. I spent a while thinking about secret permissions as the control, and then it clicked that pod-creation rights get you there anyway. It reframes RBAC reviews entirely: I no longer read a Role and ask what it can read, I ask what it can run.

Where I'm still unsure: I haven't run External Secrets Operator or Vault's sidecar injector on anything beyond a homelab cluster, so I don't have a real feel for the operational cost. Everyone says "use an external secret manager" and I believe them. I just haven't personally carried the pager for one, and I'd rather say that than pretend.

Next topic, something with a bit more teeth: container images. What's actually inside the one you're running, how a base image drags along dozens of packages nobody chose, and how small you can honestly get.

References

• Kubernetes docs: Secrets
• Kubernetes docs: encrypting confidential data at rest
• Kubernetes docs: using a KMS provider for data encryption
• Kubernetes docs: service accounts and bound tokens
• KEP-2799: reduction of secret-based service account tokens
• kubernetes/kubernetes #73514: the aescbc padding oracle issue
• External Secrets Operator

FAQ

Are Kubernetes Secrets encrypted?

Not by default. A Secret's values are base64 encoded, which anyone can reverse in one command, and by default they are written to etcd as plain text. Encryption at rest is a separate thing you configure on the API server, and it is off unless you turn it on.

What is the difference between a Secret and a ConfigMap in Kubernetes?

Very little, technically. Both store key-value data and both are consumed the same way. A Secret is base64 encoded, can be encrypted at rest, is not written to the node's disk in the same way, and is a separate resource in RBAC. That last point is the one that actually matters.

Should I inject Secrets as environment variables or mounted files?

Mounted files, in most cases. Environment variables are inherited by every child process, show up in /proc/PID/environ, and get swept into crash reports and error trackers. A mounted Secret also updates when you change it, whereas an environment variable is fixed until the pod restarts.

Can someone read Kubernetes Secrets without get permission on secrets?

Yes. Anyone who can create a pod in a namespace can mount any Secret in that namespace and print it. Permission to create workloads is effectively permission to read every Secret those workloads can reach, which is why pod-creation rights deserve as much care as secret-read rights.

Why is there a service account token sitting in a Secret?

Kubernetes stopped auto-creating them in version 1.24. Modern pods get a short-lived projected token that the kubelet rotates. A token still living in a Secret is either a legacy leftover or something created on purpose, and unlike a projected token it does not expire on its own.

Related reading

• Topic 9: Kubernetes Authorization and RBAC (the control that actually protects a Secret)
• Topic 10: Cluster Roles and Bindings (with the can-i simulator for testing a Role)
• Topic 19: Kubernetes Least Privilege (why "what can it run" beats "what can it read")
• Topic 30: OPA and Gatekeeper, Policy as Code (how you'd block a pod that mounts the wrong Secret)
• Browse the whole Kubernetes Journey