Your Scanner Doesn't Delete Anything. You Have To.

Nearly every "we've secured our images" story I've read stops at the same place: a scanner in the pipeline and a dashboard with a number on it. That's the habit I want to argue with, because a scan has never removed a single byte from an image. It reads the package list and tells you what's already there. The only thing that actually shrinks a container's attack surface is deleting things, and that decision is made in the FROM line, not in the scan step.

Here's my claim in one sentence: you don't patch your way to a small image, you delete your way there. I ran the same web server two ways and the difference wasn't a fix, it was an absence.

One line in the Dockerfile did what patching couldn't

I scanned the official Apache httpd image with Trivy, then scanned the Alpine flavour of the same web server. Trivy reads the package database inside the image and matches every installed package against known vulnerability data, so it's a fair way to count what you're carrying.

Two critical and twenty five high findings on one side, an empty table on the other, and I fixed exactly nothing in between. Apache is the same Apache. What changed is that the Debian variant brings a full distro userland along for the ride and the Alpine one brings a much smaller set of packages, so there is simply less installed for a scanner to have an opinion about.

That "0" deserves honesty, though, and I'll come back to it near the end, because zero known findings in the packages that remain is a very different claim from zero risk.

Most of that image was never yours

The reason this feels unfair is that almost none of those 124 findings came from anything I wrote. A Dockerfile that looks this innocent is inheriting an entire operating system:

# my webapp, all four lines of it
FROM httpd
COPY index.html htdocs/index.html

That FROM is not a starting point, it's an adoption. The httpd image is itself built on a Debian slim image, and that Debian image is built on scratch, the genuinely empty starting layer, with a root filesystem tarball unpacked into it. Images stack, and every layer below yours ships with you.

An image built from nothing is properly called a base image, and one you build on top of is a parent image. In practice everyone says "base image" for both, and I will too. The useful bit isn't the vocabulary, it's noticing that by the time you type your first instruction, someone else has already decided what a shell, a package manager and a hundred odd libraries are doing in your production runtime.

Which matters because those extras are attacker tooling. If someone gets code execution inside your web container, curl, wget, apt and a working shell are the difference between a dead end and a foothold. That's the same lesson as an exposed Docker daemon in a different costume: convenience left switched on for the wrong audience.

Three deletions that actually pay

So if deleting is the control, what's worth deleting? Three things, in the order I'd do them.

Move the build tools out. A multi-stage build compiles in one stage and copies only the finished artefact into a second, clean stage. The compiler, the headers, the package cache and your source tree stay behind in a layer that never ships:

# stage 1: everything messy happens here
FROM golang:1.22 AS build
WORKDIR /src
COPY . .
RUN CGO_ENABLED=0 go build -o /app ./cmd/server

# stage 2: what actually ships
FROM gcr.io/distroless/static-debian12
COPY --from=build /app /app
USER 65532:65532
ENTRYPOINT ["/app"]

Drop to a minimal or distroless base. Distroless images carry the application and the runtime libraries it needs, and stop there. No shell, no package manager, no network tools. Google publish the best known set. It's a blunt instrument and that's the point: you can't exploit a bash that isn't installed.

Split dev from prod. The image with the debugger, the shell and the package manager is fine on your laptop. It shouldn't be the artefact you ship. Keeping two images is a bit of extra work in the Makefile and it stops "just for debugging" tooling from quietly becoming production tooling.

An SBOM is a receipt, not a repair

Once you've deleted what you can, you still need to know exactly what's left. That's what an SBOM, a software bill of materials, gives you. It's a machine readable list of every component in the thing you shipped: the libraries, the versions, the licences, where they came from. A receipt for the image.

The reason I keep calling it a receipt is that people talk about SBOMs as if generating one improves anything. It doesn't. Its value shows up on the day a nasty CVE drops and you need to answer "are we affected" in minutes rather than a week of grepping. That's a real and underrated superpower, and it is still not a control.

Two formats dominate, and the choice is mostly about who's asking:

The tooling I used is a two step pairing and it's worth being clear about which does what. Syft builds the inventory. Grype reads an inventory and matches it against vulnerability data:

# build the inventory from an image, or from a source tree
syft <image>:<tag> -o spdx-json > spdx.json
syft /path/to/source -o cyclonedx-json > sbom.cyclonedx.json

# then match that inventory against known vulnerabilities
grype sbom:sbom.cyclonedx.json

Store the SBOM somewhere you'd trust with any other build artefact. An SBOM someone can quietly edit is worse than none, because now you've got a confident inventory that lies. A registry you already control works fine.

A single finding out of one of those scans, trimmed to the interesting fields, shows why the inventory matters:

"vulnerability": { "id": "CVE-2020-11724", "severity": "Medium" },
"artifact":      { "name": "libnginx-mod-http-xslt-filter",
                   "version": "1.10.3-1+deb9u3", "type": "deb" },
"matched-by":    { "matcher": "dpkg-matcher",
                   "search-key": "distro[debian 9] constraint[< 1.10.3-1+deb9u5 (deb)]" },
"locations":     [ { "path": "/var/lib/dpkg/status", "layer-index": 1 } ]

Captured from my scan output, trimmed to the fields that matter. Look at layer-index: 1: the flaw isn't in the application at all, it's in an nginx module pulled in by a Debian 9 base, sitting one layer down. Nobody chose that module. It arrived.

Where mine broke

Generating an SBOM by hand is a one off. The point is to have it happen on every build, so I wired Syft into a GitHub Actions workflow: install the binary, scan the image, upload the result as an artefact.

    - name: Generate SBOM
      run: |
        syft docker.io/kodekloud/webapp-color:latest -o spdx-json > spdx.json

    - name: Upload SBOM Artifact
      uses: actions/upload-artifact@v4
      with:
        name: sbom-report
        path: sbom.xml          # <-- this file does not exist

The job went green and there was nothing to download. Look at the two steps together: Syft writes spdx.json and the upload step asks for sbom.xml. Upload-artifact treats a missing path as a warning rather than a failure by default, so the run reports success, the artefacts section is empty, and you only notice when you go looking for the file. Took me longer than I'd like to admit, mostly because I trusted the green tick instead of reading the step output.

The other one to check before you blame yourself: the workflow triggers on master, and plenty of forks end up on main. If the branch names don't line up, nothing runs at all and there's no error to read.

The best argument against me

The strongest counter to "just delete things" is that deletion has a bill, and I don't want to pretend otherwise.

Distroless means no shell, so kubectl exec into a broken pod gives you nothing. That's genuinely painful at 3am, and the honest answer is ephemeral debug containers, which are a separate thing to learn before you need them rather than during an incident. Alpine swaps glibc for musl, which is mostly fine and occasionally produces a DNS or a native-library surprise that eats an afternoon. And smaller bases usually mean rebuilding more often to pick up fixes, because there's no apt upgrade path inside a running container to fall back on.

Then there's that zero. A scanner reporting no findings means nothing is known to be broken in the packages that are still installed, checked against one vulnerability database on one day. Alpine's package set is smaller, so there's less for the database to match, and the numbers move as tags age. If a post ever tells you a base image is at zero and leaves it there, be suspicious, including this one.

And the limit of my whole argument: none of this touches your own dependencies. A distroless image running an application with a vulnerable library inside the binary is a small, tidy, fully exploitable image. Shrinking the base removes inherited surface. It does nothing about the surface you wrote yourself, and that's the half most incidents actually come through.

What I'd tell someone starting this today

Do them in this order, because the first one is nearly free and the last one is a project.

Start by scanning what you already ship and reading the layer index on the findings, not the total. That single column tells you whether your problem is inherited or self inflicted, and it's the difference between a one line fix and a sprint. Then change the base image on one service and re-scan. If the count collapses the way mine did, you've learned more about your risk in ten minutes than a quarter of triage would have taught you.

After that, multi-stage builds everywhere, because they cost almost nothing and delete the noisiest category of surface. Distroless when you can stomach the debugging trade. SBOMs generated on every build and stored properly, so the next big CVE is a query rather than a fire drill. And if you want the deletion enforced rather than encouraged, that's an admission problem: policy as code can refuse an image from a registry you don't trust, which is the same trust question I looked at when verifying binaries before running them.

What I'm still unsure about is where the honest line sits for a team that isn't me. My lab has no on-call rota, so I get to pick distroless and feel clever about it. I don't know how that decision holds up the first time an engineer is staring at a crash loop with no shell and a customer on the phone. If you've run distroless in production through a bad night, I'd like to hear whether you stuck with it.

Next in the series I'm staying inside the image and looking at what a container is actually allowed to do once it's running.

References

• Docker docs: multi-stage builds
• GoogleContainerTools: distroless images
• Anchore Syft: SBOM generation
• Anchore Grype: vulnerability scanning from an SBOM
• SPDX: the specification
• CycloneDX: the specification
• Trivy: container image scanning

FAQ

What is a distroless container image?

A distroless image ships your application and the runtime libraries it needs, and nothing else. No shell, no package manager, no network tools. Google maintains the best known set. Because there is almost nothing installed, a scanner finds very little, and an attacker who lands inside has no tooling to work with.

Does a smaller base image actually reduce vulnerabilities?

Yes, because most findings come from distro packages you never asked for. When I scanned httpd on Debian I got 124 findings, and the Alpine variant of the same web server reported zero at that time. Nothing was patched. The packages that carried the findings simply were not in the second image.

What is the difference between SPDX and CycloneDX?

Both are standard formats for a software bill of materials. SPDX carries heavier metadata and leans towards licensing, provenance and compliance auditing. CycloneDX is lighter and built around security, so vulnerabilities, dependencies and integrity. If your driver is legal review pick SPDX, if it is patching pick CycloneDX.

What is the difference between Syft and Grype?

Syft builds the inventory. It looks inside an image or a source tree, works out every package and version, and writes an SBOM in SPDX or CycloneDX. Grype consumes that inventory and matches it against vulnerability data. Syft answers what is in here, Grype answers which of it is known to be broken.

Does a multi-stage build make an image more secure or just smaller?

Both, and for the same reason. The compiler, headers, package manager and build cache stay in the first stage and never reach the shipped image. Anything that is not copied forward cannot be scanned, exploited or used as a foothold, so the size drop and the attack surface drop are the same event.

Related reading

• Topic 32: Containers Are Isolated from Each Other, Not from the Kernel (what a small image still shares with the host)
• Topic 16: An Exposed Docker Daemon Is Root on the Host (the build side of the same trust problem)
• Topic 13: Verify the Binary Before You Trust It (supply chain trust, one level up)
• Topic 30: Policy as Code with OPA Gatekeeper (how to actually enforce an image allowlist)
• My Kubernetes pentest notes (the commands, without the argument around them)
• Browse the whole Kubernetes Journey