A Fresh Node Runs Services You Never Chose

Here's a thing that bugged me the first time I actually looked: a fresh Kubernetes node is already carrying a load of stuff nobody asked for. You spin up a VM, run kubeadm, and the box quietly ends up with services you never chose, kernel modules you'll never use, and ports listening on interfaces you didn't mean to expose. None of it is malicious. It's just default. But every one of those is a door, and a node has enough real doors without leaving the decorative ones open too.

So this topic is about walking the box and shutting what you don't need. Three levers: the services running, the kernel modules loaded, and the ports listening. It's not glamorous, it won't make a demo video, but it's the kind of tidying that quietly removes whole classes of problem. And it pairs neatly with the CIS benchmark work from earlier in the series, most of these checks live in there too.

What "attack surface" even means here

Attack surface is just the sum of everything an attacker can reach and try. Every running service is code that's live and could have a bug. Every listening port is a way in from the network. Every loaded kernel module is more of the kernel that's actually reachable. The bigger that pile, the more chances something in it is exploitable, and the more you have to keep patched. Shrink the pile and you've got less to defend and less that can go wrong. That's the whole idea.

You can't have a vulnerability in software you didn't install, or get hit on a port you closed. Removing things is the cheapest security control there is.

Why a node, specifically

A worker node isn't just any Linux box. It's the host that every pod scheduled onto it runs on top of. The kubelet, the container runtime, the kernel, they're all shared. So if an attacker who's landed in a pod can reach a weak service on the node, or trigger a buggy kernel module, or hit a port that shouldn't be open, they're not attacking one app any more. They're attacking the floor the whole workload stands on, and a node compromise tends to become a much bigger problem fast.

• Nodes drift. Somebody installs a tool to debug an incident at 2am and never removes it. Multiply that across a fleet and every node is subtly different and subtly larger than it should be.
• The pod is right there. Unlike a normal server behind a firewall, a K8s node has untrusted-ish code (your pods) running on it by design. The distance from "attacker in a pod" to "attacker poking the node" is short.
• It compounds. One exposed kubelet or an old docker socket, plus a kernel bug reachable from a pod, and the chain writes itself.

Lever one: services you're not actually using

What a service is: a background program the machine keeps running, managed by systemd (the thing that starts and supervises everything on a modern Linux box). How you look at them: the systemctl command. Why you care: a running service is live code and often a listening port, so a service you don't need is pure downside.

Start by seeing what's actually up. My reflex is to list the running services and read the list like a suspect line-up, anything I can't explain gets investigated.

systemctl list-units --type=service --state=running

That web server is the classic example. Nobody decided a Kubernetes node should run Apache, it came in as a dependency of something, or a base image had it, and now it's sitting there listening. Before you touch it, check what it is and confirm nothing real leans on it.

# what is it, and is it enabled at boot?
systemctl status apache2

# what would removing its package drag out with it?
apt-get -s remove apache2   # -s = simulate, changes nothing

The simulate flag on that second command is the bit I always forget and then remember the hard way. It shows you the full blast radius of a removal without doing anything, so you find out before you break a dependency, not after. Once you're happy it's dead weight, turn it off and take the package with it so the service file goes too.

# stop it now, and stop it coming back at boot
systemctl stop apache2
systemctl disable apache2

# then actually remove it so there's nothing left to start
apt-get remove --purge apache2
apt-get autoremove

Lever two: kernel modules you'll never use

What a kernel module is: a piece of the Linux kernel that can be loaded and unloaded on the fly, drivers, filesystems, network protocols, all live as modules so the kernel doesn't have to ship everything switched on. How they load: sometimes you load one by hand with modprobe, but the kernel also loads them automatically the moment something needs one. Why that matters: "something needs one" can include an unprivileged process. Including code in a pod.

Here's the part that made me sit up. A process doesn't need to be root to make the kernel load a module. If it creates a socket for some obscure network protocol, the kernel helpfully auto-loads that protocol's module to service the request. So a low-privileged process inside a container can reach out and pull a rarely-tested chunk of kernel code into memory, just by asking for it. If that module has ever had a bug (and the obscure network protocols have a history of them), you've just let a pod widen the kernel's attack surface on demand.

See what's currently loaded with lsmod:

lsmod | head

The fix is to blacklist the ones you're certain you'll never use, so the kernel refuses to auto-load them even when asked. You do that with a small config file under /etc/modprobe.d/. Any filename works as long as it ends in .conf.

# create a blacklist file (the name is up to you)
cat <<'EOF' | sudo tee /etc/modprobe.d/harden-blacklist.conf
# rarely-used network protocols we never run in this cluster
blacklist sctp
blacklist dccp
# and stop them loading even if something forces it
install sctp /bin/false
install dccp /bin/false
EOF

Changes to modprobe.d apply to future load attempts, but anything already loaded stays loaded until reboot. So confirm properly: reboot the node, then check the module isn't there and can't be pulled in.

# after a reboot
lsmod | grep -E 'sctp|dccp'      # should print nothing
modprobe sctp                    # should refuse / do nothing
echo $?                          # non-zero = blocked, good

I'll be honest, I went down a rabbit hole here trying to build a "block everything" list and it's the wrong instinct. You don't blacklist hundreds of modules. You blacklist the handful of genuinely useless-to-you, historically-buggy ones (the odd network protocols, some old filesystems), and you leave the rest alone. Over-blacklisting is how you break networking on a node and spend an evening working out why.

Lever three: ports that shouldn't be listening

What a port is: a numbered doorway the OS uses to route network traffic to the right program, port 22 goes to SSH, port 6443 goes to the Kubernetes API server, and so on. How you find them: list what's actually listening. Why you care: a listening port on a reachable interface is the most direct "way in" there is, and nodes love to bind things to 0.0.0.0 (every interface) when they meant to bind to localhost.

List the listeners. ss is the modern tool, netstat still works everywhere if you prefer it.

# modern: sockets that are LISTENing, with the process
sudo ss -tulpn | grep LISTEN

# older but universal
sudo netstat -tulpn | grep LISTEN

Two things to ask about every line. First, what is it, which you can usually tell from the process name ss prints, and cross-check against /etc/services for the well-known ones. Second, which interface is it on, because 127.0.0.1 (localhost only) is a very different risk from 0.0.0.0 or * (reachable from the network). Half the "scary" ports on a node are meant to be localhost and are perfectly safe until someone accidentally binds them wide.

That 8080 line is the one that should make you wince. It's a kubectl proxy someone left running, bound to every interface, with no authentication in front of the API. Anyone who can reach the node reaches the cluster API as whoever started it. That's not a hardening nit, that's a live hole.

Once you know which ports genuinely need to be reachable, and from where, everything else gets closed at the firewall. On most nodes that's a host firewall (nftables, or ufw if you like an easy life) with a default-deny inbound and explicit allows for the ports the node's role actually needs. The kubeadm port reference is the source of truth for what a control plane vs a worker needs, don't guess, look it up.

Hands-on: walk a node in ten minutes

You don't need a cluster for this, any Ubuntu VM will do, or a kind node you can exec into. The point is the routine, not the specific box.

1. Inventory the services. systemctl list-units --type=service --state=running and write down anything you can't explain in one sentence.
2. Kill one you don't need. Install something harmless to practise on (apt-get install -y apache2), confirm it's listening (ss -tlpn | grep :80), then stop, disable and purge it and confirm the port's gone.
3. Blacklist a module. Drop the harden-blacklist.conf above, reboot, and prove modprobe sctp now refuses.
4. Read the ports. sudo ss -tulpn | grep LISTEN and label every line as need-it / localhost-only / close-it.
5. Verify. The check that it worked: the service is gone, the module won't load, and the only 0.0.0.0 listeners left are ones you can name and justify.

What this looks like from both chairs

As a defender, most of this is a one-time tidy plus a check that it stays tidy. Put the service list, the blacklist file and the firewall rules into whatever config management runs your fleet, so a rebuilt node comes up hardened instead of depending on me remembering. And watch for drift: a new listening port or a freshly loaded odd module on a node is worth an alert, because it usually means something changed that nobody meant to change.

As an attacker (the reason I care), the first thing I do on any node I land on is exactly this inventory in reverse: ss -tulpn to see what's listening, lsmod to see what's loaded, systemctl to see what's running. A node that's been through this tidy gives me almost nothing to work with. A node that hasn't gives me a menu. That contrast is the whole point.

Where this leaves the node

None of this is clever. It's just the discipline of not leaving things on that you never turned on on purpose. Remove the services you don't run, blacklist the two or three modules you'll genuinely never use, and firewall down to the ports the node's role actually needs. Do that and a big slice of "what could go wrong on this box" simply stops applying, because the thing that could go wrong isn't there any more.

The bit I'm still working out, same as last topic, is doing this consistently across more than a couple of nodes without hand-editing each one. It clearly wants to live in config management next to the SSH and sudo hardening, so a node is born locked down rather than tidied afterwards. I haven't settled on the cleanest way to express "these modules blacklisted, these ports allowed, nothing else" as code yet, so if you've got a pattern you like for baking node hardening into the image or the provisioning, tell me, that's genuinely the piece I want to nail next.

FAQ

What is a node's attack surface?

It is everything on a machine that an attacker could poke at: running services, listening ports, loaded kernel modules and installed packages. On a Kubernetes node every extra one of those is a door you did not mean to leave open, so the job is to close the ones you do not need.

How do I list and disable a service with systemd?

Run systemctl list-units --type service to see what is running. To turn one off, systemctl stop <name> stops it now and systemctl disable <name> stops it starting at boot. If the package is not needed at all, remove it with your package manager so the service file goes too.

Why blacklist kernel modules on a Kubernetes node?

The kernel can auto-load a rarely used module when a process asks for it, for example by creating a socket for an obscure network protocol. Code inside a pod can trigger that. If the module has a bug, you have just widened the kernel attack surface, so you blacklist the ones you will never use.

Which ports should be open on a Kubernetes node?

Only the ones its role needs. A control plane needs the API server on 6443 and etcd on 2379-2380 (etcd cluster-internal only). Workers need the kubelet on 10250 and the NodePort range. Everything else, and anything bound to 0.0.0.0 that should be local, gets firewalled.

Related reading

• Topic 21: Linux Privilege Escalation with Sudo (what an attacker does once they reach a node)
• Topic 5: CIS Benchmarks & Kube-Bench (the automated version of these node checks)
• Topic 11: Kubelet Security (locking down the most important node-level port)
• Browse the whole Kubernetes Journey