click the screen · press Enter
Inside//UnixSingh
← back to blog
Kubernetes Journey · Topic 24

Every Container Action Goes Through One Kernel Door

By UnixSingh·28 Jul 2026·15 min read
Linux syscalls, strace and Tracee - watching the boundary between user space and the kernel - Kubernetes Journey

This one is a setup post. Next topic I want to stop a container from making a system call it has no business making, that's seccomp, and it's genuinely one of the best bang-for-effort hardening moves there is. But when I sat down to write it I hit a wall: you can't sensibly block a syscall until you can see the ones a program actually makes. Block the wrong one and the app just dies with a cryptic error.

So before the restriction comes the watching. This post is the watching half: what a syscall even is, why it turns into a real security boundary the moment you're running containers, and the two tools I use to see them, strace for a single process, and Aqua's Tracee for whole containers. Part two, where we actually clamp down with seccomp, comes next.

See every syscall: running a container blind means you can't spot a malicious call or write a seccomp policy, but tracing with strace and Tracee (eBPF) lets you see it and then restrict it - one touch makes about 30 syscalls
The whole idea in one frame: you can't restrict what you can't see, so watch the syscalls first.

What a syscall actually is

Your programs are not allowed to touch the hardware. Not the disk, not the network card, not memory beyond their own slice. That sounds like a limitation, but it's the thing that stops one buggy process from taking the whole machine down with it.

The way Linux does this is by splitting the world in two. User space is where your applications live, a touch command, nginx, a Python script, running with limited privileges. Kernel space is the protected area where the kernel itself runs, along with device drivers and the core code, and it does have full control of the machine. Between those two worlds there's exactly one doorway, and that doorway is the system call.

So when a program needs to do anything real, open a file, read some bytes, send a packet, it can't just do it. It has to ask the kernel to do it on its behalf. That request is a syscall: a program in user space asking the kernel to perform a privileged action. The program names the call (like openat) and its arguments, control crosses into kernel space, the kernel does the work and hands back a result. And because that single doorway is the only way anything gets done, it's also the only place you'd need to stand if you wanted to watch, or control, everything a program does.

The only door in
01
User space

Your app: touch, nginx, python. No direct hardware access.

▸
02
Syscall

The request across the line: execve, openat, read, write…

▸
03
Kernel space

Kernel + drivers do the privileged work.

▸
04
Hardware

CPU, memory, disk, network, only the kernel touches it.

Everything a program does that's real crosses this line as a syscall.

A quick example makes it concrete. Run touch /tmp/error.log to make an empty file and it looks like one action. Under the hood it's a little chain of syscalls: execve starts the touch program, then the program asks the kernel (via calls like openat) to create the file, the kernel talks to the filesystem, and the empty file appears. You never see any of that. Which is the whole problem, and why we need a tool that shows it.

Why this becomes a security boundary

On a normal server this is just plumbing. It becomes a security topic the moment you're running containers, and here's the bit that took me a while to properly feel rather than just know.

Every container on a host shares that host's one kernel. A container isn't a little virtual machine with its own kernel; it's just processes on the host, boxed off with namespaces and cgroups, all making syscalls into the same kernel as everything else on the box. So the set of syscalls a container is able to make is, more or less, its reach against the host. A container that can call mount, or load a kernel module, or trace another process, is holding tools that can be turned into a way out of the box.

A container shares the host kernel. So the syscalls it can make aren't a detail, they're the size of the hole it could punch if something inside it goes bad.

That's the thread straight into seccomp: if a web server only ever needs a couple of dozen syscalls, why leave it able to make the ~300 the kernel offers, including the handful that help an attacker escalate or break out? You don't. You block the rest. But, and this is the whole reason for today, you can only draw up that list if you first watch what the thing actually calls. Guess, and you'll break the app.

Watching one process with strace

The classic tool for this is strace. It traces every system call a process makes and prints them as they happen, plus the signals it gets. It's on most distributions already; check with:

which strace
# /usr/bin/strace

The simplest use is to put strace in front of a command. Let's trace our file-creating example. The very first line is the most useful thing to learn to read:

strace touch /tmp/error.log
execve("/usr/bin/touch", ["touch", "/tmp/error.log"], 0x7ffce8f8 /* 23 vars */) = 0 brk(NULL) = 0x55c9a1b2e000 openat(AT_FDCWD, "/etc/ld.so.cache", …) = 3 … openat(AT_FDCWD, "/tmp/error.log", O_CREAT|O_WRONLY…) = 3 == the file exists now ==
Reading the first line: the call, its arguments, and the = 0 result on the right.

Read that first line slowly, because once it clicks the rest of strace is easy. execve is the syscall that executes a program. Its first argument is the full path to the binary (/usr/bin/touch). The second is the argument list, the program name and the file path. The /* 23 vars */ comment means the call inherited 23 environment variables, and the = 0 on the end is the return value: zero means success. You can sanity-check that variable count yourself:

env | wc -l
# 23

A full trace is a wall of text, though. Most of the time I don't want every line, I want the shape of it: which calls, how many, what failed. That's the -c flag, and it's the view I actually use.

strace -c touch /tmp/error.log
% time calls errors syscall ------ -------- ------ ----------- 0.00 6 0 close 0.00 5 0 mmap 0.00 4 0 mprotect 0.00 3 3 access 0.00 3 0 brk 0.00 2 0 fstat 0.00 1 0 execve 0.00 1 0 openat … read, dup2, munmap, arch_prctl, utimensat … ------ -------- ------ ----------- 100.00 30 3 total
One touch: around 30 syscalls, 3 of them harmless failures (access probing for files that aren't there).

That total is the point I want you to sit with. A single touch, the simplest command there is, makes roughly 30 syscalls. A real application does hundreds or thousands a second. That sounds like an argument for giving up, but it's the opposite: run the summary and you get a clean, finite list of the call types a program uses. That list is exactly the raw material a seccomp policy is built from.

Tracing something already running

You won't always be able to launch the process yourself, sometimes it's a daemon that's already up. For that, find its PID and attach with -p. Say I want to peek at etcd, the key-value store behind the cluster:

pidof etcd
# 3596

sudo strace -p 3596
# strace: Process 3596 attached
# futex(0x1ac6be8, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
# futex(0xc000540bc8, FUTEX_WAKE_PRIVATE, 1) = 1
# ^C  (Ctrl+C detaches, it keeps running)
One gotcha that bit me Attaching strace to a busy process slows it down, sometimes a lot, because every syscall now takes a detour through the tracer. It's using ptrace under the hood, the same mechanism a debugger uses. Fine on a lab box; be careful pointing it at a production database. Ctrl+C just detaches the tracer; it doesn't kill the process.

Watching whole containers with Tracee

strace is perfect for one process on a host. But my actual problem is containers, and often I don't have a tidy single command to wrap, I want to know what any new container on the box is calling. That's where container-native tooling comes in, and the one I've been playing with is Tracee from Aqua Security.

Tracee is an open-source runtime tracer that watches syscalls across the whole machine. Instead of ptrace it uses eBPF, a way to run small, safe programs inside the kernel without patching the kernel or loading a custom module. Because the tracing happens in kernel space, it's low-overhead and it can see every process and container at once. That's the leap from "debug one process" to "watch the whole host".

One process, on demand. Uses ptrace, so it adds overhead. You wrap a command or attach to a PID. Brilliant for "what is this program doing right now?" on a host.
Whole host, continuously. Uses eBPF in the kernel, so it's light. Watches every new process or new container at once. Built for the container question: "what is anything on this box calling?"
Not a competition, two tools for two scopes. Reach for whichever the question needs.

The easy way to run it is as a Docker container. When Tracee starts it compiles its eBPF program, so it needs a few things bind-mounted in: the host's kernel headers so it can compile (/lib/modules and /usr/src, read-only), a spot to cache the compiled output (/tmp/tracee), and enough privilege to actually trace. Here it is watching a single command, every syscall an ls makes:

docker run --name tracee --rm --privileged --pid=host \
  -v /lib/modules/:/lib/modules:ro \
  -v /usr/src:/usr/src:ro \
  -v /tmp/tracee:/tmp/tracee \
  aquasec/tracee:0.4.0 --trace comm=ls

Swap that last filter and the same tool answers bigger questions. --trace pid=new follows every new process started on the host. --trace container=new is the one I care about most, it watches only new containers, which is exactly the signal you want when something spins up that shouldn't.

Three ways to point Tracee
01
comm=ls

One command. Every syscall a single ls makes.

▸
02
pid=new

Every new process on the host. Noisy, but complete.

▸
03
container=new

Only new containers. The signal you usually want.

Same tool, three scopes, from one command up to every new container on the box.

To see the last one work, run Tracee with --trace container=new in one terminal, then in another start a throwaway container:

docker run ubuntu echo hi

The container prints hi and exits, and over in Tracee's terminal you get the full list of syscalls that tiny container made just to print one word and stop. Do that a few times with things you trust, and you start to build a feel for what "normal" looks like. Which is the whole game: normal is the baseline, and anything off it is worth a second look.

Try it: what does each syscall do?

The names are the intimidating part at first, execve, ptrace, mount, setuid, so here's a tiny reference you can poke at. Click a call (or type one) and it'll tell you in plain English what it does and how much a defender cares about it. The colour is a rough "how spicy is this call in a container": green is routine, amber is sensitive, red is the kind seccomp profiles usually block.

Try itSyscall explorer: what is this call, and should a container make it?
openat execve socket ptrace mount setuid
Runs entirely in your browser. A taster for the seccomp allow/deny thinking coming in part two.

Reading it like a defender

Once you can list a workload's syscalls, a couple of habits pay off. First, baseline the boring case: trace the app doing its normal job and note the call types. That's your allow-list in waiting. Second, watch for the spicy calls showing up where they've no reason to be, a web app suddenly calling ptrace, mount, unshare or setuid is the kind of thing that says "someone's trying to get out of the box", not "the site served a page". Tracee's container=new view is a genuinely nice place to catch that live.

The short version Programs reach the kernel only through syscalls → containers share the host's one kernel → so a container's syscalls are its blast radius → use strace -c on a process or Tracee on containers to see them → that list is what you'll allow, and everything else is what seccomp denies next topic.

Where this leaves things

So that's the watching half done. A syscall is a program asking the kernel to do the privileged work it can't do itself; user space and kernel space are separated for safety and the syscall is the one door between them; and because every container shares the host kernel, the calls a container can make are the real measure of how much damage it could do. strace shows you one process, strace -c gives you the tidy summary, and Tracee, riding eBPF, scales that up to every container on the box.

The honest bit: I'm still training my eye for what counts as a suspicious call versus just an unfamiliar one. Half of these names I only learned because a trace surfaced them and I went and looked them up, and I'd not pretend I can read a raw Tracee stream and instantly spot trouble, not yet. But that's fine, because the next step doesn't need me to be fluent. It needs a list. Next topic we take the list a workload actually uses and hand the kernel a seccomp profile that blocks everything else, the payoff for all this watching. If you've got a favourite "this syscall in a container always makes me suspicious" tell, I'd love to hear it before I write part two.

FAQ

What is a Linux system call?

A system call is the request a program makes to ask the Linux kernel to do something it can't do on its own, like open a file, read from disk, or open a network socket. Your app runs in restricted user space and can't touch hardware directly, so every privileged action becomes a syscall across that boundary.

What is the difference between user space and kernel space?

User space is where your applications run with limited privileges and no direct hardware access. Kernel space is the protected area where the kernel, drivers and core code run with full control of the machine. Syscalls are the only doorway between the two, which is exactly why they matter for security.

How do I trace the system calls a program makes?

Use strace. Run strace <command> to see every syscall a program makes as it runs, add -c for a summary count, or attach to something already running with strace -p <PID>. It's on most distributions by default and is the fastest way to see what a process is really asking the kernel to do.

Why do syscalls matter for container security?

Every container on a host shares that host's single kernel, and it reaches the kernel through the same syscalls. So the syscalls a container can make are effectively its blast radius against the host. If you can see which calls it actually needs, you can block the rest with seccomp, which is exactly where this leads next.

What is Aqua Tracee and how is it different from strace?

Tracee is an open-source tool from Aqua Security that traces syscalls at runtime using eBPF, running safely inside the kernel with low overhead. strace is perfect for one process on a host; Tracee is built for containers and can watch every new process or new container on the whole machine at once.