click the screen · press Enter
Inside//UnixSingh
← back to blog
Kubernetes Journey · Topic 3

Kubernetes Nodes and Workloads: the Groundwork Security Testing Needs

By UnixSingh·5 Jul 2026·14 min read
Topic 3: Nodes, Workloads & the Foundations of Security Testing - Kubernetes Journey

It's been a while since I last posted, but I'm back with Topic 3 of learning Kubernetes. In this post I'm shifting focus towards the DevOps side of Kubernetes. From a security testing perspective, it's hard to identify real vulnerabilities unless we understand how Kubernetes actually works in practice. Security issues rarely exist in isolation - most come from the way things are deployed, configured and operated.

We don't need to become full-time DevOps engineers, but solid hands-on knowledge is essential. Understanding how workloads run, where configurations live, and how components interact helps us spot the misconfigurations attackers exploit in real environments. This post is practical, not theory-heavy.

Nodes

Nodes are the worker machines in a Kubernetes cluster - this is where your applications actually run. Each node provides CPU and memory to run workloads, networking for pod-to-pod and service communication, and a runtime environment managed by Kubernetes. Pods never run directly on the cluster; they always run on nodes. No worker nodes, no scheduled workloads.

Checking nodes in a cluster

A common first step is understanding how many nodes exist and their state. To list all nodes with extended details:

kubectl get nodes -o wide

This shows internal IPs, operating system and container runtime. For a simpler view:

kubectl get nodes

To explicitly check the kubelet version running on each node:

kubectl get nodes -o custom-columns=NAME:.metadata.name,KUBELET_VERSION:.status.nodeInfo.kubeletVersion

This is especially useful during security reviews, as outdated kubelet versions can introduce known vulnerabilities. For detailed information about a specific node:

kubectl describe node <node-name>

This reveals node conditions, resource capacity and allocation, running pods, labels and taints. Misconfigured nodes, excessive privileges or exposed resources often become the entry point for attackers.

Kubernetes version & node OS details

After identifying how many nodes exist, the next step is understanding what versions run underneath. Vulnerabilities often target specific Kubernetes, kubelet or kernel versions rather than the application itself.

The control plane defines the cluster version, but each node runs its own kubelet - if versions drift or become outdated they introduce gaps. Kubernetes also runs on top of an operating system, and that OS is part of the attack surface. To extract OS, kernel and architecture details for all nodes:

kubectl get nodes -o json | jq -r '.items[].status.nodeInfo | "\(.machineID) \(.osImage) \(.kernelVersion) \(.architecture)"'

For targeted inspection of a specific node:

kubectl describe node <node-name> | grep -i "OS Image"
kubectl describe node <node-name> | grep -i "Kernel Version"

For a cleaner, table-style output across all nodes:

kubectl get nodes -o custom-columns=NAME:.metadata.name,OS_IMAGE:.status.nodeInfo.osImage,KERNEL:.status.nodeInfo.kernelVersion,ARCH:.status.nodeInfo.architecture
Why this matters for testing Outdated kernels may expose privilege escalation paths, unsupported OS images miss patches, and mixed node versions hint at weak operational hygiene. Mapping known CVEs to real infrastructure beats guessing.

Pods

A Pod is the smallest deployable unit - one or more tightly coupled containers that share the same network namespace and storage volumes, and are scheduled together. Security-wise this matters because containers in a pod trust each other by default: compromise one, and the others are usually affected too.

Checking how many pods exist

kubectl get pods --all-namespaces

If you only want the total number of pods:

kubectl get pods --all-namespaces --no-headers | wc -l

A high number of unexpected pods can indicate misconfigurations, leftover test workloads, or even compromised deployments.

Creating and verifying a pod

kubectl run nginx-pod --image=nginx
kubectl get pods

Health and restarts

A high restart count usually means something is wrong - application crashes, resource limits being hit, misconfigured liveness/readiness probes, or permission issues. Ignoring frequent restarts often leads to unstable workloads and hidden security risks.

Inspecting pod details

kubectl describe pod <pod-name>

The Events section at the bottom is especially valuable - image pull failures, crash loops, scheduling problems, and permission or volume mount errors. These frequently expose misconfigurations attackers may exploit.

Deleting pods

kubectl delete -f <manifest-file>
kubectl delete pod nginx

Deleting pods is normal, but unexpected deletions or constant recreation can indicate unstable deployments or malicious activity.

Namespaces

A namespace is a logical partition within a cluster - it groups and isolates resources such as pods, services and configs while still running on the same underlying cluster. They're used for environment separation (dev/staging/prod), multi-team isolation, and applying security controls and resource limits.

Namespaces are often mistaken for strong isolation. In reality they're organisational boundaries unless combined with proper RBAC, network policies and admission controls.

Listing and inspecting

kubectl get namespace
kubectl get pods -n <namespace-name>
kubectl get pods -n kube-system

The kube-system namespace contains core components and system services - especially sensitive, and should be tightly restricted.

Creating a namespace and scoped pods

kubectl create ns demo-singh

To create a pod inside a specific namespace, define it in the manifest metadata:

apiVersion: v1
kind: Pod
metadata:
  name: nginx
  namespace: demo-singh
spec:
  containers:
  - name: nginx
    image: nginx:1.14.2
    ports:
    - containerPort: 80
kubectl apply -f podmanifest.yml
kubectl get pod -n demo-singh
Security note Namespaces alone don't prevent lateral movement. If RBAC is too permissive or network policies are missing, an attacker who gains access to one namespace may still move across others.

ResourceQuota configuration

A ResourceQuota controls how much CPU and memory can be consumed within a namespace. Missing or weak quotas are a common operational and security issue - an attacker or misconfigured app can easily cause denial-of-service if usage isn't restricted.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: tiny-rq
spec:
  hard:
    cpu: "1"
    memory: 1Gi
kubectl apply -f resource_quota.yaml -n demo-singh
kubectl describe ns demo-singh

From a testing perspective, ResourceQuotas reduce the impact of compromised workloads, prevent cluster-wide resource exhaustion, and limit damage from runaway or malicious containers - an important defensive layer alongside limits, RBAC and monitoring.

Resource monitoring

Without visibility into CPU and memory usage, it's hard to detect performance issues, misconfigurations or early signs of abuse. Kubernetes relies on the Metrics Server to expose basic usage data.

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

Confirm the Metrics API is available, then view usage:

kubectl get apiservices | grep metrics
kubectl top nodes
kubectl top pods -A
Why monitoring matters for security Resource metrics can reveal crypto-mining or abuse workloads, pods consuming unexpected CPU/memory, early denial-of-service conditions, and poorly defined limits attackers could exploit.

Requests and limits

Requests and limits define how much CPU and memory a container is guaranteed and how much it may consume at most. In many real incidents, missing or incorrect limits let a single pod degrade or crash an entire node.

apiVersion: v1
kind: Pod
metadata:
  name: demo-pod
spec:
  containers:
    - name: nginx
      image: nginx:1.14.2
      resources:
        requests:
          cpu: 250m
          memory: "65M"
        limits:
          cpu: 500m
          memory: "130M"

Requests are the minimum a container is guaranteed; limits are the hard maximum. Exceed the memory limit and the container is killed and restarted; exceed the CPU limit and it's throttled. Pods without limits can exhaust node resources, and compromised containers can be abused for things like crypto-mining - so limits are a defensive measure that caps the blast radius.

Probes

Probes are health checks that tell Kubernetes whether a container is running correctly, ready to receive traffic, or has started properly. Poorly configured or missing probes are a common cause of unstable workloads and can hide real issues during assessments.

  • Liveness probe - restarts the container if it becomes unhealthy.
  • Readiness probe - removes the pod from service traffic when it's not ready.
  • Startup probe - prevents slow-starting apps from being restarted too early.
apiVersion: v1
kind: Pod
metadata:
  name: sise-lp
spec:
  containers:
    - name: sise
      image: mhausenblas/simpleservice:0.5.0
      ports:
        - containerPort: 9876
      livenessProbe:
        initialDelaySeconds: 2
        periodSeconds: 5
        timeoutSeconds: 1
        failureThreshold: 3
        httpGet:
          path: /health
          port: 9876

From a security perspective, probes surface real problems instead of hiding them: crashing containers can indicate exploitation attempts, missing probes let broken services keep receiving traffic, and restart loops may signal resource abuse. They're not a control by themselves, but they're valuable signals during incident response.

What this sets up

This post covered the practical side of Kubernetes from a developer and operational perspective. Understanding how nodes, pods, namespaces, resource management and health checks work is essential before diving deeper into security testing. In the upcoming posts we'll move into real-world security issues, common misconfigurations and practical remediation techniques.

If you have any questions or would like to connect, feel free to reach out on LinkedIn. Thanks for reading - see you in the next post.

FAQ

What is the difference between a pod and a node?

A node is a machine, physical or virtual, that runs your workloads. A pod is the smallest unit Kubernetes schedules: one or more containers that share a network and storage. Pods run on nodes.

What are namespaces for?

Namespaces split one cluster into logical sections so teams or environments do not collide. They also give you a boundary for quotas and access control, which is where security testing starts to care about them.

What are liveness and readiness probes?

They are health checks. A liveness probe tells Kubernetes when to restart a stuck container; a readiness probe tells it when a pod is ready for traffic. Get them wrong and you either drop requests or hide crashes.