containers-internals

Containers Internals

Purpose

Guide agents through Linux container internals: namespaces (clone, unshare, nsenter), cgroups v2 resource limits, overlayfs storage, runc and the OCI runtime spec, seccomp-bpf filtering, Linux capabilities for privilege dropping, and container escape mitigations.

When to Use

• Understanding how Docker/Podman isolate processes under the hood
• Debugging container resource limits (OOM, CPU throttling)
• Writing custom seccomp profiles for sandboxed workloads
• Building minimal containers without Docker
• Investigating container escape vulnerabilities
• Tuning cgroups v2 for Kubernetes pods

Workflow

1. Namespaces

# List namespaces for a process
ls -la /proc/self/ns/
readlink /proc/self/ns/pid
readlink /proc/1234/ns/net

# Enter container namespaces
nsenter -t <pid> -m -u -i -n -p bash

# Unshare namespaces (manual container)
unshare --fork --mount-proc --pid --net --uts --ipc bash

Namespace	Isolates
CLONE_NEWNS (mount)	Mount points, filesystem roots
CLONE_NEWPID	Process IDs
CLONE_NEWNET	Network stack
CLONE_NEWUTS	Hostname
CLONE_NEWIPC	SysV IPC, POSIX message queues
CLONE_NEWUSER	UID/GID mappings
CLONE_NEWCGROUP	cgroup root view

// clone() with namespaces
#define _GNU_SOURCE
#include <sched.h>

int container_init(void *arg) {
    sethostname("container", 9);
    mount("proc", "/proc", "proc", 0, NULL);
    execv("/bin/sh", (char *[]){"/bin/sh", NULL});
    return 1;
}

int stack[1024 * 1024];
clone(container_init, stack + sizeof(stack)/sizeof(int),
      CLONE_NEWPID | CLONE_NEWNS | CLONE_NEWNET | SIGCHLD, NULL);

2. cgroups v2

# Unified hierarchy (cgroup v2)
mount -t cgroup2 none /sys/fs/cgroup

# Create cgroup and set limits
mkdir /sys/fs/cgroup/mycontainer
echo $$ > /sys/fs/cgroup/mycontainer/cgroup.procs

# Memory limit 256MB
echo 256M > /sys/fs/cgroup/mycontainer/memory.max

# CPU weight (relative to siblings, default 100)
echo 50 > /sys/fs/cgroup/mycontainer/cpu.weight

# IO weight
echo default 100 > /sys/fs/cgroup/mycontainer/io.weight

# Check current cgroup
cat /proc/self/cgroup

# OOM events
cat /sys/fs/cgroup/mycontainer/memory.events

3. overlayfs

overlayfs layers
├── lowerdir (read-only image layers)
├── upperdir (container writes)
├── workdir (internal bookkeeping)
└── merged (mount point seen by container)

mount -t overlay overlay \
  -o lowerdir=lower1:lower2,upperdir=upper,workdir=work \
  merged

# Docker stores layers in /var/lib/docker/overlay2/

Copy-on-write: reads from lower, writes go to upper.

4. runc and OCI spec

# Generate default OCI config
mkdir -p mycontainer/rootfs
runc spec

# Edit config.json — namespaces, mounts, process args
# Run container
sudo runc run mycontainer

# List
runc list

config.json key sections:

{
  "ociVersion": "1.0.2",
  "process": {
    "args": ["/bin/sh"],
    "capabilities": { "bounding": ["CAP_NET_BIND_SERVICE"] }
  },
  "linux": {
    "namespaces": [
      {"type": "pid"}, {"type": "network"}, {"type": "mount"}
    ],
    "seccomp": { ... },
    "resources": {
      "memory": { "limit": 268435456 }
    }
  },
  "root": { "path": "rootfs", "readonly": false }
}

5. seccomp-bpf

// libseccomp example — block mount
#include <seccomp.h>

scmp_filter_ctx ctx = seccomp_init(SCMP_ACT_ALLOW);
seccomp_rule_add(ctx, SCMP_ACT_ERRNO(EPERM), SCMP_SYS(mount), 0);
seccomp_rule_add(ctx, SCMP_ACT_ERRNO(EPERM), SCMP_SYS(pivot_root), 0);
seccomp_load(ctx);

# Docker default seccomp profile (JSON)
# https://github.com/moby/moby/blob/master/profiles/seccomp/default.json

# Audit blocked syscalls
# kernel: seccomp log via auditd
ausearch -m SECCOMP

Action	Effect
SCMP_ACT_KILL	Kill process
SCMP_ACT_ERRNO(n)	Return error
SCMP_ACT_TRACE	Notify tracer
SCMP_ACT_ALLOW	Permit syscall

6. Linux capabilities

# Drop all caps except needed
capsh --drop=all --add=net_bind_service -- -c '/app/server'

# File capabilities
setcap cap_net_bind_service+ep /usr/bin/myserver
getcap /usr/bin/myserver

In containers: default Docker drops CAP_SYS_ADMIN, CAP_NET_RAW, etc. Run as non-root with minimal bounding set.

7. User namespace rootless

# Rootless podman/docker maps root in container to unprivileged UID on host
cat /proc/self/uid_map
#          0       1000          1
# container UID 0 → host UID 1000

Rootless limits: cannot mount most filesystems, no CAP_SYS_ADMIN.

8. Escape mitigations

Defense layers
├── User namespace (rootless)
├── seccomp (block dangerous syscalls)
├── AppArmor/SELinux (MAC)
├── Capabilities drop (--cap-drop=ALL)
├── Read-only rootfs
├── no-new-privileges
└── Seccomp + Landlock for filesystem

docker run --read-only --cap-drop=ALL --security-opt=no-new-privileges \
  --security-opt seccomp=default.json myimage

Known escape vectors: mounted docker.sock, privileged mode, kernel CVEs, /proc leaks.

Common Problems

Symptom	Cause	Fix
Container OOMKilled	memory.max exceeded	Raise limit or fix leak
CPU throttled	cpu.max quota low	Adjust cpu.max or weight
Permission denied in container	Capability dropped	Add specific cap or fix app
seccomp kill on start	Missing syscall in profile	strace to find; allow syscall
overlay mount fail	workdir not empty	Clean workdir; check permissions
Rootless mount fail	User namespace limits	Use volume mounts from host

Related Skills

• skills/virtualization/qemu-kvm — VM isolation vs containers
• skills/security/kernel-security — SELinux, AppArmor, seccomp depth
• skills/observability/ebpf — trace container syscalls
• skills/runtimes/binary-hardening — seccomp and capabilities in production
• skills/kernel/kernel-internals — cgroups and namespaces in kernel
• skills/profilers/strace-ltrace — syscall tracing for seccomp tuning