docker-troubleshooting

Docker Troubleshooting — 问题排查与调试

Systematic approach to diagnosing and resolving common Docker issues.

When to Use

ALWAYS use this skill when the user mentions:

• "docker 排查", "docker debug", "容器问题"
• "容器启动失败", "exit code", "crash"
• "OOM", "磁盘满", "disk full"
• "网络不通", "network issues"
• "docker daemon", "daemon not running"

Diagnostic Framework

1. Check container state    → docker ps -a, docker inspect
2. Check logs               → docker logs --tail 100
3. Check resources          → docker stats, docker system df
4. Check exit code          → docker inspect --format='{{.State.ExitCode}}'
5. Interactive debug        → docker run --rm -it image sh

Container Startup Failure

Exit Code Reference

Code	Meaning	Action
0	Success	—
1	Application error	Check docker logs
125	Docker daemon error	Check daemon status
126	Command cannot execute	Check permissions
127	Command not found	Check CMD/ENTRYPOINT path
137	SIGKILL (OOM or docker kill)	Check memory limits
139	SIGSEGV (segfault)	Application bug
143	SIGTERM (graceful stop)	Health check timeout?

# Quick diagnosis
docker ps -a | grep Exited
docker inspect --format='{{.State.ExitCode}} {{.State.Error}}' container
docker logs container

Disk Space Exhaustion

# Check disk usage
docker system df

# What's using space?
docker system df -v   # Verbose breakdown

# Clean up
docker system prune -a     # Remove all unused: images, containers, volumes, networks
docker image prune -a      # Unused images only
docker volume prune        # Unused volumes only

# Log rotation (prevent future issues)
docker run --log-opt max-size=10m --log-opt max-file=3 myapp

OOM (Out of Memory)

# Check if container was OOM-killed
docker inspect --format='{{.State.OOMKilled}}' container

# Check memory usage
docker stats --no-stream

# Solution: increase memory limit
docker update --memory=1g --memory-swap=1g container

Network Connectivity

# Check network configuration
docker network inspect bridge
docker exec container ip addr show
docker exec container ping other-container

# DNS check
docker exec container nslookup other-container
docker exec container cat /etc/resolv.conf

# Host networking check
iptables -t nat -L DOCKER -n   # Port forwarding rules

Permission Issues

# Bind mount permission denied
# Solution 1: Match UID
docker run -u $(id -u):$(id -g) -v $PWD:/app myapp

# Solution 2: Fix permissions in entrypoint
# In entrypoint script: chown -R appuser:appgroup /data

# Docker socket permission
sudo usermod -aG docker $USER   # Add user to docker group
newgrp docker                    # Refresh group (or re-login)

Workflow — 排查流程

Step 1: 查看状态: docker ps -a → 确认退出码和 OOMKilled Step 2: 查看日志: docker logs --tail 100 <container> 获取错误信息 Step 3: 深入检查: docker inspect → 检查网络/挂载/资源限制 Step 4: 交互调试: docker run -it --rm --entrypoint sh <image> 进入排查 Step 5: 资源排查: docker stats + docker system df 检查 CPU/内存/磁盘

Gotchas — Common Pitfalls

• docker system prune is destructive: Removes ALL unused objects. Don't run blindly in production. → Recovery: Use docker system df first to preview; add --filter "until=24h" for safe cleanup; never run prune -a without confirmation.
• Logs fill disk silently: JSON-file driver has NO rotation by default. A noisy app can fill the disk in hours. → Recovery: docker run --log-opt max-size=10m --log-opt max-file=3 app; existing: truncate -s 0 $(docker inspect -f '{{.LogPath}}' myapp).
• OOMKilled=TRUE but no error in logs: SIGKILL can't be logged. → Recovery: Check docker inspect -f '{{.State.OOMKilled}}' myapp; increase memory: docker update --memory 512m myapp.
• Exited (0) but service not working: Application started but crashed after health check passed. → Recovery: docker logs --tail 100 myapp; check if app listens on 0.0.0.0 (not 127.0.0.1); verify with docker exec myapp netstat -tlnp.

Boundary — 能力边界（适用与不适用场景）

分类	场景	说明
✅ 能做	容器启动失败排查	退出码分析 + docker logs + inspect
✅ 能做	磁盘/日志/资源问题	docker system df + prune + log rotation
✅ 能做	OOM/CPU 排查	docker stats + --memory 限制 + OOMKilled
⚠️ 需条件	网络包级分析	需 tcpdump/Wireshark + 网络知识
⚠️ 需条件	内核级问题	需 Linux 内核调试技能
❌ 超范围	应用代码 bug 修复	开发者修复代码
❌ 超范围	生产环境部署	使用 docker-production
❌ 超范围	安全漏洞修复	使用 docker-security + docker-scout

When NOT to Use This Skill

❌ Skip	✅ Use Instead
Setting up production	docker-production
Security issues	docker-security
Performance tuning	docker-run (resource limits)
Docker basics	docker-basics

Security & Stability

• docker exec into production containers can expose sensitive data. Prefer log-based debugging.
• docker system prune -a removes ALL unused images and volumes — use with caution in production.
• Inspecting containers with --privileged debug tools can bypass security controls.
• Never expose Docker socket (/var/run/docker.sock) to containers for debugging — it grants root access.
• Rotate debug logs — they may contain sensitive information. Use log drivers with TTL.

📚 官方文档参考

文档	地址
Docker 引擎故障排查	https://docs.docker.com/engine/troubleshoot/
docker logs 命令	https://docs.docker.com/reference/cli/docker/container/logs/
docker inspect	https://docs.docker.com/reference/cli/docker/inspect/
Docker CLI 参考	https://docs.docker.com/reference/cli/docker/
守护进程配置	https://docs.docker.com/reference/cli/dockerd/
常见问题 FAQ	https://docs.docker.com/faq/

🧭 Docker Skills Journey

📍 You are here: docker-troubleshooting — 问题排查

← Previous: docker-production