Docker no space left on device: the Complete Cleanup Ladder, Safe to Aggressive

Symptom: build or pull dies halfway — out of space

# During build
failed to solve: ... write /var/lib/docker/tmp/...: no space left on device

# During pull
failed to register layer: Error processing tar file(no space left on device)

# A running container writing logs
Error response from daemon: ... no space left on device

One-line diagnosis: the host partition holding /var/lib/docker is full. Docker stacks images with copy-on-write layers — every build, pull, and run piles more into the same directory, and most of that pile is regenerable garbage.

First, see who ate the space

df -h /var/lib/docker        # how much is left on the partition
docker system df             # Docker's own view of four categories

The docker system df output maps to four cleanup dimensions:

| Category | What it is | Cleanability | |---|---|---| | Images | images (incl. dangling layers) | mostly regenerable | | Containers | writable layers of stopped containers | business call | | Local Cache | build cache | regenerable | | Volumes | persistent data | ⚠️ data — leave alone by default |

Adding -v shows per-image and per-container sizes. Locate before you delete — don't start with prune -a.

The cleanup ladder: safe to aggressive, one rung at a time

# Rung 1: stopped containers only (the non-Up ones in docker ps -a)
docker container prune

# Rung 2: dangling images (<none>:<none> layers left by failed builds)
docker image prune

# Rung 3: every image "not referenced by any container" — including old tags
docker image prune -a

# Rung 4: build cache
docker builder prune

# The combo: all of the above (volumes excluded)
docker system prune

Most "disk full" incidents resolve by rung two or three. Each command lists what it will delete and how much it reclaims — read it before pressing y.

The most dangerous switch: --volumes

docker system prune --volumes   # ⚠️ deletes "unreferenced" volumes too

A database container was recreated and its old volume no longer referenced? --volumes deletes it with the data inside. Unless you know exactly what you're doing, leave this flag alone. On production, back up before deleting any volume.

Also never rm -rf /var/lib/docker/* — bypassing Docker's metadata desynchronizes the registry from the actual files, and every subsequent image operation fails in bizarre ways. Always clean up through docker's own commands.

Two root causes of "cleaned, then full again"

1. Unbounded container logs

The json-file log driver has no size limit by default — one log-spamming container can fill the disk:

du -sh /var/lib/docker/containers/*/*-json.log | sort -rh | head -5

The fix: log rotation in /etc/docker/daemon.json (applies to new containers; old ones need recreation):

{
  "log-driver": "json-file",
  "log-opts": { "max-size": "50m", "max-file": "3" }
}

Then sudo systemctl restart docker. A single container can override with docker run --log-opt max-size=50m.

2. The disk isn't full — the inodes are

df -h shows free space but the error persists? Check inodes:

df -i /var/lib/docker
# IUse% 100% ← tiny files (overlay layers, cache) exhausted the inodes

The remedy is the same prune — one leaking source inside an overlay filesystem can generate oceans of tiny files.

Prevention

1. On CI/build machines, add a daily docker system prune -af (pure build machines have no volumes to worry about);
2. Configure log rotation in daemon.json (see above) — the item most people miss;
3. Monitor the partition holding /var/lib/docker, alert at 80%;
4. On production, mount /var/lib/docker on a dedicated partition so a full disk can't take down system logs or databases.

Checklist

1. df -h /var/lib/docker + docker system df — space and breakdown at a glance
2. Climb the ladder: container prune → image prune → builder prune
3. No --volumes unless desperate; never rm -rf /var/lib/docker
4. Full again soon after? Check container logs with du -sh .../*-json.log
5. Space available but still failing? df -i for inodes
6. Is log rotation configured in daemon.json?

Provided by ToolVault. Related tools: Dockerfile Generator, Byte Converter, Linux Cheatsheet. Related reading: Docker daemon unreachable, Port already in use (EADDRINUSE). See the homepage for more developer tools.