Docker no space left on device: the Complete Cleanup Ladder, Safe to Aggressive

Symptom: build or pull dies halfway — out of space

# During build
failed to solve: ... write /var/lib/docker/tmp/...: no space left on device

# During pull
failed to register layer: Error processing tar file(no space left on device)

# A running container writing logs
Error response from daemon: ... no space left on device

One-line diagnosis: the host partition holding /var/lib/docker is full. Docker stacks images with copy-on-write layers — every build, pull, and run piles more into the same directory, and most of that pile is regenerable garbage.

The same failure can masquerade as other symptoms: a container that OOMs and never comes back, a compose service stuck at Creating, a CI box that was green yesterday and red today. Do not buy a larger disk first — measure who ate the space.

First, see who ate the space

df -h /var/lib/docker        # how much is left on the partition
docker system df             # Docker's own view of four categories
docker system df -v          # per-image / container / volume breakdown

The docker system df output maps to four cleanup dimensions:

Adding -v shows per-image and per-container sizes. Locate before you delete — don't start with prune -a.

When two engineers disagree about what filled the disk, these commands end the debate:

# inodes vs bytes — "disk full" with free bytes is often inode exhaustion
df -i /var/lib/docker

# honest directory map (-x stays on one filesystem)
du -xh --max-depth=1 /var/lib/docker | sort -rh | head

The cleanup ladder: safe to aggressive, one rung at a time

# Rung 1: stopped containers only (the non-Up ones in docker ps -a)
docker container prune

# Rung 2: dangling images (<none>:<none> layers left by failed builds)
docker image prune

# Rung 3: every image "not referenced by any container" — including old tags
docker image prune -a

# Rung 4: build cache
docker builder prune

# The combo: all of the above (volumes excluded)
docker system prune

Most "disk full" incidents resolve by rung two or three. Each command lists what it will delete and how much it reclaims — read it before pressing y.

When you need a larger reclaim and you are sure old tags are disposable:

docker system prune -a          # every image not used by a container
docker builder prune -af        # force-clear BuildKit cache

Pure CI builders can schedule -af daily. Laptops and production nodes should not default to -a, or yesterday's base image becomes today's re-download.

The most dangerous switch: --volumes

docker system prune --volumes   # ⚠️ deletes "unreferenced" volumes too

A database container was recreated and its old volume no longer referenced? --volumes deletes it with the data inside. Unless you know exactly what you're doing, leave this flag alone. On production, back up before deleting any volume, or at least:

docker volume ls
docker volume inspect <name>    # check Mountpoint before you prune

Also never rm -rf /var/lib/docker/* — bypassing Docker's metadata desynchronizes the registry from the actual files, and every subsequent image operation fails in bizarre ways. Always clean up through docker's own commands.

Three root causes of "cleaned, then full again"

1. Unbounded container logs

The json-file log driver has no size limit by default — one log-spamming container can fill the disk:

du -sh /var/lib/docker/containers/*/*-json.log | sort -rh | head -5

The lasting fix: log rotation in /etc/docker/daemon.json (applies to new containers; old ones need recreation):

{
  "log-driver": "json-file",
  "log-opts": { "max-size": "50m", "max-file": "3" }
}

Then sudo systemctl restart docker. A single container can override with docker run --log-opt max-size=50m. For logs that already exploded, and only when the content is disposable:

truncate -s 0 /var/lib/docker/containers/<id>/<id>-json.log

2. The disk isn't full — the inodes are

df -h shows free space but the error persists? Check inodes:

df -i /var/lib/docker
# IUse% 100% ← tiny files (overlay layers, cache) exhausted the inodes

The remedy is the same prune — one leaking source inside an overlay filesystem can generate oceans of tiny files. If prune does not free inodes, use du -xh to find the outlier directory (bad hostPath mounts and leftover temp layers are common).

3. BuildKit cache and multi-arch layers

With BuildKit on, docker builder prune and docker image prune do not clear the same pile. Multi-arch manifests (--platform linux/amd64,linux/arm64) leave multiple layer sets on disk; weeks of buildx build without pruning often shows "Images look fine, Cache ate tens of GB". Ending each CI job with docker builder prune -af is cheaper than buying another disk.

overlay2 and rootfs: do not stare only at the Images line

The Images row in docker system df is not the whole of /var/lib/docker. Writable layers (containers that wrote huge temp trees), leftover tmp, and storage-driver metadata show up in du but not always as RECLAIMABLE Images. Keep a fixed order:

1. df -h + df -i — bytes or inodes
2. docker system df -v — what Docker itself accounts for
3. du -xh --max-depth=1 /var/lib/docker — drill into overlay2 / containers / buildkit when the books disagree
4. Only then pick a prune rung — never prune first and guess later

The space-forensics one-liners that settle arguments

• docker system df -v — per-object breakdown; the summary view hides the offending volume
• df -i alongside df -h — inodes can exhaust while bytes still show 40% free
• du -xh --max-depth=1 /var/lib/docker | sort -rh | head — the honest size map across one filesystem

The inode case deserves emphasis because it presents identically to byte exhaustion, survives careless cleanups that never delete the small-file source, and is fixed by removing that source — not by random prune -a alone.

Prevention

1. On CI/build machines, add a daily docker system prune -af (pure build machines have no volumes to worry about); BuildKit hosts should also run docker builder prune -af;
2. Configure log rotation in daemon.json (see above) — the item most people miss;
3. Monitor the partition holding /var/lib/docker, alert at 80%; monitor inode use too;
4. On production, mount /var/lib/docker on a dedicated partition so a full disk cannot take down system logs or databases;
5. On laptops, prune unused tags regularly instead of waiting for a once-a-quarter emergency prune -a.

When writing Dockerfiles, keep layers and build context lean — unrelated files in the context inflate cache. Use the Dockerfile Generator to sanity-check base images and multi-stage layout, then lock runtime flags with Run to Compose. When estimating sizes, the Byte Converter avoids mixing GiB and GB.

Checklist

1. df -h /var/lib/docker + docker system df — space and breakdown at a glance
2. Climb the ladder: container prune → image prune → builder prune
3. No --volumes unless desperate; never rm -rf /var/lib/docker
4. Full again soon after? Check container logs with du -sh .../*-json.log
5. Space available but still failing? df -i for inodes
6. Is log rotation configured in daemon.json? Was BuildKit cache cleared?
7. When du and system df disagree, drill into overlay2 / buildkit

Triage notes

The cleanup commands form a ladder rather than jumping straight to system prune -a, because most full disks resolve by rung two or three (stopped containers, dangling images), and each rung lists exactly what it would delete — locate before you delete. The "cleaned but fills up again" section exists for the inode-exhaustion case where df -h still shows space, and for the overlay2 writable layer that makes du and system df disagree — system df only counts what Docker itself accounts for. --volumes gets its own section marked most dangerous because after a database container is recreated, the old volume is no longer referenced and gets deleted along with its data.

Provided by ToolVault. Related tools: Dockerfile Generator, Run to Compose, Byte Converter, Linux Cheatsheet. Related reading: Docker daemon unreachable, Port already in use (EADDRINUSE), executable file not found. See the homepage for more developer tools.