"No space left on device" on Linux: find what's filling the disk

Writes fail with ENOSPC and services crash. How to find what's using the space in a few commands, free it safely (logs, Docker, package caches, deleted-but-open files), and handle the cases where df says there's room.

· 7 min read · By the Spot Downtime team

A full disk breaks things in odd ways: databases refuse writes, logins hang, deploys fail halfway, and apps crash with No space left on device (ENOSPC). The fix is always the same shape: find the biggest thing that shouldn't be there and remove it, then stop it growing back.

1. Which filesystem is full?

bash
df -h

Look at the Use% column. Usually it's /, but /var, /boot or a data volume can fill on their own. If nothing is near 100%, skip to when df says there's room.

2. Find the big directories

bash
# Biggest directories on the full filesystem, without crossing into other mounts
sudo du -xh --max-depth=2 / 2>/dev/null | sort -rh | head -20

# Then drill into the biggest one
sudo du -xh --max-depth=1 /var | sort -rh | head

ncdu (sudo apt install ncdu) does the same interactively and is faster to explore. The usual suspects, in order:

Logs

bash
sudo du -sh /var/log/* | sort -rh | head
journalctl --disk-usage
sudo journalctl --vacuum-size=200M      # keep the newest 200 MB of the journal

A single runaway log, often an app in a crash loop logging a stack trace per request, can fill a disk in hours. Truncate it rather than delete it while the app has it open: sudo truncate -s 0 /var/log/myapp/app.log. Then set up rotation in /etc/logrotate.d/ and cap the journal with SystemMaxUse=500M in /etc/systemd/journald.conf.

Docker

bash
docker system df                 # images, containers, volumes and build cache
docker image prune -a            # images no container uses
docker builder prune             # the build cache
docker container prune           # stopped containers

Old images from every deploy and the build cache are the usual culprits. Volumes hold data, so look before pruning them: docker volume ls -f dangling=true. Container logs also grow without limit by default; cap them in /etc/docker/daemon.json:

json
{ "log-driver": "json-file", "log-opts": { "max-size": "50m", "max-file": "3" } }

That applies to containers created after Docker restarts.

Package caches and old kernels

bash
sudo apt clean && sudo apt autoremove --purge    # Debian, Ubuntu
sudo dnf clean all                               # RHEL family

Backups, dumps and core files

Database dumps kept forever in /var/backups or a home directory, core dumps in /var/lib/systemd/coredump, and files in /tmp that nothing cleans. Move backups off the server; that's where they're useful anyway.

When df says there's room

Out of inodes

Every file uses an inode, and a filesystem has a fixed number. Millions of tiny files (session files, a mail queue, a cache directory) use them up while space remains:

bash
df -i
# find the directory with the most files
sudo find / -xdev -type f | cut -d/ -f2-3 | sort | uniq -c | sort -rn | head

Deleted files still open

Deleting a log that a process still writes to doesn't free its space until the process closes it. du no longer sees the file, but df still counts it:

bash
sudo lsof +L1 | sort -k7 -rn | head

Restart the process that holds it (or reload it, for services that reopen their logs) and the space comes back.

Reserved blocks

ext4 reserves 5% for root, so ordinary users see "full" before df says 100%. On a big data disk that's a lot of space; sudo tune2fs -m 1 /dev/sdb1 lowers it to 1%. Keep the reserve on /: it's what lets root log in and fix things.

Grow the disk when it's simply too small

If everything on it belongs there, cleaning only buys days. On most clouds you can grow a volume online, then the partition with growpart and the filesystem with resize2fs (ext4) or xfs_growfs (XFS).

Know before it's full

  1. Install the Spot Downtime agent on the server; see monitoring CPU, memory and disk.
  2. It alerts as soon as any disk goes over your limit (90% by default) and shows which mount point, with every disk's use on the server's page.
  3. Set the limit where you still have a day or two of growth left: lower for disks that fill fast, like log and database volumes.
  • The disk chart shows the trend, so you can see whether it's a slow climb (add space) or a sudden jump (something's logging like mad).

If the disk filled because of Docker, see also containers that keep restarting: a crash loop is a common cause of exploding logs.

Get new guides by email

Practical posts on monitoring, outages, error fixes and on-call, when they're published. No spam.

Unsubscribe from any email with one click. See our privacy policy.

Keep reading