Disk full: how to recover

Topic: Servers linux

Recover when the root or critical partition is full: free space quickly by clearing logs and caches, find and remove or move large files, then fix rotation and monitoring so it does not recur. Use this when the system is read-only or services fail with no space.

Intent: Troubleshooting

Quick answer

  • Free space immediately: journalctl --vacuum-size=50M; delete /var/log/*.gz; truncate large log (echo > file); apt clean or dnf clean all; remove old kernels if safe.
  • Find what filled disk: du -x / 2>/dev/null | sort -n | tail -20; or ncdu; fix or move the source (log rotation, app writing to wrong path).
  • If root is read-only: free enough space so writes work again; then fix fstab or fsck if the filesystem went read-only due to errors.

Prerequisites

Steps

  1. Free space immediately

    sudo journalctl --vacuum-size=50M; sudo find /var/log -name '*.gz' -delete; sudo truncate -s 0 /var/log/huge.log; sudo apt clean; check df -h.

  2. Identify largest consumers

    du -x / 2>/dev/null | sort -n | tail -30; find / -xdev -type f -size +100M 2>/dev/null; stop or limit the process writing (e.g. app, cron).

  3. Remove or relocate

    Move large files to another disk or archive and delete; fix logrotate or journald config; add disk or expand volume if growth is expected.

  4. Prevent recurrence

    Configure rotation and journald limits; alert when usage > 85%; document what was removed and the new limits.

Summary

You will recover from a full disk by freeing space quickly, finding what filled it, and fixing rotation and monitoring. Use this when the system is unresponsive or reports “no space left on device.”

Prerequisites

  • Root or console access; ability to delete or truncate files and restart services.

Steps

Step 1: Free space immediately

sudo journalctl --vacuum-size=50M
sudo find /var/log -name "*.gz" -delete
sudo truncate -s 0 /var/log/problem.log
sudo apt clean
df -h

Step 2: Identify largest consumers

sudo du -x / 2>/dev/null | sort -n | tail -30
sudo find / -xdev -type f -size +100M -exec ls -lh {} \;

Step 3: Remove or relocate

  • Delete or archive old data; fix the app or cron that wrote too much; configure logrotate and journald.

Step 4: Prevent recurrence

  • Set SystemMaxUse in journald; add logrotate config; monitor disk and alert; expand storage if needed.

Verification

  • df shows free space; services run; writes succeed; rotation and limits are in place.

Troubleshooting

Read-only filesystem — Free space from another session or boot single-user; if the fs went ro due to errors, run fsck and fix fstab; see read-only recovery guide.

Cannot delete file (device busy) — Process has it open; stop the process or restart the service; or truncate the file instead of delete.

Next steps

Continue to