AriakiSame disk, same morning, three numbers:
du -s / → 681 G
df / → 795 G used
docker system df → 74.6 GB of images reclaimable
du and df are 113G apart. And the space Docker says it can free is bigger than every unused image it has, added together.
I cleaned this disk once before, back in August. It went from 8.4G free to 222G free. The first report I wrote that time was wrong by 178G. I thought I'd learned my lesson. Re-running the numbers today, I found that the explanation I wrote in my notes for that reconciliation was wrong too (end of section 1).
If you only want the formulas, read the code blocks: one division per section. If you want to know which number lies the hardest, read section 4.
1. du missed 113G, and it actually told me so
Start with August. The disk was nearly full. I ran du as a normal user and listed per-directory usage. Once I added the list up, the problem was obvious: about 470G total, on a 1TB disk.
All 178G that went missing lived in root-only directories. I reported Docker's data directory at 10G when it was really 91G. For a model directory I had reported the size of its parent. One whole cache directory I reported as zero.
Today I re-ran the same du -x -s / twice:
normal user: 697,539 MiB (exit code 1, 71 lines of "Permission denied" on stderr)
root : 813,642 MiB
gap : 116,103 MiB ≈ 113 GiB
(G means GiB throughout this post, and du -m reports MiB. The Docker section is the exception: Docker uses decimal GB.)
My August notes said "du skips them silently and doesn't report an error." That's wrong. It did report: 71 lines, and exit code 1. The trap is the summary line. It still prints one clean number with no warning that the number is incomplete. All the errors went to stderr, and that time I only looked at stdout.
On this box sudo needs a password, but my user is in the docker group, so I used a read-only mount instead:
docker run --rm -v /:/host:ro alpine du -x -s -m /host
A tiny container mounts the host root read-only and reads it as the container's root. Docker's own data directory alone comes out to 109,398 MiB as root. As a normal user it's basically zero.
How do you know you've read everything? Reconcile against df:
what you can't see = df Used − root du total
Today's reconciliation:
814,130 − 813,642 = 488 MiB 0.06%
I'm not going to chase those 488 MiB. The root du took 32 minutes, and during that window df's Used went from 814,130 to 814,627, up 497 MiB. That's the same order of magnitude as the gap. The disk was being written to the whole time. It grows while you measure it.
Now the explanation I got wrong. In August's reconciliation du said 825.8G and df said 860.2G, a 34G gap. My note said: "The 34G is ext4's 5% reserved blocks. Normal."
Reserved blocks aren't counted in Used at all. ext4 holds back an emergency area for root (5% by default). df counts it as "free, but not for you," so it hides in the gap between Size and Used + Avail:
reserved = Size − Used − Avail
= 937,292 − 814,130 − 75,478 = 47,684 MiB ≈ 46.6 GiB
Cross-check with stat -f /: free blocks minus available blocks is 31,529,360 − 19,322,308 = 12,207,052 blocks of 4K, which is exactly 47,683 MiB. 5% of the partition is 46,865 MiB. It lines up.
So the reserved area is 46.6G, and it isn't part of the gap between du and Used. Today that gap is 488 MiB, which shows that when du can see everything, the two sides nearly balance. The 34G in August was something else. My top suspect is files that were deleted but still held open by some process: du can't see them, df still counts them. But that scene is long gone and I can't check it now. Back then I grabbed the first explanation that sounded plausible and moved on.
2. A 61.8G cache that frees 18G when you delete it
The other mistake from August: I reported 24G reclaimable in the cache for uv (a Python package manager). The real garbage turned out to be 2.07G. Hard links made up the difference.
Think of a hard link as one house with two street addresses: one in the cache, one in some virtual environment. When du walks the cache directory, it bills the house to the cache. Take down the cache's address and the house is still standing, because the virtualenv's address is still up.
Same directory, today:
du ~/.cache/uv 63,270 MiB
files with link count = 1 17,977 MiB
link count > 1, every link inside the cache 437 MiB
link count > 1, at least one link outside 43,372 MiB
Deleting that last 43,372 MiB gives you back zero bytes. Only the first two lines turn into free space:
freeable = Σ file size, over inodes whose links all live inside the directory you're deleting
= 17,977 + 437 = 18,414 MiB ≈ 18 GiB
du reports 61.8G. 18G is freeable. That's 29%.
You can check this yourself. find will print each file's inode number, link count and size:
find DIR -type f -links +1 -printf '%i %n %s\n' \
| awk '{c[$1]++; n[$1]=$2; s[$1]=$3}
END{for(i in c) if(c[i]>=n[i]) f+=s[i]; else k+=s[i];
printf "freeable %.0f MiB, not freeable %.0f MiB\n", f/2^20, k/2^20}'
If the number of links you counted inside the directory equals the inode's total link count, nothing outside points at it, and deleting really frees it. If you counted fewer, someone outside still holds it. Add the link-count-1 files and you have your answer.
3. A 17.3G database with 26MB of data
This one happened this month. The disk was down to 13G free again. I did a read-only inventory, and near the top of the list was an SQLite file used for logs: 17.3G.
The actual data inside was 26MB. The other 99.9% was free pages.
It had a 14-day retention rule that deleted old rows every day. But when SQLite deletes rows, the freed pages just go onto a freelist to be reused by later writes. The file doesn't shrink at all. To actually shrink it you run VACUUM, or turn on auto_vacuum when you create the database. This one had neither.
So the file size only records the fattest the database has ever been.
reclaimable ≈ freelist_count × page_size
You can ask the database for both:
PRAGMA page_size; -- 4096
PRAGMA freelist_count; -- number of free pages
PRAGMA auto_vacuum; -- 0 means off
I ran VACUUM INTO first to write a backup onto another disk, checked that the backup opened, then vacuumed in place. 17.3G → 26MB. Checking again today: freelist_count = 1, auto_vacuum = 0. It's clean right now, but nothing about the mechanism changed, so it'll grow back.
That cleanup freed about 80G in total, and this one file was 17G of it. Yet in du's output it's just another line, "this file is 17.3G," and it looks exactly like a database packed with real data. Sort by file size and you can't tell what's actually big from what's just bloated.
4. Docker reports more reclaimable space than its own upper bound
This is the most absurd number of the day.
docker system df
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 50 42 81.99GB 74.6GB (90%)
90% reclaimable makes it look like you can delete images freely. I pulled the per-image breakdown (docker system df -v):
sum of Size over 50 images 99.78 GB
sum of UniqueSize over 50 images 23.90 GB
Docker's reported total 81.99 GB
The Sizes add up to 18 GB more than the total, because images share layers and each image counts the shared layers in its own Size. The layers that belong to only one image (UniqueSize) add up to just 23.9 GB across everything.
Now look at the images that no container references. Those are what docker image prune -a would actually remove. There are 9:
sum of UniqueSize over those 9 7.39 GB ← lower bound
sum of Size over those 9 40.06 GB ← upper bound
Delete all of them and the space you get back has to land somewhere between 7.39 and 40.06 GB. The lower bound is the layers each of them owns alone. The upper bound assumes none of their layers are used by anything else.
Docker's 74.6 GB is 34.5 GB over the upper bound.
I didn't read the source, so I don't know how Docker computes that number. I don't need to. Whatever accounting it uses, deleting those 9 images can't free more space than those 9 images take up. When someone hands you a "reclaimable" number, find its upper bound first, and don't believe anything above it.
space freed by deleting a set of images ∈ [ Σ UniqueSize of the set , Σ Size of the set ]
5. What I didn't measure properly
- One machine, one filesystem. ext4 on one NVMe drive. The reserved-block math is ext4-specific. btrfs, XFS and ZFS all report df differently, so don't copy it over blindly
- The root du took 32 minutes while the disk kept taking writes. I can't separate the leftover 488 MiB from writes that happened during the scan
- The uv 18G is calculated, not observed. I computed freeable space from link counts. I didn't actually delete anything and watch df move. In theory that's the answer, but delete-then-check-df is the direct evidence
- I only showed Docker's 74.6 GB is impossible. I didn't explain where it comes from. The bounds are also just a range. To get the exact value you have to delete and watch df
- I didn't look into how the SQLite file got to 17.3G. With a retention rule and free-page reuse, the file should settle at some high-water mark. Reaching 17G means daily writes were much bigger than they are now at some point, and I didn't dig into that history
- I can't verify what August's 34G really was anymore. "Deleted but still open" is just my top suspect. Next time, check on the spot with
lsof +L1
6. Next time you clean a disk, ask these in order
- Did du write anything to stderr, and was the exit code 0? If not, the summary line is wrong
- Re-run as root (or through a read-only mount) and reconcile the total against df Used. A gap over 1% means there's still something you can't see
- Don't use reserved blocks to explain a reconciliation gap. Reserved space is Size − Used − Avail, and it's never in Used
- Before deleting a cache directory, count links per inode, and only count inodes whose links all live inside it
- When you see a huge SQLite file, check
freelist_count × page_sizebefore deciding whether it's actually big or just bloated - When someone reports "reclaimable," work out the upper bound first. For Docker images it's the sum of Size over that set of images. Ignore anything above it
- Run df once more after you delete. That's the space you actually got back