Skip to main content
Linux · lesson 7 of 7

Beginner

2

Intermediate

5

Also part of Linux

7
Open the Linux page →
Linux intermediate Lesson 7 of 7

Inodes: The Filesystem Is Full, but df -h Says 1% Used

A real No space left on device error on a filesystem with 11MB free, caused by exhausting inodes rather than disk blocks — plus what hard links and symlinks reveal about what an inode actually is.

By SkillByExample · Published

linux filesystems inodes links disk

df -h reporting 99% free space says nothing about whether the next touch will succeed — a filesystem tracks two separate resources, and only one of them shows up in that command.

Building a filesystem with almost no inodes

docker run -d --name fs-lesson --privileged ubuntu:22.04 sleep infinity
docker exec fs-lesson bash -c "dd if=/dev/zero of=/tmp/inodefs.img bs=1M count=16 && mkfs.ext4 -N 64 -F /tmp/inodefs.img"
docker exec fs-lesson bash -c "mkdir -p /mnt/tiny && mount -o loop /tmp/inodefs.img /mnt/tiny"
docker exec fs-lesson df -i /mnt/tiny
Filesystem     Inodes IUsed IFree IUse% Mounted on
/dev/loop2         64    11    53   18% /mnt/tiny

mkfs -N 64 forced this 16MB filesystem to have only 64 inodes total, instead of the thousands ext4 would normally allocate for that size — 11 are already claimed by the filesystem’s own root directory and reserved structures.

The bug: out of inodes, not out of space

docker exec fs-lesson df -h /mnt/tiny
docker exec fs-lesson bash -c "for i in \$(seq 1 60); do touch /mnt/tiny/file\$i || break; done"
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop2       12M   24K   11M   1% /mnt/tiny
touch: cannot touch '/mnt/tiny/file54': No space left on device

1% Use on the space check, and yet the 54th empty file — each using essentially zero bytes of actual data — fails with the same error text a genuinely full disk would give.

docker exec fs-lesson df -h /mnt/tiny
docker exec fs-lesson df -i /mnt/tiny
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop2       12M   24K   11M   1% /mnt/tiny
Filesystem     Inodes IUsed IFree IUse% Mounted on
/dev/loop2         64    64     0  100% /mnt/tiny

Same filesystem, two different pictures: df -h still shows 1%, while df -i shows 100%. Every file — no matter how small — consumes exactly one inode, and this filesystem was built with only 64 of them to begin with.

The fix: free inodes, not space

docker exec fs-lesson bash -c "rm /mnt/tiny/file1 /mnt/tiny/file2 /mnt/tiny/file3 /mnt/tiny/file4 /mnt/tiny/file5"
docker exec fs-lesson df -i /mnt/tiny
docker exec fs-lesson bash -c "touch /mnt/tiny/newfile && echo TOUCH_OK"
Filesystem     Inodes IUsed IFree IUse% Mounted on
/dev/loop2         64    59     5   93% /mnt/tiny
TOUCH_OK

Deleting 5 files freed 5 inodes regardless of how little data they held — the fix for “disk full” and the fix for “inodes full” look identical (rm), but only one of them actually helps here; deleting a single large file on this same filesystem would have freed disk blocks while leaving IUse% at 100%.

docker exec fs-lesson bash -c "cd /tmp && echo 'original content' > original.txt && ln original.txt hardlink.txt && ln -s original.txt symlink.txt && ls -li original.txt hardlink.txt symlink.txt"
367367 -rw-r--r-- 2 root root 17 Sep 14 06:30 hardlink.txt
367367 -rw-r--r-- 2 root root 17 Sep 14 06:30 original.txt
367405 lrwxrwxrwx 1 root root 12 Sep 14 06:30 symlink.txt -> original.txt

original.txt and hardlink.txt share inode 367367 and both show link count 2 — they are two directory entries pointing at the same underlying file, not two copies. symlink.txt has its own separate inode (367405) that just stores the text original.txt.

docker exec fs-lesson bash -c "cd /tmp && rm original.txt && cat hardlink.txt; echo ---; cat symlink.txt"
original content
---
cat: symlink.txt: No such file or directory

rm only removed one directory entry and dropped inode 367367’s link count from 2 to 1 — the data stays until the count hits zero, so hardlink.txt reads fine. symlink.txt never referenced that inode at all, only the now-gone name original.txt, so it’s left dangling.

Practice

1. Confirm the hard link's count dropped after the original was removed. 
docker exec fs-lesson bash -c "ls -li /tmp/hardlink.txt"
367367 -rw-r--r-- 1 root root 17 Sep 14 06:30 /tmp/hardlink.txt

Link count is now 1, down from 2 — one more rm /tmp/hardlink.txt would finally drop it to 0 and actually free the inode and its data, which is the real rule rm follows: delete a name, and free the data only when nothing names it anymore.

2. Check the total inode capacity a normal-sized ext4 filesystem gets, without forcing it low. 
docker exec fs-lesson bash -c "dd if=/dev/zero of=/tmp/normalfs.img bs=1M count=16 2>/dev/null && mkfs.ext4 -F /tmp/normalfs.img 2>&1 | grep inodes"
Creating filesystem with 4096 4k blocks and 4096 inodes

The exact same 16MB size, formatted without -N, gets 4096 inodes by default — 64 times more than this lesson’s deliberately constrained filesystem. Hitting real inode exhaustion in practice almost always means either millions of tiny files (a cache or session-file directory that never cleans up) rather than a filesystem this artificially small.

3. Find which directory is actually consuming the exhausted inodes. 
docker exec fs-lesson df -i /mnt/tiny
docker exec fs-lesson bash -c "find /mnt/tiny -printf '.' | wc -c"
Filesystem     Inodes IUsed IFree IUse% Mounted on
/dev/loop2         64    60     4   94% /mnt/tiny
51

IUsed (60) and the raw find walk (51) don’t match, and that gap is itself informative: ext4 reserves a handful of low inode numbers for internal bookkeeping — the journal, bad-block list, ACL support — that df -i counts but that never appear as a directory entry find can walk. On a large filesystem, running this same find -printf '.' | wc -c per top-level subdirectory (instead of a single count for the whole mount) is the real move to localize which directory is actually responsible for exhausted inodes.

Next: back to the DevOps track’s automation tools, or revisit bash for script-level Linux administration patterns this topic deliberately left to it.

Frequently Asked Questions

How can a filesystem be 'full' when df -h shows plenty of free space?
df -h reports data-block usage — how much of the filesystem's storage capacity holds file content. Every file also consumes one inode, a fixed-size metadata record, and the number of inodes a filesystem has is set permanently when it's formatted. Enough small or empty files can exhaust the inode count while barely touching the block usage, and the kernel refuses to create anything new either way.
Why did touch fail with the exact same 'No space left on device' error as running out of disk blocks?
ENOSPC, the kernel error code behind that message, means 'no space available to complete this allocation' generically — it doesn't distinguish which resource ran out. df -i, not the error text, is what tells inode exhaustion apart from real disk-full: df -h stays low while df -i shows IUse% at 100%.
What's the actual difference between a hard link and a symlink?
A hard link is a second directory entry pointing at the exact same inode number as the original — there's no 'original' after creation, just two names for one inode, and the file's content is only freed once every hard link to it is removed. A symlink is its own separate inode containing a text path to another name; deleting the file that path points to leaves the symlink dangling, because it never referenced the target's inode at all.
Why did cat on the hard link still work after rm'ing the 'original' file, but the symlink broke?
rm doesn't delete a file's data directly — it removes one directory entry and decrements that inode's link count; the data is only actually freed once the link count reaches zero. The hard link was a second reference to that same inode, so the count went from 2 to 1 and the data stayed. The symlink held a path string to a name that no longer existed, so following it failed.
Does every filesystem have a fixed inode count like ext4?
No — this is specifically an ext-family design decision, where the inode table is sized once at mkfs time and can't grow later without reformatting (mkfs -N sets it explicitly, as in this lesson). Filesystems like XFS and Btrfs allocate inodes dynamically as needed, so this particular failure mode — plenty of space, zero inodes — is an ext2/ext3/ext4-specific thing to watch for, not universal.