Recently, my Raspberry Pi was complaining that the disk was full. Which was strange because I just recently removed a bunch of files from there, yet
df -h was reporting 99% usage. But wait,
du -h --summarize was only reporting a couple gigs used instead of the whole 14Gb…
Starring at Red Herrings 🐰
Once I first noticed the problem, I ran a quick
df -h to confirm disk usage, which was reported as 99% full.
My first thought was to look for large files or folders running a custom alias based on
du, but it reported nothing suspicious.
Even stranger, a
du -h --summarize to get a grand total of the space taken up by files on the full partition only reported about 2Gb of space vs the 14Gb of the partition.
Time for some detective work! 🔎
Well, actually… Time for some StackOverflow searching 😅
Three possible reasons for this discrepancy came up with a quick search: files in a folder that gets mounted over, deleted large files and running out of inodes.
fstab and current
mount points unfortunately didn’t reveal anything.
I was hoping that a bunch of large files had been mistakenly copied to a path that was unmounted at the time, and which a subsequent mount would have shadowed.
Not it :(
lsof | grep deleted
Looking for deleted large files didn’t reveal any obvious culprit. A good old restart for good measure didn’t appear to do much good, so it was obviously not the issue…
Running out of inodes
Apparently running out of inodes can cause a filesystem to appear full, which can be verified with the above command. Not it either… 😨
At that point I’m starting to scratch my head.
And that’s when I notice something that doesn’t ring a bell: both
df shows that the partition’s filesystem is of
I don’t remember choosing this filesystem, but I wasn’t really paying that much attention when setting up this particular Pi… But after looking into it, it turns out that’s a pretty good candidate 💡
btrfs did it!
This relatively new(ish) filesystem has two features called
snapshots (actually snapshots are subvolumes).
Subvolumes can be mounted recursively, and when unmounted
du doesn’t pick them up.
Snapshots are special subvolumes that can be easily made either manually or automatically and capture the state of a subvolume in an incremental way.
This is pretty useful if you want to be able to roll back to a previous state of the system, without consuming too much disk space (thanks to the incremental nature).
Looking at the list of subvolumes in my partition, I quickly realized that somehow there was automatic snapshots made around
apt-get usage AND on a weekly and daily basis.
Subvolumes (normal ones AND snapshots) can be listed using the following command:
btrfs subvolume list -a /
-a instructs to list all subvolumes, not only the ones directly under the provided path (here the root
I got a list including something like this:
ID 4767 gen 253284 top level 274 path <FS_TREE>/storage/@btrfs-auto-snap_weekly-2019-04-23-0854 ID 4823 gen 264859 top level 259 path <FS_TREE>/storage/@btrfs-auto-snap_daily-2019-04-29-1011
btrfs-auto-snap_weekly… Could it be it? 😧
It took be a bit of time, trial and error, but I was able to confirm this was the issue and fix it:
Finding Out the Size of the Snapshots
Turns out this is not that easy. You need to:
- Ensure at least once that
btrfsquotas are turned on:
btrfs quota enable /
- List the sizes by subvolume id using said quotas feature:
btrfs qgroup show /
- Cross link the result to the list of subvolumes, using the subvolume ID in
btrfs subvolume list -a /
Easier said than done when the qgroup output is like this:
qgroupid rfer excl -------- ---- ---- 0/4685 0.00B 0.00B 0/4759 0.00B 0.00B 0/4761 0.00B 0.00B 0/4763 32.29MiB 16.00KiB 0/4765 0.00B 0.00B 0/4767 16.00KiB 16.00KiB 0/4819 0.00B 0.00B 0/4820 0.00B 0.00B 0/4821 2.50GiB 0.00B
I looked for an easier alternative and found a script that, after inspection, I deemed safe to run on that machine. But remember kids, don’t accept any script from strangers 🍯
See https://ownyourbits.com/2017/12/06/check-disk-space-of-your-btrfs-snapshots-with-btrfs-du/ for the script presentation and content.
With it I was able to confirm that, indeed, I had 4 weekly snapshots that were in the 2Gb-6Gb range, each.
Deleting the Snapshots
Snapshots and subvolumes can supposedly be deleted like any other file, but it is better to delete them via
btrfs which is way faster.
The command is:
btrfs subvolume delete NAME_OF_SNAPSHOT
But I was confused, because I was unable to find a way to perform that command without a “file doest not exist” kind of error…
s/Deleting/Mounting/ the Snapshots
In order to be able to find the subvolumes listed by the script above, I needed to first mount the root subvolume somewhere on my current filesystem.
That is, after finding the
/dev/file corresponding to my partition, of course 😕
mount /dev/mmcblk0p8 /mnt -o subvol=/ cd /mnt/storage
This revealed the
storage/ subfolder under the mount point
/mnt/, in which lay all of my snapshots 🎉
btrfs subvolume delete commands of the above, unmounting via
umount /mnt for good measure and running
df -h again, everything was finally back to normal:
Filesystem Size Used Avail Use% Mounted on /dev/mmcblk0p8 14G 3.5G 11G 26% /
So Today I Learned
btrfs… the hard way 😉