I was trying to find the source of some weird iowait on my main workstation and came across https://haydenjames.io/what-is-iowait-and-linux-performance/ which recommended the following
iostat– try it with the-xm 2options for extended statistics, in megabytes and in two-second intervals.iotop– top-like I/O monitor. Try it with the-oPaoptions to show the accumulated I/O of active processes only.ps– useauxf, then under the “STAT” column “D” usually indicates disk iowait.strace– view the actual operations issued by a process. Read thestraceman page.lsof– aft
Not bad. iostat -x is very wide so iostat -xsm 2 can be useful. It will show iowait unlike iostat -sm 2
iotop -oPa was great and showed that btrfs-transaction kernel thread was the big writer.
With apt install inotify-tools I got some nice inotify-based tools(as the name would suggest). Turns out that no files were actually written to the disks seeing iowait so I’ll demo it on another folder:
root@amd:~# inotifywatch -r /home/cjp/.cache/ Establishing watches... Finished establishing watches, now collecting statistics. ^Ctotal modify close_write open create delete filename 53 33 5 5 5 5 /home/cjp/.cache/google-chrome/Default/Cache/Cache_Data/
It’s probably just another btrfs write-amplification-thing-a-majig. This showed the same thing:
lsof +D /path/to/btrfsmount
I’ve seen this before, just not with zero writes to the filesystem… Like 100KB=>500KB write I’ve seen. But 0=>500KB? That’s new.
Memo to self: be careful about putting Btrfs on SSDs.