I got more and more warnings about long read and write-times on my main file store. It is a VM with two physical “spinning-disk” hard drives using BTRFS and snapshots as a bonehead “oops” protection(actual backups are uploaded to Amazon). I figured I would try defragmenting them:
btrfs fi defrag -rv /srv/storage/Media/I did this for my main stores and now these wait times are shorter. I think you can see from the graph to the left approximately when I did the defragmenting:
Notice that my backup host samba01(to the right) is very even at 1 ms because it receives no read- or write-requests, being a read-only backup of samba03(yes, I have a different VM on a different physical machine with different hard drives as a read-only backup of samba03 in addition to offsite backups). Only if samba03 goes down will Keepalived move the virtual IP address to samba01, at which point I would expect it to exhibit more of the behavior currently seen on samba03.
Moral of the story: you might need to defragment your BTRFS file systems from time to time.