I first set up my new server arrangement with GlusterFS which is easier to get up and running than Ceph and has great disaster recovery properties since it stores data in regular file systems on hard drives. So if GlusterFS stops working with error message PC LOAD LETTER I can just copy all the files back from the FS used on each drive.
Buuut… GlusterFS is slow as tar. I tried to put a MySQL database on a virtual disk stored on GlusterFS and it didn’t go well. IO wait was absurdly high. That made me set up a Percona XtraDB Cluster on each Proxmox node’s local storage which worked well at first… Well that’s another issue. For metadata operations there’s a setting readdir-ahead ( https://staged-gluster-docs.readthedocs.io/en/release3.7.0beta1/Features/readdir-ahead/ ) which helps a bit but it’s still slow.
Apparently GlusterFS is okay for people streaming big files, like video editors. At the very least it should be good for archival purposes. Though the necessity of clustered expand-online storage for archives is probably not super common.