Bahnhof needs to do some service so we’re running internet via Telia 4G for a few hours. I took the opportunity to get Galera on galera02 going again after I used that node for VPN experiments. It took some time since I confused backend0X with galera0X. I couldn’t quite figure out why mariadbgalera was down on all nodes… My monitoring system explained the discrepancy…
Turns out zabbix has a bunch of stuck queries in Galera so I guess the max_user_connection warning makes sense. I used for PD in $(mysqladmin processlist | awk ‘{print $2}’); do mysqladmin kill $PD; done to kill ongoing queries quickly and now it works.