I kept seeing connections dropping out in Grafana and sometimes I even saw it in the browser with “No data” in all panels.
Things seemed to be in order with pdnsauth and pdnsrecursor so we had pihole as a suspect. Indeed keepalived didn’t think it was stable:
Sep 27 17:40:41 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 0 Sep 27 17:41:02 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 9 Sep 27 17:41:26 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 0 Sep 27 17:42:17 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 9 Sep 27 17:42:30 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 0 Sep 27 17:43:32 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 9 Sep 27 17:43:56 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 0 Sep 27 17:45:02 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 9 Sep 27 17:45:26 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 0 Sep 27 17:47:02 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 9 Sep 27 17:47:26 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 0 Sep 27 17:50:32 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 9 Sep 27 17:50:56 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 0 Sep 27 17:51:32 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 9 Sep 27 17:51:56 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 0 Sep 27 17:53:02 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 9 Sep 27 17:53:15 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 0 Sep 27 17:54:47 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 9 Sep 27 17:55:11 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 0 Sep 27 17:56:47 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 9 Sep 27 17:57:11 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 0 Sep 27 17:58:32 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 9 Sep 27 17:58:56 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 0 Sep 27 18:00:32 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 9 Sep 27 18:00:56 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 0 Sep 27 18:01:47 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 9 Sep 27 18:02:11 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 0 Sep 27 18:03:32 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 9 Sep 27 18:03:56 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 0 Sep 27 18:05:17 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 9 Sep 27 18:05:41 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 0 Sep 27 18:07:32 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 9 Sep 27 18:07:56 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 0 Sep 27 18:10:17 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 9 Sep 27 18:10:41 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 0 Sep 27 18:11:02 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 9 Sep 27 18:11:26 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 0 Sep 27 18:12:02 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 9 Sep 27 18:12:26 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 0 Sep 27 18:13:32 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 9 Sep 27 18:13:56 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 0 Sep 27 18:14:47 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 9 Sep 27 18:15:11 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 0 Sep 27 18:16:32 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 9 Sep 27 18:16:56 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 0 Sep 27 18:18:17 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 9 Sep 27 18:18:41 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 0 Sep 27 18:20:17 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 9 Sep 27 18:20:41 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 0 Sep 27 18:21:02 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 9 Sep 27 18:21:26 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 0 Sep 27 18:22:02 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 9 Sep 27 18:22:30 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 0 Sep 27 18:23:02 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 9 Sep 27 18:23:11 runner02.incandescent.tech Keepalived_vrrp[418200]: Script `chk_dns` now returning 0
Now pihole is out, we’re going straight to pdnsrecursor which checks certain names with pdnsauth, otherwise it’s the usual DNS resolution. Seems more stable:
