Why didn’t my own handcrafted work when HAproxy was the middleman? It worked with curl! Maybe compression? No, turns out that was OK. Some other Accept-header? I ended up running tcpdump:
I ended up suspecting the port 80 being sent by Prometheus. I tried it with curl but even with this below curl didn’t send the portnumber(and so didn’t fail):
curl -sH 'Accept-encoding: gzip' -H "Accept: application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,/;q=0.1" http://networkmon.incandescent.tech:80/ | gunzip -
I changed it in HAproxy instead and it worked:
Note how the portnumber isn’t shown in the output above but tcpdump showed us that the port number is sent along. Anyway, I needed HAproxy’s Consul service discovery to translate the networkmon pointer into a fixed address. This is what I use now:
acl ACL_networkmon hdr_sub(host) -i networkmon.incandescent.tech
use_backend networkmon-backend if ACL_networkmon
backend networkmon-backend
balance roundrobin
option httpchk HEAD /
server-template networkmon 1 _networkmon._tcp.service.consul resolvers consul resolve-opts allow-dup-ip resolve-prefer ipv4 check
I’ll post the entire Ansible cookbook soon.