So I haven’t been super-pleased with Kubernetes historically. I spent the past… five weeks(?) on setting up Nomad and Consul instead. It’s been frustrating but manageably so. Podman support is okey but not great and I of course had to double-down on using Podman instead of Docker because why not?
Having spent… a day on Kubernetes? Did I start this Kubernetes project today? No, last night according to the snapshots for the virtual machines. Okey, but it still feels like it’s been a week. And this has helped me put words to things. Feedback in Kubernetes is very flimsy. I had to turn on debug-logging to be told that I hadn’t provided a configuration for the load balancer! Turns out the people who made the Helm chart for metallb don’t coordinate things entirely with the people who make metallb so the docs don’t line up with the chart.
Normally that would be a minor problem since a tool like Nomad complains very well when things are wrong. Something as major as “You provided no config ‘metallb'” would have popped up in like three seconds if I tried that with Nomad. But with Kubernetes? Three hours of debugging flannel(which was also not working it turned out), kube-router(which didn’t work initially because flannel CNI-files were left behind), iptables(because who the hell knows?) and namespaces(because I don’t exactly understand how well insulated things are in different namespaces in Kubernetes) solved the issue. In the official docs the configmap is named “config”, in the helm chart it’s “metallb”.
I thought that I would be more okey with Kubernetes yaml-files now that I’ve worked with Nomad and Consul which uses similar concepts of services, ports, container images and so on but no… still awful… Oh, this new Kubernetes setup consists of three flimsy master nodes and three beefier worker nodes, all running as VMs on my main workstation which now has an uptime of…
00:19:53 up 67 days, 1:25, 16 users, load average: 1,33, 1,91, 2,05
That’s right! (I say, expecting people to look impressed)
So anyway… The point is that I have snapshots of the VMs for easy restore and these things aren’t used for anything important or even “I kind of like having X available”. That’s running on my actual servers with Rocky Linux, HAproxy, Consul, Nomad, Minio etc. Solid as a rock. So now Kubernetes can fail all it wants and I can debug it in my own good time. Of course, by the looks of it the limits of my natural life seem to be a bigger restriction in that regard than I would like. Even if I live to be 100 I’m not sure how many Kubernetes configuration issues I can resolve in that time. 7? 8?
If I can get Kubernetes to run the same things I run on Nomad/Consul then I will try to keep it going and make changes to it alongside the “production” setup to see how things fare. I’m going to push Loki+Promtail and Cortex+Prometheus next I think along with Cadvisor and stuff. Hopefully this will resolve some of the feedback issues I’ve been having by centralizing all logs. I guess I’ll switch everything to debug-logging initially to be on the safe side.
Another thing that’s very nice about Nomad and Consul is that it is very nice to use incrementally. You don’t need 300 lines of yaml-data to get the Nomad dashboard up and running…
root@k8s-master01:~/kubernetes_dev/dashboard# wc -l * 17 dashboard_ingress.yaml 314 dashboard_multi.yaml 304 dashboard_multi_default.yaml 20 dashboard_svc.yaml 306 recommended.yaml 961 total
It’s just there automatically… Same with Consul. So at this stage I would recommend anyone who wants to learn Kubernetes to start with Nomad and Consul to get a feel for the concepts. Maybe you will find that Nomad and Consul is all you need(that’s where I am right now), if not it will be a good starting point.