A few years ago, I attended my first Linux conference, DevConf 2014. Many of the speakers talked about containers and how wonderful they were, and my interest was piqued, but I’ve never really had an opportunity to use them.

As the sysadmin for a school, there just isn’t much need for the scalability provided for by containers. Our internal web site runs on a single VM, and the short downtimes required for system updates and upgrades are not a problem, especially if I plan them for the weekends. On the flip side, having something that we can use to spin up web services quickly isn’t a bad idea, so, over the last few months, I’ve been experimenting with Kubernetes.

My main goal was to get something running that was easy to set up and Just Works™. Well, my experience setting up Kubernetes was anything but easy, but, now that it’s up, it does seem to just work. My main problem was that I wanted to use my three oVirt nodes (running CentOS 7) as both Kubernetes nodes and masters, which meant the tutorials took a bit of finessing to get working.

I mostly followed this guide for the initial setup, and then this guide, but I did run into a few problems that I’d like to document. The first was that my containers were inaccessible on their cluster IP range, the primary symptom being that the kube-dashboard service couldn’t connect to the kubernetes service. It turned out that I, rather stupidly, forgot to start kube-proxy, which does all the iptables magic to direct traffic to the correct destination.

The second problem I ran into was that I couldn’t get pretty graphs in kube-dashboard because the heapster service wouldn’t start because I hadn’t set up the cluster DNS service, kube-dns. To be fair, the instructions for doing so are pretty unclear. In the end, I downloaded skydns-rc.yaml.sed and skydns-svc.yaml.sed, and replaced $DNS_DOMAIN and $DNS_SERVER_IP with the values I wanted to use.

The final problem I ran into is that I’m using our school’s local Certificate Authority for all the certificates we use, and I’ve had to keep on adding new subject alternative names to the server cert and then regenerate it. At the moment, it’s got the following:

DNS:kubernetes.example.com
DNS:node01.example.com
DNS:node02.example.com
DNS:node03.example.com
DNS:localhost
DNS:localhost.localdomain
DNS:kubernetes.default.svc.local

Where I replaced $DNS_DOMAIN with “local”

DNS:kubernetes.local
DNS:kubernetes.default
IP Address:127.0.0.1
IP Address:172.30.0.1

Where our cluster IP range is 172.30.0.0/16

I suspect I could now get rid of some of those hostnames/addresses, and I’m not even sure if this method is best practice, but at least it’s all working.

So I’m at the point now where I need to see if I can setup our MikroTik router as a load balancer and then see if I can get our web based marking system, LESSON, moved over to a container with multiple replicas. Hurrah for redundancy!

Broken Wagon Wheel by Kevin Casper is in the public domain / used under a CC0 license

Since I last posted, there have been a number of small updates, but nothing that seemed big enough to write about. So I figured it might be worth posting a short summary of what I’ve been up to over the last couple of months.

In no particular order:

FOSDEM

I had the opportunity to visit FOSDEM for the first time last month. Saw lots of cool things, met lots of cool people and even managed to bag a LibreOffice hoodie. Most importantly, it was a chance to build friendships, which have a far higher value than any code ever will.

Wireless access points

I should probably write a proper post about this sometime, but a number of years ago we bought about 30 TP-LINK WR741ND wireless APs and slapped a custom build of OpenWRT on them. We installed the last spare a couple of months ago and ran into problems finding a decent replacement (specific hardware revisions can be quite difficult to find in Lebanon). After much searching, we managed to get ahold of a TP-LINK WR1043ND for testing and our OpenWRT build works great on it. Even better, it has a four-port gigabit switch which will give us much better performance than the old 100Mbps ones.

LizardFS patches

I ran into a couple of performance issues that I wrote some patches to fix. One is in the process of being accepted upsteam, while the other has been deemed too invasive, given that upstream would like to deal with the problem in a different way. For the moment, I’m using both on the school system, and they’re working great.

Kernel patch (tools count, right?)

After the F26 mass rebuild, I ran into problems building the USB/IP userspace tools with GCC 7. Fixing the bugs was relatively simple, and, since the userspace tools are part of the kernel git repository, I got to submit my first patches to the LKML. The difference between a working kernel patch and a good kernel patch can be compared to the difference between a Volkswagen Beetle and the Starship Enterprise. I really enjoyed the iterative process, and, after four releases, we finally had something good enough to go into the kernel. A huge thank you goes out to Peter Senna, who looked over my code before I posted it and made sure I didn’t completely embarrass myself. (Peter’s just a great guy anyway. If you ever get the chance to buy him a drink, definitely do so.)

Ancient history

As of about three weeks ago, I am teaching history. Long story as to how it happened, but I’m enjoying a few extra hours per week with my students, and history, especially ancient history, is a subject that I love. To top it off, there aren’t many places in the world where you can take your students on a field trip to visit the things you’re studying. On Wednesday, we did a trip to Nahr el-Kalb (the Dog River) where there are stone monuments erected by the ancient Assyrian, Egyptian, and Babylonian kings among others. I love Lebanon.