At my current day job at Spearline, we have call servers set up in data centers around the world. When I started back at Spearline in January, all of these servers were running CentOS 6, so one of my first tasks was to figure out a plan for upgrading the servers to something more recent. The most obvious answer would be to set them up with CentOS 8, but we were also running into issues where different call servers might have different versions of certain packages, depending on how often they were updated.

One of the other changes we made was to move the telephony software into a container so we could have an easy split between the OS management (which is my team’s responsibility) and the telephony management (which belongs to another team that we work closely with). This change meant that we were able to look into some alternate OS solutions.

Given our shift towards containers, the most obvious solution would have been to switch to Fedora CoreOS, but a number of our call servers have Sangoma telephony cards with kernel drivers that are, unfortunately, out-of-tree. While there are some elegant ways to load custom kernel modules into Fedora CoreOS, we needed a more stable kernel, due to the (lack of) speed in which these modules are updated to build with new kernels.

So we decided to go with a custom OSTree distribution (surprisingly named SpearlineOS), built using rpm-ostree and CentOS 8. SpearlineOS has two streams, staging and production. At the moment, we’re manually building each new release, pushing it to staging, running it through some smoke tests, and, then, finally, pushing it to production. We are in the process of setting up a full staging environment with automatic builds and automatic promotion to production once a build has been functioning correctly for set period of time. We’ve also setup greenboot in SpearlineOS so that our servers are able to fail back to an older release if the current one fails for any reason.

We are using podman for container management because we’re using rootless containers pretty much everywhere. We have had some issues with the versions of podman in CentOS, so I’ve been rebuilding Fedora’s podman for SpearlineOS.

SpearlineOS has served us very well for new installations, with a quick installation time (about 45 minutes including all initial configuration) and minimal maintenance problems. In my next post, I’ll discuss how we’re going about upgrading our current servers from CentOS 6 to SpearlineOS.

This blog is produced using Hugo, a snazzy piece of software that allows you to create static web sites that look and feel like they’re dynamic. The obvious plus is that I can run my website on a tiny AWS VM, but the downside is that extra features, like comments, need some form of external dynamic code. I chose to use Staticman which creates a GitHub PR for each comment, removing the need for a database and giving me the opportunity to review each comment before it’s posted, thus keeping spam from making it to my site.

Unfortunately, it doesn’t keep spam from making it to me. Since I migrated from WordPress two and a half years ago, I’ve had 484 comments. Of those comments, 29 were not spam (and that’s including 10 comments which were my responses). The other 455 were spam. In the last year, there were 145 comments, exactly three of which were not spam.

With those numbers, I don’t think there’s much point in keeping comments on this blog, so I’m removing the ability to post comments effective immediately. I’m still reachable at my email address and will happily respond to any questions, comments, criticisms and insults. If you would like me to publish your comment, make it clear in the email and I will publish it using the same framework that I use to show the old comments.

Needless to say, this invitation does not extend to those who would like to sell me their marketing packages (“We will selflessly spam others on your behalf!”), SEO packages (“We’re so much smarterer than the Goggles and will trixz them into putting ur pagez firs1”), payday loans (“Low interest rates of 0.5%¹! We can’t be beat!²”) and cheap Viagra (“What could possibly go wrong?”).

Spam by Cypher789