A year ago, I put together a post about the multiseat Fedora systems we’re using in our school. Over the past month, I’ve been putting together an upgrade from our Fedora 19 image to Fedora 21.

While doing the upgrade, I ran into a few bugs, and the first one was a doozy! Roughly half the time our multiseat systems started, the login screen would only show on two or three of the four seats. The only way to fix it was to restart the display manager, and even that only had a 50% chance of success.

At first I tried bodging around the bug by staggering the timing of Xorg’s startup, but that only made things worse. So I started looking at the logs and then looking at the Xorg code. It became obvious that the problem was that the first seat (seat0) would try to claim all the GPUs on the system. If it beat the other seats to their GPUs, they would, oddly enough, refuse to start. I put together a patch, filed a bug, and watched as those who know a lot more about Xorg’s internals take my ugly patch and make it beautiful. This patch has been merged into Xorg 1.17 and I’m hoping we’ll get it backported for F20 and F21 as I really don’t want to have to maintain internal Xorg packages until we switch to F22.

There do seem to be a couple of other bugs related to lightdm/xorg, but they’re far rarer and I haven’t spent much time on tracking them down, much less filing bugs. Occasionally lightdm starts the X server, but never gets a signal back saying that it’s ready, so they both sit there waiting for the other process. And far more rarely, the greeter crashes, which causes lightdm to shut down the seat. I think lightdm should retry a few times, but either it doesn’t or I haven’t found the right config option yet.

We did run into one interesting race condition in anaconda when we started mass-installing F21 on our systems. We use iPXE and Fedora’s PXE network install images with a custom kickstart to do the install (in graphical mode, because pretty installs make it less likely that a student will press the reset button while the install is progressing). On some systems, I’d get an error message that basically said that a repository that was supposed to be enabled had disappeared, which would crash anaconda.

Thanks to anaconda’s wonderful debugging tools, I was able to work out what list was being emptied and finally tracked it down to a race between the backend filling the frontend with its list of repositories and the frontend telling the backend to remove any repositories that aren’t in its list of repositories. Another ugly patch attached to the bug report, and we’ll see what happens with this one. At least I’m able to rebuild the squashfs installer image so the bug is fixed for us internally.

So most of our computers have now been upgraded to Fedora 21 and the reaction from our students has been positive. Now to get some Fedora 22 test systems built…

I was reading the backlog of the Fedora development mailing list and came across a post in which Richard Hughes made a very interesting comment:

I know lots of Red Hat developers worn down by the low-level harassment on this mailing list, so much so, that they just stop pushing the boundaries and go work on something else cool, e.g. ChromeOS.

I’ve been following this particular mailing list for many years, and the sad thing is, I think he’s right. There’s this underlying current of “us versus them” that can pop up, especially in longer-running threads, and “them” is someone with a @redhat.com email address.

On some levels this makes sense. Red Hat is the single largest entity in Fedora and many (if not most) of the movers and shakers in Fedora are Red Hat employees. A quick glance at the Fedora 21 System Wide Changes shows many more Red Hat employees than not. Is it any wonder that individual contributors can feel a bit like a sailboat in the way of an aircraft carrier?

So, is this some conspiracy to keep Fedora under Red Hat control? Is it something we should fight against? Or is there a reasonable explanation for Red Hat’s influence?

First off, there’s the question of whether people are hired at Red Hat to work on Fedora or whether they’re hired because of their work on Fedora. I had the opportunity at Devconf earlier this year to sit down with Patrick Uiterwijk, who did most of the work on Fedora’s OpenID provider, and was then hired by Red Hat because of that work. Patrick’s is not the only story like that. While not all competent Fedora contributors are Red Hat employees, Red Hat employees who contribute to Fedora are generally pretty darn competent, and competency in Fedora is rewarded with influence.

There’s also the fact that Red Hat pays people to work on Fedora. Many individual contributors are working on Fedora in their spare time. While this doesn’t necessarily affect the quality of their work, it does tend to affect the quantity. To give an example, at DevConf, I also talked with Stephen Gallagher about joining the Fedora Server working group. After DevConf, I signed up for the mailing list and then did… nothing. I’m the sysadmin and a teacher at my school, and at home I’m a husband and father of four children under six. While I have great intentions of helping out with the Server working group, it’s just not high enough on my list of priorities for me to have the time… and I suspect I’m not the only individual contributor in that boat.

Finally, there’s the fact that Red Hat’s employees actually get to know each other, at least to some extent. One of the big things I’ve learned in my years working here in Lebanon is the importance of relationship. It’s a lot easier to work with someone after you’ve sat down with them, had a coffee (or, in my case, a Coke) and chatted. This was the main reason I enjoyed DevConf and one reason I really wish I could make one of the Flock conferences.

So where does this leave us? Red Hat does have a large influence on Fedora. It’s not a conspiracy, it’s life, and attacking Red Hat employees because of its influence is counterproductive.

So, going back to Richard’s original message, we need to stop tearing each other down. When people speak, let’s assume good faith, and not assume that any ideas we disagree with will spell the end of Fedora, Linux or the world as we know it. Most of all, we need to make a conscious choice to value each other, even when we disagree.

Have a great 2015!