On a Sunday a few weeks ago, I finally decided to take the plunge and install the Fedora 15 Alpha on my primary workstation. I’ve been using GNOME Shell pretty much exclusively since Fedora 13, and I was looking forward to an even cleaner setup as it got closer to its first official release. The installation went smoothly, and, soon enough, I had the new interface up and running, and, I have to say, it’s looking great!

Just a quick aside to say that I really appreciate where GNOME Shell is going. I love the favorite apps/running apps on the left, desktops on the right concept. I do miss having persistent desktops (Firefox is always alone in the first desktop, and it’s a bit of a pain to get it back there when I restart Firefox). It’s also much harder to get to my files since the recent documents list on the left has gone. But though it’s taken some getting used to, the notifications menu on the bottom has ended up being really nice.

Anyhow, on Monday after the installation, I brought my laptop to my office in the morning, booted it up, and then went off to teach my first two classes. When I returned to my office, my screen had a nice kernel panic on it saying something about sda write errors, unable to write to disk, the end of the world, etc.

Being the incredibly sophisticated hacker (read “complete idiot”) that I am, I proceeded to do a hard reboot of the laptop without taking a picture of the screen. Oops. Then, on the reboot, the system ran into a small problem. It wasn’t able to mount any of my filesystems. A quick reboot into my livecd and one e2fsck later, and my system partition was back up again (apparently with no major errors). Unfortunately, that wasn’t the end of the story.

I should probably take this opportunity to explain my incredibly cunning partition setup here. You see, I have a boot partition, two 20GB system partitions that I switch between every time I update my system, and a swap partition. I then have a 400+GB encrypted home partition, a btrfs encrypted home partition, created back in the F13 days. This home partition contains all of my data. In the world. Everything.

So I booted from my now repaired system partition and… my home partition refused to mount. It said the filesystem was unrecognizable. Oookay. How about a btrfsck /dev/sda5. “No valid Btrfs on /dev/sda5.” That can’t be good. Ok, I’m a system administrator, I should have a good backup somewhere. Check around, and there it is, dated… May 8, 2010. Well, that sucks.

This was the point where I started to get slightly worried. I hopped onto the #btrfs IRC channel on freenode and that’s where my bacon was saved. Apparently whatever caused the kernel panic also caused some major problems when btrfs was writing its metadata. Unfortunately, since I had no record of the panic except my spotty memory, we weren’t able to track down the cause. All we knew was that the primary superblock had been corrupted, and that was why none of the tools could read it.

At this point, Chris Mason (the creator of btrfs) walked me through compiling btrfs-progs from the “next” branch in git, and then compiling btrfs-select-super (which isn’t built using the normal Makefile). I used btrfs-select-super to switch to the second superblock, and voilà, I was able to mount the filesystem (read-only, of course)! Some of the metadata was pointing to junk, and I ended up losing all my files that had been changed in the last few days, but most of them were emails, which I did have backed up elsewhere.

I still don’t know what caused the problem. There were some SMART errors on the drive, but repeated extended offline scans found no errors, and manually overwriting the entire partition using dd and then reading it found nothing amiss. There was some talk of it possibly being related to luks, but no evidence pointing in that direction.

So, now I’m running Fedora 15 Alpha, with a newly created encrypted btrfs filesystem as my home partition… and daily backups. A huge thank you to Chris and the others on #btrfs on freenode who gave me such great help!

Steep cliff credit: Steep cliff by Rob Lee. Used under CC BY-ND

Deltarpm is brittle. When it works correctly, it’s brilliant. But, like a tightrope walker crossing the Niagara falls while balancing an egg on his head, all it takes is one slip and…*splat*.

At the beginning of the Fedora 15 release cycle, a new version of xz was pushed in which the defaults for compression level 3 were changed (as far as I can tell, to what used to be level 4). This doesn’t cause any problems for newly compressed data, but if you decompress an rpm whose payload was compressed using old level 3 (like makedeltarpm does) and then recompress it with new level 3 (like applydeltarpm does), the compressed files no longer match. *Splat*.

I wrote about the root problem here over a year ago, but to summarize: almost no compression algorithms ever guarantee that, over all releases, they will create the same compressed output given the same uncompressed input.

Our fix for Fedora 15 was pretty simple. Delete all of the old deltarpms in Rawhide. As long as the users have the new xz before doing a yum update, all new deltarpms will work correctly. Yay.

The problem is that this is all still extremely fragile. Take Fedora bugs #524720, #548523, and #677578 for example. All three bugs have cropped up because of mistakes in handling changes in the compression format, and it’s all a bit ridiculous. Would anyone use gzip if an old version couldn’t decompress data compressed with a newer version?

A possible solution?

There is no simple solution. So what if we change the rules? Instead of trying to keep the compression algorithms static, what if we stored just enough information in the deltas to recompress using the exact same settings, whatever they are.

For gzip, this would mean recording things like each block size, dictionary, etc. For xz, it would mean recording the LZMA2 settings. The problem is that this information is different for each compression type and the functions to extract the needed information haven’t been included in any compression libraries (to my knowledge).

However, if we could write these functions and get them into the upstream libraries, it would benefit all programs that try to generate deltas. Deltarpm would continue to work when compression algorithms change. Rsync could actually delta gzipped files, even if the “–rsyncable” switch hasn’t been used in gzip.

There are a couple of possible problems with this solution. First, I’m not sure how big the extra needed information is. Obviously, for each compression format, it’s different, but, unless it’s at most 1/100th the size of the uncompressed file, storing the extra data in the deltarpm will probably not be worth the effort.

Second, no code has actually been written. In an open source world of “Show me the code”, this is obviously a major issue. I’d love to do a reference for one of the simpler compression formats (like zlib), but just haven’t had the time yet.

Obviously, the best solution would be for the various upstreams to provide the necessary functions, as they understand both their algorithms and what information should be stored. However, most upstreams have enough on their plates without needing extra stuff thrown in from random blogs.

Another good solution would be for someone who is interested in deltas and compression to take on this project themselves. Any volunteers? 🙂

Broken eggs credit: Broken Eggs by kyle tsui. Used under CC BY-NC-ND