2020–09–14: Putting 13 PinePhone distributions on a 8GiB uSD card

This is a summary of how I was able to make a 5 GiB multi-boot 13-distro image, that easily fits on a 8GiB SD card, and can be used to test-drive most of the OSes that are available for Pinephone today.

Starting with a 32GiB image and an old-school approach

At first I did the obvious, and started with traditional partitioning. I created 10 partitions on a 32GiB SD card, and was barely able to fit 9 Linux distributions into that image. Each paritition needed its own filesystem and some free space to allow for proper operation of the OS. It was also hard to mange. Each time I needed to change anything, I had to figure out which partition held which OS, mount it, fix things and unmount again.

It was not good.

The breakthroug came with a realization that I can mount subvolume in btrfs as if it was a filesystem in its own right. This allowed me to have just a single partition with a btrfs filesystem on the SD card, and have each distribution contained in its own subvolume. This way, all distributions could share from the same pool of free space for the data. There was no longer any need to plan partition sizes for each distro. This was the first major win.

Subvolumes are also much easier to manage than partitions. Need a new subvolume? Just create one and copy some files to it. Don't need the subvolume anymore? Just delete it.

Subvolumes appear as normal directories in the filesystem when the root subvolume is mounted, so one mount operation is all that's needed to have access to files of all included Linux distributions.

I also used another feature of subvolumes to debug startup issues in some distributions. Snapshotting. It's possible to create a snapshot of a current state of any of the distributions indivudally, and restore it in the future. Ubuntu Touch gave me a headscratch, when it didn't boot during the first boot, but it did on the second one. So I made a snapshot of the initial state of the filesystem, and kept comming back to it and tweaking it until I found the reason. (One of the boot scripts checked for the presence of a file named userdata/.writable_image and if it was missing it tried to reboot the system and failed, crashing the lightdm process.)

In fact, the latest multi-boot image keeps the initial state of all included distributions in a snapshot, so it's possible to selectively restore state of any of the distributions to the original.

Compression

Looking at btrfs, I found that it supports transparent compression for the stored files. Enabling zstd compression and decompressing rootfs tarballs of 9 distributions resulted in a filesystem that used 5.8 GiB of space with a compression ratio of 50%. This was exciting, because it looked like having a 8 GiB SD card image will be possible.

Using compression is also good for antoher reason. SD card access in Pinephone is limited to 24MiB/s max. At this speed, 4 core Cortex-A53 CPU can easily keep up, and this makes the loading of data from the SD card faster.

Some interesting features of COW filesystems

So far the optimization steps were fairly trivial. Just enabling features of existing filesystem and using them in a smart way.

Getting from 5.8GiB image with 9 Linux distributions to a 5 GiB image with 13 Linux distributions was harder. It was likely that those distributions have a lot of files in common. I made a simple test, and found that there's a space for further savings in the range of 12–13% if I could make the filesystem share data for duplicate files among the distributions.

COW filesystems, like btrfs, can sometimes allow to share file data among multiple files without resorting to hardlinking. In fact, there's now a generic VFS-level Linux API to make a filesystem share file data between files if it supports this feature.

Side note: Hardlinking would not be a great solution for reducing file data duplication in the image anyway, because it would have meant that if user changed a file in Ubuntu Touch, the change might be visible in other distros that may share that file.

The next hurdle was figuring out how to effectively use this API in my specific scenario. There are tools that allow scanning the existing filesystem, search for duplicities and use this API to remove them. Then it would be possible to scrub the filesystem, discard empty space, and compress the resulting block device image for easy re-distribution. I checked a bunch of those tools, and they seemed exceedingly complicated, or buggy when used on btrfs subvolumes.

Instead I decided to write a extraction tool using libarchive that takes multiple tarballs on the input (one per Linux distribution) and decompresses them while keeping track of content of already extracted files and using the above mentioned FICLONE API to share data whenver it finds file that was already extracted before. It does deduplication on the fly, so there's no need to apply any further cleanup steps to the filesystem afterwards. This approach to deduplication is also very fast and efficient, because there's no extra IO necessary.

At this point I tried to add 4 more Linux distributions to the image. I did this just to stress test the new extraction tool. In particular, I added several variants of postmarket OS, which were bound to have a lot of the files in common.

The resulting image had the same size as the previously made 9 distribution variant.

Final space savings

The final opportunity for size optimization was a cheap one. Remove files that are not used/needed. I didn't want to tweak the distributions too much, so that they are as close to their official state as possible. I had to make one change, though. I was not able to use distributions' own official kernels, because neither had a driver for btrfs built in. I also was not very fond of supporting outdated EOLed kernels many of the distributions use, or re-building them. So I used my own kernel and made all the distributions share it.

Due to this I was able to remove the modules, firmware, and kernels that distributions package themselves. With 13 distributions this led to another 0.8GiB of saved space.

The result is a 5 GiB 13-distro image, that easily fits on a 8GiB SD card, and can be used to test-drive most of the OSes that are available for Pinephone today.

The image also serves as a demo of the GUI bootloader I wrote for Pinephone.