megi's PinePhone Development Log RSS

2020–09–20: Downsizing the multi-boot image

If the multi-boot image doesn't fit your 8GiB uSD card, because it's a tad too big, you can downsize it a bit by using this script:

#!/bin/bash

set -e -x

mkdir -p m
L=`losetup -P --show -f multi.img`
mount -o compress-force=zstd ${L}p2 m
btrfs filesystem resize 7000M m
echo ",7000M" | sfdisk -N 2 ${L}
umount m
losetup -d "$L"

truncate -s $((128+7000))M multi.img

If something fails in the middle, you may need to recover by calling umount and losetup -d yourself. losetup -l can tell you if the image is still exported as a loop device, and which one.

The image will have the size of 7128 MiB after resize and this should fit more „8“ giga something uSD cards, as there are obviously some other giga units than gigabyte and gigibyte used by some manufacturers.

2020–09–20: Some ways to improve Pinephone safety

This is a follow up on some issues from the previous article. On surface, solutions to some of the previously presented issues can seem simple. Toggle a few registers in PMIC, and we're mostly done. Trouble is that safety mechanisms are barely ever triggered, by definition. Safety events occur rarely. That means that the mechanisms are not regularly tested, and it is not known that they work.

Also it's not clear which code's responsibility fixing the issues should be. Bootloader, or kernel, or userspace? Finally, there are a bunch of devils in the details, that complicate the upstreamability of any solution. And having fixes upstream is necessary to make sure they reach the most users.

Necessary minimum

Nevertheless, at least enabling some pre-existing PMIC functionality blindly is better than nothing, so that's precisely what I decided to do in p-boot. It's the easiest place to start resolving these issues for me personally, and for other p-boot users.

I fixed two issues:

I didn't measure the 3kOhm NTC used in Pinephone battery and third party batteries I bought. I just used a table from some random 3kOhm NTC spec on The Internet, that seemed like it could match. Hopefully it's close enough.

Trouble with the second fix is that it's a hard power cut-off, so data loss may occur when PMIC overheats. There are three fixed temperature levels in AXP803. On level 1 the charging is limited, on level 2 the interrupt is sent to SoC, on level 3 the PMIC shuts down if configured to do so (by default it keeps running, and this is what my p-boot fix changes). Ideally, the crossing of level 2 would be handled by Linux to make it safely shut down the system, and level 3 forced power cut-off would never happen. Arguably, if charging was source of the heating, crossing level 1 will lead to resolving the issue, so the next level will not be reached.

Suggested fixes elsewhere

These fixes will reach a very limited audience. It would be nice to have these fixes in U-Boot too, but that's not possible at the moment, because U-Boot doesn't have access to PMIC.

Other places to put the fix is to ATF or Linux kernel. That can reach more people faster, but there would have to be some generic mechanism to make the fix upstreamable, otherwise it will not reach people using the mainline Linux kernel or mainline ATF.

There are some ways to approach this:

First, the most generic solution would be to have a description of the battery in DT describing the Pinephone. Sadly, the current bindings don't include battery temperature limits.

Also converting from temperature to NTC resistance (which is necessary to determine the code word from ADC for the limits used by the temperature monitor logic in PMIC) is not straightforward. It is usually defined in NTC datasheet as a table. Do I have NTC datasheet? No. I bought the batteries online from some mobile phone service shop.

There are also equations that approximate the temperature – resistance relationship for the NTC, which could be used instead of a fixed table, if one knows the relevant coefficients. These can be calculated after measuring the NTC's resistance at a few temperature points when we lack the datasheet.

So generic solution may look like this:

Kernel also has support for NTC devices, so maybe NTC can be described outside of the battery node (even though it's part of the battery).

This may all fail to be upstreamed on one thing: the battery is user swappable, so it's arguably not part of the Pinephone, and describing it inside the pinephone DT will not be appropriate.

I don't have any plans implementing any of the above, atm. Maybe with the exception of adding a 4th approach to the fix to my Linux kernel (the easiest one ;)). I'd like to work on my multi-boot image. So these are mostly pointers for somebody else who'd like to tackle this.

Other issues

Fast charging is not necesary in many situations, so having it as a default is not great. User should be able to decide if he wants to trade off slower, safer charging and battery longetivity over speed. This tradeoff can be realized in many ways.

All this is already controllable from userspace via sysfs. Ideally there would be some charging monitoring daemon that would take into account users's wishes and select proper strategy for charging, based on preference for battery longetivity or speed.

There are several trade offs the deamon would be able to handle:

All this is decision making that doesn't belong to the kernel.

Similar daemon could monitor power usage of the phone and try to limit it to safer levels, or warn the user if that's not possible.

2020–09–18: Let's talk about safety of Pinephone

My gf read me some articles about exploding phones today. :) I think there needs to be some serious conversation about Pinephone safety. Safety needs to become an important concern now, when more and more people are getting their Pinephones every month. It's just a matter of time before the first major safety incident hits this community, and it may be more than just a hacked store. It's just a numbers game.

Pinephone is an interesting device in one way. You can run whatever software you like on it (and you do!), and this software comes almost universally with zero guarantees. Read the license to any of the program you run on your Pinephone and it will almost certainly tell you:

THIS SOFTWARE IS PROVIDED ``AS IS'' AND WITHOUT ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.

or

THIS SOFTWARE IS PROVIDED BY <COPYRIGHT HOLDER> AS IS AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL <COPYRIGHT HOLDER> BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES

or

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

etc.

In case of Pinephone you have to take these warnings very seriously, because this software is not provided by the manufacturer (Pine64), and as far as I know, there's no software related safety testing going on at all.

Some skeletons, hiding at the lower levels…

I'll give you a few reasons why things may not be so rosy, when it comes to safety.

There's no unchangeable well tested guardian angel management engine that safely manages battery, power supplies, thermal behavior, that is provided by the manufacturer, and that is independent of the operating system.

Pinephone's SoC is quite bare when it comes to software/firmware (that's why FOSS enthusiasts like it, no blobs, you know!). This has a dark side, too. All the safety critical parts are written (or rather were not written, yet) by some random people on The Internet.

You can already choose among more than 10 Linux distributions to run on your Pinephone. How do you know any one of these is safe to run on your Pinephone?

How well safety critical parts of Pinephone function (or if at all), depends on how well Linux distribution vendors understand the platform, and how well they test its safety features. Pinephone safety depends on the software they do put together, afterall.

Is device safety even a thing distribution vendors test for or plan for at the moment? I don't know, but I doubt it.

For many months pretty much all distributions used misconfigured „official“ kernel (it's a meaningless moniker, btw, pine64 doesn't do software), that didn't regulate CPU temperature at all. It could go up as high as the thermals of the surrounding environment allowed. There were indications something is wrong. People reporting their phones felt too hot, displays showing burn artifacts, temperature indicator in sysfs returning error. Regardless, it persisted for many months, until I discovered the root cause while helping one user with thermal issues on her Pinephone.

Maybe I'm a bit biased, because I was one of the people working on the sunxi thermal driver over the years, and thus understand this element of the SoC quite well, but I was surprised nobody figured out such a serious issue, or realized what it meant. So let's go through some other issues that I'm aware of, of which other people may not be:

1. Pinephone battery uses a 3 kOhm NTC to monitor the temperature. Power management chip in Pinephone expects 10 kOhm variant by default. So early on, when the times were adventurous, someone decided to patch the kernel to disable battery thermal monitoring completely. Quick and dirty fix for Pinephone not charging due to false under-temperature alarm.

Now guess what… up to now, all distributions run with battery temperature sensing and regulation disabled. If you're unlucky and use a dud battery that will heat up more easilly during fast charging, you can burn down your house.

There will be nothing to stop it going past 50, 60, 70, 600°C. Phone will happilly provide current to the battery until it explodes and burns. Unlikely scenario for any single user, but this safety mechanism that's present on regular customer phones is missing here, just because nobody cared to configure it, yet.

2. And you know what, I also suspect that all distros fast charge all the way through the constant-current phase of charging, by default. Other than contributing to making the above mentioned house burning scenario more likely to happen, this also contributes to overheating issues on Pinephone.

3. Another thing. PMIC has an emergency thermal shutdown feature, for a situation when the chip itself overheats. It's disabled by default. It's also not well documented. ¯\_(ツ)_/¯

4. And, another one! Battery is rated for some sustained continuous discharge current (0.5C, 1500mA, ~6W). I guess it overheats if it is dicharged at a significanlty faster rate for prolonged periods of time, and potentially becomes a safety hazard again. Maybe again only if it's a suboptimal piece that passed QA.

Now, Pine64 sells convergence edition Pinephone meant for use with a dock and a monitor. When you connect the phone to the dock with monitor connected over HDMI, power consumption jumps by 4W. That alone is close to this specified limit of the battery. Baseline power consumption is 2.8W (with phone display on). Now if you actually use the phone the CPU alone can add further 2.5W during full load (likely, when trying to use the pinephone as a desktop machine). That is >9W total. That is well over the specified safe limit. Of course, the rest of the phone heating up a lot from this huge power draw don't help the at all, either.

So it's quite possible for the user to load the battery well out of spec, with just normal use. This needs to be managed somehow.

And those are just things that I'm aware of.

So what…

I'm trying to be a bit inflamatory here, to start the conversation. Nevertheless, the above issues are real. It's really just a numbers game. When the distros will not take safety seriously, by planing for it, testing for it, and verifying the mechanisms they are supposed to ensure are in place for the safety of their users, the odds somehting will happen will stay needlessly high.

Also, don't be a person whose house burns down for FOSS. Ask your favorite distribution's authors what they're doing to make their OS safe. Pine64 itself can only go so far to ensure safety of Pinephone, software you put on your Pinephone matters a lot, too!

2020–09–17: Video acceleration experiments with PinePhone

Recently, gstreamer gained support for utilizing v4l2-requests API, for h264 acceleration, so I compiled it for Arch Linux, patched the kernel with the help of ndufresne from #cedrus to fix kernel panics, figured out how to make gstreamer play to kms, and made a bunch of experiments with these results:

https://www.youtube.com/watch?v=dHOgVmxH_dA

https://www.youtube.com/watch?v=6P76lQX70iI

https://www.youtube.com/watch?v=yyRm8kccyG4

External display video playback, even to a fairly large display works very nicely. Quite a feat for an entry-level phone. :)

CPU utilization during playback is miniscule. Comparing power consumption, between idle and video playback, the video decoding takes just additional 0.2W of power, compared to CPU based decoding which could take up to 2.4W.

There's an issue with internal display being locked to 36 FPS, which makes playback of 30FPS video a bit choppy.

I also hacked my kernel to workaround gstreamer kmssink bugs, to test playback to an external 1440p@60fps monitor. 1440p@30 video decoding works fine. 1440p@60 video is too much, and drops frames. 1080p@60 video decodes and scales to 1440p fine, too.

Another issue is a heavy power consumption (4W) from the dock when HDMI output is enabled. Also dock's display output doesn't seem to work when powering the dock from the USB-C power supply.

Overall, it's nice. :)

2020–09–16: PineBook Pro and Levinboot again

Crystalgamma released Levinboot 0.7.2 yesterday so I decided to update, and try to make my Pinbook Pro boot times even faster. With his help I found that I accidentally disabled optimizations in my build and that was the reason for 4s decompression times for my kernel/initrd payload, and some stack overflows in levinboot's decompression code.

Unoptimized code turns out to be really slow. What a surprise.

I switched to lz4, enabled -O3, and now levinboot decompression and load times changed from 4s to 600ms. Significantly better! :)

With the latest levinboot my Pinebook Pro Arch Linux setup lets me to enter LUKS password in 4s since I start pressing the power button:

Perfect!

2020–09–14: Putting 13 PinePhone distributions on a 8GiB uSD card

This is a summary of how I was able to make a 5 GiB multi-boot 13-distro image, that easily fits on a 8GiB SD card, and can be used to test-drive most of the OSes that are available for Pinephone today.

Starting with a 32GiB image and an old-school approach

At first I did the obvious, and started with traditional partitioning. I created 10 partitions on a 32GiB SD card, and was barely able to fit 9 Linux distributions into that image. Each paritition needed its own filesystem and some free space to allow for proper operation of the OS. It was also hard to mange. Each time I needed to change anything, I had to figure out which partition held which OS, mount it, fix things and unmount again.

It was not good.

Swithching filesystems and sharing free space

The breakthroug came with a realization that I can mount subvolume in btrfs as if it was a filesystem in its own right. This allowed me to have just a single partition with a btrfs filesystem on the SD card, and have each distribution contained in its own subvolume. This way, all distributions could share from the same pool of free space for the data. There was no longer any need to plan partition sizes for each distro. This was the first major win.

Subvolumes are also much easier to manage than partitions. Need a new subvolume? Just create one and copy some files to it. Don't need the subvolume anymore? Just delete it.

Subvolumes appear as normal directories in the filesystem when the root subvolume is mounted, so one mount operation is all that's needed to have access to files of all included Linux distributions.

I also used another feature of subvolumes to debug startup issues in some distributions. Snapshotting. It's possible to create a snapshot of a current state of any of the distributions indivudally, and restore it in the future. Ubuntu Touch gave me a headscratch, when it didn't boot during the first boot, but it did on the second one. So I made a snapshot of the initial state of the filesystem, and kept comming back to it and tweaking it until I found the reason. (One of the boot scripts checked for the presence of a file named userdata/.writable_image and if it was missing it tried to reboot the system and failed, crashing the lightdm process.)

In fact, the latest multi-boot image keeps the initial state of all included distributions in a snapshot, so it's possible to selectively restore state of any of the distributions to the original.

Compression

Looking at btrfs, I found that it supports transparent compression for the stored files. Enabling zstd compression and decompressing rootfs tarballs of 9 distributions resulted in a filesystem that used 5.8 GiB of space with a compression ratio of 50%. This was exciting, because it looked like having a 8 GiB SD card image will be possible.

Using compression is also good for antoher reason. SD card access in Pinephone is limited to 24MiB/s max. At this speed, 4 core Cortex-A53 CPU can easily keep up, and this makes the loading of data from the SD card faster.

Some interesting features of COW filesystems

So far the optimization steps were fairly trivial. Just enabling features of existing filesystem and using them in a smart way.

Getting from 5.8GiB image with 9 Linux distributions to a 5 GiB image with 13 Linux distributions was harder. It was likely that those distributions have a lot of files in common. I made a simple test, and found that there's a space for further savings in the range of 12–13% if I could make the filesystem share data for duplicate files among the distributions.

COW filesystems, like btrfs, can sometimes allow to share file data among multiple files without resorting to hardlinking. In fact, there's now a generic VFS-level Linux API to make a filesystem share file data between files if it supports this feature.

Side note: Hardlinking would not be a great solution for reducing file data duplication in the image anyway, because it would have meant that if user changed a file in Ubuntu Touch, the change might be visible in other distros that may share that file.

The next hurdle was figuring out how to effectively use this API in my specific scenario. There are tools that allow scanning the existing filesystem, search for duplicities and use this API to remove them. Then it would be possible to scrub the filesystem, discard empty space, and compress the resulting block device image for easy re-distribution. I checked a bunch of those tools, and they seemed exceedingly complicated, or buggy when used on btrfs subvolumes.

Instead I decided to write a extraction tool using libarchive that takes multiple tarballs on the input (one per Linux distribution) and decompresses them while keeping track of content of already extracted files and using the above mentioned FICLONE API to share data whenver it finds file that was already extracted before. It does deduplication on the fly, so there's no need to apply any further cleanup steps to the filesystem afterwards. This approach to deduplication is also very fast and efficient, because there's no extra IO necessary.

At this point I tried to add 4 more Linux distributions to the image. I did this just to stress test the new extraction tool. In particular, I added several variants of postmarket OS, which were bound to have a lot of the files in common.

The resulting image had the same size as the previously made 9 distribution variant.

Final space savings

The final opportunity for size optimization was a cheap one. Remove files that are not used/needed. I didn't want to tweak the distributions too much, so that they are as close to their official state as possible. I had to make one change, though. I was not able to use distributions' own official kernels, because neither had a driver for btrfs built in. I also was not very fond of supporting outdated EOLed kernels many of the distributions use, or re-building them. So I used my own kernel and made all the distributions share it.

Due to this I was able to remove the modules, firmware, and kernels that distributions package themselves. With 13 distributions this led to another 0.8GiB of saved space.

The result is a 5 GiB 13-distro image, that easily fits on a 8GiB SD card, and can be used to test-drive most of the OSes that are available for Pinephone today.

The image also serves as a demo of the GUI bootloader I wrote for Pinephone.

2020–09–11: Adding postmarket OS to multi-distro image

So I finally tried the pmbootstrap tool to create a bunch of rootfs tarballs for several postmarket OS UI variants: GNOME, Phosh, Plasma Mobile, fbkeyboard.

I've added them all to my multi-distro image without any increase in size since the previous public build of the image. So now it's a 13 distro multi-boot image with pretty much every major and a bunch of minor PinePhone supporting distributions, and it still fits on a 8GiB SD card!

2020–09–11: Ways to help improve Pinephone kernel

There are a bunch of open questions about Pinephone's HW. When answered, these could help improve the kernel behavior. Here's one of them.

Minimal display brightness is too bright

This gets asked quite a bit, but just maybe 2 people helped do anything about it, so here's a more detailed guide on how to help.

Sadly fixing this issue requires gathering a lot more samples than 2, regardless of whether your Pinephone has the problem or doesn't. Trouble is that if only people who have this issue do the following test, the results will be useless. Even people who's backlight seems to work correctly need to take part. So it's a hard problem to solve. :)

If enough people report at what brightness level the backlight turns on on their Pinephone, we could improve the default minimum brighness level, so that most users can reduce their Pinephone screen brightness to lower levels.

There's an open question on how to set the backlight brightness values on post 1.0 revision phones, since lower PWM duty cycles lead to backlight being basically off. It would be nice if more people can test the various backlight levels on 1.1 and 1.2 revision with this change in dts in the backlight node:

/backlight {
        brightness-levels = <0 1000>;
        num-interpolated-steps = <1000>;
};

The above change can be made by fdtput, which is part of dtc package:

fdtput -t u your.dtb /backlight brightness-levels 0 1000
fdtput -t u your.dtb /backlight num-interpolated-steps 1000
fdtput -t u your.dtb /backlight default-brightness-level 500

You don't need to recompile the kernel or anything. Just identify what dtb file is your OS using (it will be in /boot directory) backup it up and apply the above commands to the original (just replace yout.dtb with the name of your dtb file).

This change will make it so that PWM backlight driver sets up 1000 linear backlight steps, so you can enter values 1–1000 to /sys/class/backlight/backlight/brightness and these are directly mapped to PWM duty cycle. So level 1 is 0.1% level 100 is 10%, and so on.

What's improtant is to report:

2020–09–11: PinePhone multi-boot image deduplication tool complete

Since the last post, I was able to create the deduplicating tarball extractor tool exactly as I described it, and everything seems to work as expected.

The image size after compression shrank from 5.8 GiB to 5.2 GiB. :)

Now I have to figure out how to build a few different postmarket OS rootfs variants, to give this shiny new tool a real stress test.

I'm pretty sure I can add maybe 3 postmarket OS mobile UI variants without increasing the image size significantly. After all, the image already contains sxmo, which is based on pmOS, so most of the base data is already in the image.

Then, I'll implement p-boot DTB auto-selection, and the image will be ready to enter beta state.

New kernels

I've released new 5.9 kernels with a few more new tweaks:

2020–09–10: PinePhone multi-boot image deduplication

Btrfs has a nice feature where you can tell it when creating a file to initially use content of some already existing file. This saves space on the filesystem, because the duplicit files can share the same data space without needless data copies. This is almost like hard links, except that data stop being shared, when the file is written.

This can be used in the multi-boot image to save even more space. The multi-distribution image offers many opportunitites for this kind of space saving, due to distributions sharing quite a lot of identical files.

I've written a tool that takes all the rootfs tarballs on input, and goes through all of them, finds duplicities, and calculates space used in total by all files, and space used by unique files. For the current multi-distro image, the numbers look like this:

So the needless overhead of duplicate files accross all included distributions currently is 12.6%, btrfs compression ratio is 50% and the wasted space on the compressed filesystem by the duplicate files is 762 MiB.

That's fairly significant. I can fit another distribution in the wasted space alone. :)

If the included distributions would be much less diverse. For example when including multiple UI variants of postmarket OS, the savings could be even greater. I can potentially add 3 to 4 variants of pmOS if I was able to share duplicate files content, without the current image size changing at all.

I'll try to add FICLONE support to my extraction tool. The tool will basically put content hashes of the files and their paths into a hash table as they are being extracted, and if there's a duplicity, instead of extracting the content of a file, it will use FICLONE to share the data of previously extracted copy of the file.

Resizing the partition after flashing the image

To get more free space when using the multi-image, you can resize it to the full size of your SD card. You can do this from a PC using a SD card reader.

Warning! You'll need to run all the commands as root.

Multi-distro main data partition block device can be found with:

blkid -lt PARTUUID="12345678-02"

For me it prints:

/dev/mmcblk0p2: UUID="0cb50b0b-77a3-45bd-a605-857472b88281" \
   UUID_SUB="a1d75445-1ffe-4e6f-860a-1acdbb4aad2a" \
   BLOCK_SIZE="4096" TYPE="btrfs" PARTUUID="12345678-02"

So for me, btrfs partition is /dev/mmcblk0p2 and the device containing the partition table is /dev/mmcblk0. For you it may be different. Make absolutely sure you're using the correct block device!

Resize the main partition

First, dry run:

echo ", +" | sfdisk -n -N 2 /dev/mmcblk0

Outputs:

Quote:
Disk /dev/mmcblk0: 29.74 GiB, 31914983424 bytes, 62333952 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x12345678

Old situation:

Device        Boot  Start      End  Sectors  Size Id Type
/dev/mmcblk0p1 *      8192  262143  253952  124M 83 Linux
/dev/mmcblk0p2      262144 15359999 15097856  7.2G 83 Linux

/dev/mmcblk0p2:
New situation:
Disklabel type: dos
Disk identifier: 0x12345678

Device        Boot  Start      End  Sectors  Size Id Type
/dev/mmcblk0p1 *      8192  262143  253952  124M 83 Linux
/dev/mmcblk0p2      262144 62333951 62071808 29.6G 83 Linux
The partition table is unchanged (--no-act).

Now, verify changes look sane, and do a real resize:

echo ", +" | sfdisk -N 2 /dev/mmcblk0

Resize the main filesystem

Now that the partition is stretched to the end of the SD card, let's resize the contained filesystem too. Btrfs needs to be mounted during resize. The steps thus are:

mount /dev/mmcblk0p2 /mnt
btrfs filesystem resize max /mnt
umount /mnt
df -h

You should see:

Resize '/mnt' of 'max'
/dev/mmcblk0p2  30G  6.6G  24G  23% /mnt

And that's all. :)

Forum post

I've posted about my image on the pine64.org forum. That post contains some additional information:

https://forum.pine64.org/showthread.php?tid=11347

2020–09–08: PinePhone multi-boot finishing touches / modem improvements

So I dealt with a few of the „annoying details“ from the last post, and created a repository for my build scripts and overrides for each distro.

The last thing to do prior to final testing is to add support for selecting dtb for PP 1.1 or 1.2 automatically.

Modem power manager

I've also improved my modem power manager a bit, and added support for fast poweroff mode that's available in the newer modem firmwares. (I believe it's those firmware that are in Braveheart and onwards.) Modem can now be powered off in ~2 seconds.

I've also fixed issues with RDY not being received. Looks like one of the persistent configuration options makes the modem not send RDY. I just poll for the successfull empty AT command, as a way to detect when the modem is ready, and that seems to be more reliable.

All these changes are avialable in my 5.8 and 5.9 kernel branches.

2020–09–08: PinePhone multi-boot image optimizations

Since I'll be using my kernel for all the distros anyway, I thought I might spice it up a bit, and use something a bit more modern than ext4.

Instead of complicated partitioning scheme that wastes space and is not easy to optimize, I decided to use a single btrfs filesystem for everything with each distribution having its own subvolume, and all files being compressed by zstd. This made it possible to create a 8GiB SD card image, with 9 distributions on it.

The image creation process is now much simpler. First I create a distros/ directory with a subdirectory for each distribution. Then I download and prepare rootfs.tar.zst tarballs, for each distribution I want to have in the image, as previously described in this blog.

And then I create the image:

#!/bin/sh

if [ "$(whoami)" != "root" ] ; then
        exec sudo sh "$0" "$@"
fi

set -e -x

# make sure image fits on a typical 8GB SD card

rm -f multi.img
truncate -s 7000M multi.img

sfdisk -W always multi.img <<EOF
label: dos
label-id: 0x12345678
unit: sectors
sector-size: 512

4M,124M,L,*
128M,,L
EOF

L=`losetup -P --show -f multi.img`

mkfs.btrfs ${L}p2

mkdir -p m
mount -o compress-force=zstd ${L}p2 m

for ddir in distros/*
do
        name=${ddir#distros/}
        btrfs subvolume create m/$name
        bsdtar -xp --numeric-owner -C m/$name -f $ddir/rootfs.tar.zst
done

./mkimage-apply-fixes.sh

umount m
losetup -d "$L"

./mkimage-boot.sh

This does away with the need for having to deal with partition numbers. Subvolumes have the name of the distro, and are easily referenced in the boot arguments.

During boot I just mount the appropriate subvolume instead of the main volume of the btrfs filesystem. All that's needed is to pass rootfsopts=subvol=distro-name.

Modifying and accessing files in all distributions at once also becomes trivial. All I need to do is to mount the main volume, and the subvolumes for each partition appear nicely available there as subdirectories in the main volume. Previously I had to juggle with mounting 10 arbitrarily named partitions.

It's also very easy to add/remove distributions without the need for re-partitioning, and all distributions share the remaining free space on the SD card.

I find this setup excellent! Btrfs partition with 9 simultaneously installed distributions takes just 6.5GiB of space. Previously just mobian alone took 3GiB of used space, and a 4 GiB partition.

Making the image bootable

To make the image bootable, I just build the boot.conf file for p-boot, based on a per-distribution config file, and use it to format the boot partition.

Config file looks like:

version=2020-09-10
name="Ubuntu Touch"
bootargs="logo.nologo vt.global_cursor_default=0"

And the script to format make the image bootable:

#!/bin/sh

if [ "$(whoami)" != "root" ] ; then
        exec sudo sh "$0" "$@"
fi

# loglevel=15
serial="console=ttyS0,115200 earlycon=ns16550a,mmio32,0x01c28000"
silent="quiet loglevel=0 systemd.show_status=false"
bootargs_base="$serial $silent cma=256M console=tty1 consoleblank=0 panic=3 rw rootwait root=PARTUUID=12345678-02 rootfstype=btrfs rootflags=compress-force=zstd,nodatacow,subvol"
kbuilds=../builds

(
        echo "device_id = Distro Demo Image 2020-09-14"
        no=0
        for ddir in distros/*
        do
                dist=${ddir#distros/}

                (
                        source ./$ddir/config
                        
                        echo "no = $no"
                        echo "  name = $name $version"
                        echo "  atf = ../p-boot/dist/fw.bin"
                        echo "  dtb = $kbuilds/ppd-5.9/board-1.1.dtb"
                        echo "  dtb2 = $kbuilds/ppd-5.9/board-1.2.dtb"
                        echo "  linux = $kbuilds/ppd-5.9/Image"
                        echo "  bootargs = $bootargs_base=$dist $bootargs"
                        echo "  splash = files/$dist.argb"
                )

                no=$(($no+1))
        done
) > boot.conf

set -e -x

L=`losetup -P --show -f multi.img`
../p-boot/.build/p-boot-conf-native . ${L}p1
../p-boot/.build/p-boot-conf-native . boot-part.img
losetup -d $L

dd if=../p-boot/.build/p-boot.bin of=multi.img bs=1024 seek=8 conv=notrunc

And that's really all there's to it, as far as making a bootable multi-distro image is concerned.

Some annoying details are still remaining

Support

If you'd like to support this effort, you can contribute at https://xnux.eu/contribute.html

2020–09–07: PinePhone multi-boot image boot testing

So let's say that distros are really finicky about where they're booted from if their initramfs image is used to boot them. Initramfs images mostly get in the way and don't offer much for the multi-boot image, so the simple solution is to not use them at all. All distros should be bootable fine without them.

The process should be quite simple: Linux mounts root partition provided via root= boot parameter at / and runs /bin/init. The distro itself should not care about what the real partition is, because it's already running from it. It should just not do anything stupid to the current rootfs mount, and just run.

Actually most modern distros should run fine with an empty /etc/fstab, in this situation.

Using my kernel

To have some unity/sanity, I decided to use my own kernel with all necessary drivers built in. This way I don't need to copy my kernel's modules to rootfs of each distro. I did also built firmware binaries into the kernel image, so it's all self-contained.

This approach has some other benefits:

Problems encountered so far

Other than this, each distro seems to boot fine with my 5.9 kernel.

The last step is figuring out how to make modem initialize properly on each distro. This will involve finding the script that powers up the modem and replacing it with echo 1 > /sys/class/modem-power/modem-power/device/powered.

See https://megous.com/dl/tmp/multi2.mp4 for another preview. :)

2020–09–05: PinePhone multi-boot image

I made a further progress on the multi-boot image:

The result is:

I have yet to boot test all the distributions. Prior to that, I have to check whether the distros are not doing something destructive on boot, like resizing or deleting partitions, etc.

So far my boot.conf for p-boot looks like this:

#!/bin/sh

#serial="console=ttyS0,115200 earlycon=ns16550a,mmio32,0x01c28000 loglevel=15"
bootargs="cma=256M console=tty1 consoleblank=0 quiet loglevel=1 panic=3 rw rootwait root=PARTUUID=12345678"

dists=/mnt/dists

#/dev/mmcblk0p2: PARTUUID="12345678-02"  mobian (f2fs, 5.7 kernel, PP 1.0-1.2)
#/dev/mmcblk0p3: PARTUUID="12345678-03"  KDE neon (ext4 only, 5.7 kernel, PP 1.1 only)
#/dev/mmcblk0p5: PARTUUID="12345678-05"  arch (f2fs, my 5.9 kernel, , PP 1.0-1.2)
#/dev/mmcblk0p6: PARTUUID="12345678-06"  sxmo (ext4 only, 5.7 kernel, PP 1.1, 1.2)
#/dev/mmcblk0p7: PARTUUID="12345678-07"  lune (ext4 only, 5.5 kernel, PP 1.1 only)
#/dev/mmcblk0p8: PARTUUID="12345678-08"  maemo (ext4 only, 5.6 kernel, PP 1.1 only)
#/dev/mmcblk0p9: PARTUUID="12345678-09"  ut  (ext4 only, 5.6 kernel, PP 1.1 only)
#/dev/mmcblk0p10: PARTUUID="12345678-0a" sailfish (ext4 only, 5.6 kernel, some custom DTB with partial support for up to PP 1.2)
#/dev/mmcblk0p11: PARTUUID="12345678-0b" pureos (f2fs, 5.8 kernel, PP 1.0-1.2)

zcat $dists/part2/boot/Image.gz > $dists/part2/boot/Image
#zcat $dists/part3/boot/vmlinuz-5.7.0-pine64-g3823929aa > $dists/part3/boot/Image
#zcat $dists/part8/boot/Image.gz > $dists/part8/boot/Image
#zcat $dists/part9/boot/vmlinuz > $dists/part9/boot/Image

cat << EOF
device_id = Multi-Distro Demo Image

no          = 0
  name      = Mobian
  atf       = fw.bin
  dtb       = $dists/part2/boot/dtb/allwinner/sun50i-a64-pinephone-1.1.dtb
  linux     = $dists/part2/boot/Image
  initramfs = $dists/part2/boot/initrd.img
  bootargs  = $bootargs-02 splash plymouth.ignore-serial-consoles vt.global_cursor_default=0
  splash    = files/mobian.argb

no          = 1
  name      = KDE Neon
  atf       = fw.bin
  dtb       = $dists/part3/boot/dtb
  linux     = $dists/part3/boot/Image
  initramfs = $dists/part3/boot/initrd.img-5.7.0-pine64-g3823929aa
  bootargs  = $bootargs-03 splash
  splash    = files/neon.argb

no          = 2
  name      = Arch Linux ARM
  atf       = fw.bin
  dtb       = $dists/part5/boot/dtbs/allwinner/sun50i-a64-pinephone-1.1.dtb
  linux     = $dists/part5/boot/Image
  initramfs = $dists/part5/boot/initramfs-linux.img
  bootargs  = $bootargs-05
  splash    = files/arch.argb

no          = 3
  name      = Sxmo
  atf       = fw.bin
  dtb       = $dists/part6/boot/sun50i-a64-pinephone-1.1.dtb
  linux     = $dists/part6/boot/vmlinuz-postmarketos-allwinner
  initramfs = $dists/part6/boot/initramfs-postmarketos-allwinner
  bootargs  = $bootargs-06 init=/init.sh PMOS_NO_OUTPUT_REDIRECT PMOS_FORCE_PARTITION_RESIZE pmos_root=/dev/mmcblk0p6
  splash    = files/sxmo.argb

no          = 4
  name      = Lune OS
  atf       = fw.bin
  dtb       = $dists/part7/boot/sun50i-a64-pinephone.dtb
  linux     = $dists/part7/boot/Image
  initramfs = $dists/part7/boot/initramfs-uboot-image-pinephone.uboot
  bootargs  = $bootargs-07 bootmode=normal LUNEOS_NO_OUTPUT_REDIRECT
  splash    = files/lune.argb

no          = 5
  name      = Maemo Leste
  atf       = fw.bin
  dtb       = $dists/part8/boot/allwinner/sun50i-a64-pinephone-1.1.dtb
  linux     = $dists/part8/boot/Image
  bootargs  = $bootargs-08 fbcon=rotate:1
  splash    = files/maemo.argb

no          = 6
  name      = Ubuntu Touch
  atf       = fw.bin
  dtb       = $dists/part9/boot/dtb
  linux     = $dists/part9/boot/Image
  initramfs = $dists/part9/boot/initrd.img
  bootargs  = $bootargs-09 systempart=/dev/mmcblk0p9 devnum=0 logo.nologo vt.global_cursor_default=0
  splash    = files/ut.argb

no          = 7
  name      = Sailfish
  atf       = fw.bin
  dtb       = $dists/part10/boot/sun50i-a64-pinephone-1.1.dtb
  linux     = $dists/part10/boot/Image
  bootargs  = $bootargs-0a
  splash    = files/sailfish.argb

no          = 8
  name      = Pure OS
  atf       = fw.bin
  dtb       = $dists/part11/boot/dtb/allwinner/sun50i-a64-pinephone-1.1.dtb
  linux     = $dists/part11/boot/Image
  initramfs = $dists/part11/boot/initrd.img
  bootargs  = $bootargs-0b init=/sbin/init splash plymouth.ignore-serial-consoles
  splash    = files/pureos.argb


EOF

2020–09–04: Pinebook Pro and Levinboot

Pinebook Pro is a very nice laptop. Software support just has a few annoying quirks:

Thankfully, a discord user CrystalGamma took to creating Levinboot, which is a specialized bootloader for Pinebook Pro and other similar RK3399 based computers. Its goals are almost identical to my p-boot project: boot the kernel as fast as possible from local storage and get out of the way. Its limitations are similar too. No or very limited support for partition schemes, no support for traditional filesystems, etc.

Version 0.7.1 was released a few days ago, so I decided to give it a try on my Pinebook Pro, and I'm happy to report that it works quite nicely, and it looks like it's going to have a bright future. :)

Boot process now shows a feedback. At first, red LED turns on for 1 sec, then green LED turns on for another sec, then red LED turns off and in another 2–3 seconds the tty shows up, and the system is interactive. That's with a small 7MiB gzip compressed kernel payload. I also tried a large 20MiB zstd compressed payload from eMMC, but that doesn't yet work as expected. I get load speeds of about 5MiB/s with that.

With lack of support for system suspend, fast boot times help make the laptop more useable.

My Levinboot setup

I use eMMC to load the payload, and SD card to load Levinboot itself. This way I can easily upgrade and test new Levinboot versions, and not worry about it breaking, because I just pop out a SD card, and update the bootloader from my PC, in case something fails.

To get Levinboot, I compile it from source code:

CROSS=aarch64-linux-gnu
export CC=$CROSS-gcc
export OBJCOPY=$CROSS-objcopy
export LD=$CROSS-gcc

git clone https://gitlab.com/DeltaGem/levinboot.git
cd levinboot
# currently contains fixes for reading GPT partition table
git checkout dev
./configure.py --payload-emmc --payload-zstd --payload-initcpio --with-tf-a-headers $ATF_DIR/include/export
ninja

# Copy levinboot-sd.img to my Pinebook Pro
scp levinboot-sd.img root@pbp:/boot/levinboot-emmc.img

Then I install it to SD card from Pinebook Pro itself, along with flashing payload to eMMC:

#!/bin/bash
set -e -x

ROOT_UUID=6795bcbb-fba8-4d22-81ef-dc628324b021

BOOTOPTS=(
  console=tty1
  video=eDP-1:1920x1080@60
  rd.luks.uuid=$ROOT_UUID
  rd.luks.name=$ROOT_UUID=root
  rd.luks.options=timeout=0,discard
  root=/dev/mapper/root
  rootfstype=f2fs
  rootflags=x-systemd.device-timeout=0
  rootwait
  rw
  mitigations=off
  quiet
  loglevel=2
)
BOOTOPTS="${BOOTOPTS[@]}"

cp -f board.dtb board-lv.dtb
fdtput -pt s board-lv.dtb /chosen bootargs "$BOOTOPTS"

truncate -s 0 payload-lv.img
zstd -zc bl31.elf >> payload-lv.img
zstd -zc board-lv.dtb >> payload-lv.img
zstd -zc Image >> payload-lv.img
zstd -zc initramfs.img >> payload-lv.img

dd if=payload-lv.img of=/dev/mmcblk2p1
dd if=levinboot-emmc.img of=/dev/mmcblk1 seek=64
#dd if=levinboot-sd.img of=/dev/mmcblk1 seek=64

sync

When booting from eMMC, Levinboot expects to find payload in a special GPT partition. I just use a single partition with GUID e5ab07a0-8e5e-46f6-9ce8-41a518929b7c.

This is all documented in detail in the Levinboot README file.

2020–09–02: Progress on the multi-boot image

Basic process of creating a multi-boot image is:

At this point each distro will be bootable, but may fail if it has some hardcoded expectations for the partition table structure, filesystem labels, etc. This will need to be fixed manually.

Getting the rootfs tarballs

Many distros just publish a block device image. Getting the rootfs files from the image requires mounting the filesystems contained in the image and backing up all files with bsdtar. Something like this can be used in most cases:

L=`losetup -P --show -f "$1"`
BOOT_PART=1
ROOT_PART=2

mkdir -p m
mount ${L}p${ROOT_PART} m
test -n "$BOOT_PART" && mount ${L}p${BOOT_PART} m/boot
bsdtar -cvf - -C m --numeric-owner . | zstd -z -4 - > rootfs.tar.zst
test -n "$BOOT_PART" && umount m/boot
umount m

losetup -d "$L"

I excluded the distros that don't publish rootfs tarballs or block device images, because they're too much hassle, at the moment. That is pmOS and nemo mobile.

Images/tarballs can be downloaded at:

Excluded:

Partitioning

p-boot requires MBR partitions, so we need to use that with an extended partitioning scheme, to be able to have more than 4 partitions. Creating the partition table is quite simple. We just need to decide how much space each distro will need. zstdcat rootfs.tar.zstd | wc -c can give a clue. There are some larger than 3GiB, but most fit within 2GiB, so let's create a table like this:

At least 32GiB SD card is needed.

Warning: The following script is broken, and leads to first two partitions overlapping. Fixes welcome. :)

sfdisk -W always /dev/mmcblk0 <<EOF
label: dos
label-id: 0x12345678
unit: sectors
sector-size: 512

4M,1020M,L,*
,4G,L
,4G,L
,21G,Ex
,3G,L
,3G,L
,3G,L
,3G,L
,3G,L
,3G,L
,,L
EOF

Filesystems

Now let's create filesystems. My kernel supports f2fs, so if I boot any of the above distros with my kernel, I can just use f2fs everywhere. Unfortunately, I don't know which of the above distros are able to boot from f2fs with their own kernel.

They all probably support ext4. F2FS is much better for SD cards, though. A conundrum :-)

After an investigation, I've decided on this layout (PARTUUIDs will be useful later on).

The table also has the current status of the distros and their assignment to the partitions.

/dev/mmcblk0p1: PARTUUID="12345678-01"  p-boot
/dev/mmcblk0p2: PARTUUID="12345678-02"  mobian (f2fs, 5.7 kernel, PP 1.0-1.2)
/dev/mmcblk0p3: PARTUUID="12345678-03"  KDE neon (ext4 only, 5.7 kernel, PP 1.1 only)
/dev/mmcblk0p5: PARTUUID="12345678-05"  arch (f2fs, my 5.9 kernel, , PP 1.0-1.2)
/dev/mmcblk0p6: PARTUUID="12345678-06"  sxmo (ext4 only, 5.7 kernel, PP 1.1, 1.2)
/dev/mmcblk0p7: PARTUUID="12345678-07"  lune (ext4 only, 5.5 kernel, PP 1.1 only)
/dev/mmcblk0p8: PARTUUID="12345678-08"  maemo (ext4 only, 5.7 kernel, PP 1.0-1.2)
/dev/mmcblk0p9: PARTUUID="12345678-09"  ut  (ext4 only, 5.6 kernel, PP 1.1 only)
/dev/mmcblk0p10: PARTUUID="12345678-0a" sailfish (ext4 only, 5.6 kernel, some custom DTB with partial support for up to PP 1.2)
/dev/mmcblk0p11: PARTUUID="12345678-0b" pureos (f2fs, 5.8 kernel, PP 1.0-1.2)

The status is not great, only maybe 3 distros really support PinePhone 1.2, and only one uses a kernel that's not EOLed upstream. Triste!

Anyway, this leads to:

for bd in /dev/mmcblk0p{3,6,7,8,9,10}
do
  mkfs.ext4 $bd
done

for bd in /dev/mmcblk0p{2,5,11}
do
  mkfs.f2fs $bd
done

After this we can just mount all the partitions one by one and extract the prepared rootfs contents there, with:

mount /dev/mmcblk0p# /mnt
bsdtar -xp --numeric-owner -C /mnt -f rootfs.tar.zst
umount /mnt

That's about it for today. I'll do first boot tests in the following days, and you can look forward to a bootloader setup guide. :)

2020–09–01: More p-boot cleanups and an example configuration

With some suggestions from Yoda, I added a few more sanity checks to the boot partition configurator, and more importantly, I prepared an example boot configuration directory with some ready-made p-boot themes, splashscreens and scripts.

This should make it much easier to get people started using p-boot, even if they will not read the (currenlty somewhat outdated) README file.

The sample config is in the example/ directory in p-boot git repo.

2020–08–31: Releasing p-boot GUI bootloader

Today I finally decided to release display support for p-boot. Along with it, p-boot's code also got quite a bit of cleanup. At this point main.c is fairly readable. There's also a new support for 3GiB variant of PinePhone, that TL Lim sent me some time ago.

I've copied my megatools.megous.com website template, and made it work better on mobile phones, to make a nice landing page for it, too: xnux.eu/p-boot

Since previous publicly released version, p-boot can now be configured to show the name of the phone on the boot screen, which is quite useful if you're like me and TL Lim keeps sending you PinePhones for development every other full moon. :)

I also use the name to describe HW mods I did to the particular phone.

There are some things I'd still like the p-boot to do. Since p-boot has direct access to PMIC, I thought of using it for following things:

And more:

In the other news, TODO

I tried to cleanup and organize my kernel upstreaming TODO list, and it's still too long. So instead I decided to give p-boot a little boost in exposure by preparing a multi-boot image that will feature a range of diverse distributions. With help of #pinephone channel users, I came up with this list:

Looks like making some of these share space and play nice with the others will be quite a challenge. If you're a distro maintainer, publishing a ready-made rootfs tarball is a great way to simplify your inclusion into a multi-boot image.

The image will be great for people new to PinePhone to try various distros with very little hassle. Just flash one image to SD card, and select the distro on boot.

I'll try documenting the image creation steps, but it will most probably be a one shot experiment for me. So if anyone will want to maintain this image further into the future, they'll be very wellcome.

I'm still undecided whether I'll force my kernel on all the above distros, or let them use their own. Probably the latter.

2020–08–31: Getting started

I already maintain xnux.eu which is a regular website that provides summarized information about my projects organized by device/topic. I decided to start this log to provide more of a in-progress information to people interested in my PinePhone related work. That is mostly kernel drivers/bootloader development, and performance optimizations.

Send any feedback you like to x@xnux.eu.