megi's PinePhone Development Log RSS

2021–10–18: Pinephone Pro – support merged into my kernel tree

I've integrated initial support for Pinephone Pro, that was previously done by Martijn Braam and Kamil Trzciński (ayufan) into my kernel tree, and spent yesterday looking at various issues that I've found with that kernel tree and fixing them.

I've fixed some of the most annoying issues, some of them with the help of carlos from #pinedev IRC chat:

After all these fixes I've made more suspend/resume tests, and it seems that it all works much better now. Pinephone Pro now idles at ~1.7 W, which is very close to what kernel can achieve by suspending all devices it has under control currently (s2idle). With suspend to RAM, it can get as low as 400 mW.

Next time it will be useful to figure out how to reliably power down and reset the phone, because that's not working great at the moment either.

It will be also interesting to figure out how to achieve 160 mW in suspend to RAM that the Android factory image seems to be capable of, from my previous measurements.

Pinephone Pro code is now integrated into my kernel in two branches: ppp-dt and ppp-drivers. I squashed all the original DT code and split the driver code from device tree changes and started building on top. I've excluded Type-C related changes, for now, because they are not documented well and I don't trust that they don't break non pinephone kernel builds.

I've also switched to 5 Ghz wifi (40MHz channel) and Pinephone Pro's seems to be able to achieve 15 MiB/s with that, which is a nice upgrade over original Pinephone.

More to come…

2021–10–15: Pinephone Pro – A Quick Review

I received Pinephone Pro (2021–05–11 batch) at the end of August. Here's a summary of my experiences with it so far.

Obligatory boot video :)

U-Boot is terribly slow in the default configuration, so I didn't even try it. But I ported Levinboot to PPP and here's Levinboot booting up the Arch Linux ARM from eMMC: (Levinboot is on SD card and the payload is on eMMC)

Whoa, that was fast! 6.15s to desktop. Exactly just like the Pinebook Pro boot times. ;) All that is thanks to CrystalGamma, and his work on Levinboot. :)

Here's a serial console view of Levinboot starting up on Pinephone Pro.

Power consumption

It's a portable mobile device, so I guess power consumption is what people will care about a lot. It has two aspects: how long it will run on battery, and how much it will heat up under load.

I have a nice setup for measuring the load at 1000 samples per second via this fake battery setup I made for original Pinephone:

Pinephone Pro came with Android factory image, that I used to get some initial numbers to compare against in the future. This image most likely doesn't limit the maximum CPU frequency, so the peak power consumption is quite high, due to that.

So this basically says that heavy use will drain the battery in about 1.5h, idling with screen on in aboutu 3–4h, and the phone will become unstable near the lower end of the capacity, because I don't think the single cell battery will sustain high enough voltage with load peaks of 10W or more near the lower end of the capacity. (that would be 3A+) Multi-cell battery in Pinebook Pro probably fares better at this, due to lower internal resistance in parallel battery cell configuration. CPU throttling based on remainging capacity will be needed to control this issue.

When suspended to RAM (standby) the phone will last for about 2 and half days with the Rockchip's TF-A blob. This is worse than Pinephone, which has power consumption in suspend around 60 mW (2.5× better).

Once I had Arch Linux working on the phone, I also measured the power consumption on mainline Linux + mainline TF-A.

I used the default backlight brightness of 2000 or so, wifi was turned on, and I varried CPU frequency limits (cpupower frequency-set --freq $FREQ) and loaded the CPU cores with openssl speed -multi $N.

Baseline idling with brightness set to kernel default and 1.42GHz limit was 3.1W.

N   FREQ
---------------------
6    408 MHz    3.5 W
6    600 MHz    3.8 W
6    816 MHz    4.2 W
6   1.01 GHz    5.5 W
6   1.20 GHz    5.5 W
6   1.42 GHz    6.9 W
2   1.42 GHz    5.1 W
1   1.42 GHz    4.1 W

N = 1 (loading one performance core to max)
N = 2 (loading two performance cores to max)
N = 6 (loading all cores)

Backlight brightness vs power:

100     2.8 W  (almost not visible)
2000    3.0 W
3000    3.4 W
3500    3.7 W
4000    4.2 W
4095    4.3 W  (max)

The scale is non-linear.

So at full brightness/full CPU load at 1.42GHz, the phone will consume about 8.3W. This test is without the GPU doing anything. GPU will probably add something, too.

Some board photos

Some things I noticed:

What works

I did not yet test the rest.

Kernel status

I'd like to add cameras to DT, but that will need a working 5.15 kernel (because it has support for multiple cameras). Linux 5.15 is currently not booting on Pinephon Pro. It hangs mid-boot possibly on some issue related to RCU.

Other than that, non-mainlined drivers for the PMIC battery and charger support are needed.

The rest of the support is very similar to Pinebook Pro, ie. non-working convergence in mainline Linux.

Closing words

Pinephone Pro is basically Pinebook Pro with more useful PMIC, modem, cameras, and a better supported wifi chip (cypress supported cyw43455 is more open than broadcom supported brcm43456 in PBP) in a much smaller form factor.

Performance is thus much better than the original Pinephone, but the power consumption is higher too. Baseline power consumption with normal backlight brightness and screen turned on is ~2.8–3W. Doing something in the browser or just moving the mouse around rises the power consumption to ~5W. This was tested in sway, with GPU acceleration turned on, and only one CPU core enabled.

That's not a huge problem for Pinebook Pro with its 36Wh battery (4–7h runtime on a notebook is nice) 2–3h runtime with screen on, on a phone, is a problem. It will be a nice mini-notebook with a pinephone keyboard, though. :)

2021–10–13: How to adapt multi-boot image to your needs

I've added a program to p-boot that allows you to extract contents of the boot partition to a directory, so that you can modify it and re-import back into the boot partition.

This allows you to change menu items in the boot image, customize graphics of the boot menu, update the kernel, change kernel boot parameters, add new distributions to the existing multi-distro image, etc.

It's as simple as:

p-boot-unconf new-dir /dev/mmcblk0p1

Now you can edit contents of new-dir in any way you want. If you don't want to change the kernel, you'll only want to modify the boot.conf file.

To save the changes back to the boot partition, you can run:

p-boot-conf new-dir /dev/mmcblk0p1

And that's it. :)

Multi-boot image has some nice builtin structure that you can use to your advantage. Each distribution is contained in its own subvolume in the main btrfs filesystem on the second partition.

For example, the subvolume strucuture may look like this if you mount the second partition of the multi-distro SD card on your PC:

manjaro-phosh
.pre-boot/manjaro-phosh
.pristine/manjaro-phosh
pmos-mplasma
.pre-boot/pmos-mplasma
.pristine/pmos-mplasma

Here manjaro-phosh is the subvolume that's actually booted, .pre-boot/manjaro-phosh is snapshot of the manjaro-phosh before it was booted for the first time and .pristine/manjaro-phosh is the snapshot of the original root filesystem of the distribution, before I applied any fixes to it, to make it not mess up the multi-distribution image with so called first boot scripts.

So if you want to restore one of the distributions to the original state, you can simply run:

btrfs subvolume delete manjaro-phosh
btrfs subvolume snapshot .pre-boot/manjaro-phosh manjaro-phosh

And that's it. Next time you boot manjaro-phosh, it will be in the original state. This is useful if some update breaks the distro, or whatever.

You can use this to your advantage, and make a snapshot before trying some new thing that can potentially make the distribution unusable.

btrfs subvolume snapshot manjaro-phosh manjaro-phosh-2021-10-13

Go wild and if anything breaks, you can recover with:

btrfs subvolume delete manjaro-phosh
btrfs subvolume snapshot manjaro-phosh-2021-10-13 manjaro-phosh

In fact you can create several snapshots of the same distribution, add boot menu for each snapshot (snapshot is itself a subvolume and snapshots in btrfs are writable) and just use both snapshots in parallel. :)

If you don't want to keep losing your data in /home or /root when you switch between OS snapshots, you can create subvolumes for those directories and mount those subvolumes via fstab. That way you can revert from failed OS update, but keep your data (or share data between distributions in the multi boot image in general).

If you want to save space you can delete distributions you don't like from the multi-distro image:

btrfs subvolume delete manjaro-phosh
btrfs subvolume delete .pre-boot/manjaro-phosh
btrfs subvolume delete .pristine/manjaro-phosh
p-boot-unconf boot-export /dev/mmcblk0p1
# remove manjaro phosh from boot-export/boot.conf
p-boot-conf boot-export /dev/mmcblk0p1

The similar way (just in revers) you can add any new distribution to the multi-boot image:

btrfs subvolume create new-distro

# get rootfs tarball for the distro from somewhere
bsdtar -xp --numeric-owner -C new-distro -f tarball.tar.gz

p-boot-unconf boot-export /dev/mmcblk0p1
# add new-distro to boot-export/boot.conf (you can get inspiration
# by looking at the existing boot entries on how to do it)
p-boot-conf boot-export /dev/mmcblk0p1

And that's pretty much it. You may even boot the distro's own kernel and initramfs, if it supports btrfs out of the box. Otherwise, you can stick with my kernel.

If you want to update my kernel, the latest version is always available at: https://xff.cz/kernels in the ppd.tar.gz package. You only need to copy the Image file over linux-0.img and board-1.1.dtb over dtb-0.img and board-1.2.dtb over dtb2-0.img in the boot-export directory.

Once you have the boot filesystem exported to some directory, you don't need to keep re-exporting it over and over again. You can skip the p-boot-unconf boot-export /dev/mmcblk0p1 from then on.

You can also have multiple boot options for the same distribution in the boot menu. I use this to try different kernels during development. If one fails, I just boot the older one that is known to work.

The same way you can experiment with different crust or TF-A versions, without ever having to remove the back cover of your pinephone again.

All this management is also doable from the booted pinephone itself.

P-boot is that good. :)

The necessary tools are pre-built in the p-boot repository.

(WARNING: All filenames in the above text are examples, you have to modify the commands to your own situation/needs!)

2021–09–04: Pinephone HDMI hot-plug-detection HW bug

I've figured out what is the cause of an issue that some Pinephone owners experience, when using the convergence dock. It's a HW bug in the pinephone's design of level shifting circuitry for the hot plug detect (HPD) signal between HDMI bridge and A64 SoC.

Relevant circuit is here: https://megous.com/dl/tmp/452023169b0cce41.png

If you have this issue, and you're using my kernel v5.13 or later, which includes HPD change logging patches, you will:

NOTE: If you're using Mobian, the above may not apply to you, because Mobian cherry-picks patches from my tree, and may not include the HPD debugging one. If you want to help me debug kernel or hardware issues, debug with a distro that uses my kernel tree more directly.

This may also explain why for some people HDMI works only at certain battery charge level, because the issue depends on precise voltages and part tolerances in the particular phone, and these may shift a bit depending on the current voltage of the battery.

It's fixable in software. Quick workaround may be to ignore the HHPD pin's state, and try forcing the HDMI-A-1 connector to be enabled. You may also need to dump EDID data from your monitor and force load them on Pinephone. How to do it is described below.

It may be enough to make it work (I haven't tried the workaround myself), for people who experience this issue.

Proper software workaround will be a bit more involved, requiring to perform hot plug detection in software by querying the HDMI bridge chip's status over I2C bus, and passing it to DRM driver via a software path.

How to test with a quick workaround

  1. Dump your monitor's EDID binary data on some other Linux computer using cat /sys/class/drm/card-[connector_name]/edid > edid.bin
  2. Copy edid.bin to your pinephone and pass it to the kernel using cat edid.bin > /sys/kernel/debug/dri/[card_id]/HDMI-A-1/edid_override
  3. Force enable enable HDMI output by echo on > /sys/kernel/debug/dri/[card_id]/HDMI-A-1/force

Now you should be able to get past the dreaded DP state 0x03.

You need to figure out correct values for [connector_name] and [card_id] yourself by browsing the parent directories /sys/kernel/debug/dri/ and /sys/class/drm/card, because these can differ netween systems.

2021–08–10: Wrapping up Pinephone keyboard firmware development

The keyboard production seems to start soon.

I'm ending the firmware development for now. The final code is here:

https://xff.cz/git/pinephone-keyboard/

Samuel Holland helped with the final testing, and this code will be flashed in factory. You can read about the design of the code here:

https://xff.cz/git/pinephone-keyboard/tree/README

and in other readme files in that repository.

What I've achieved since June

Initially I received the vendor's flashing tool, vendor's proprietary firmware code for the keyboard and some schematics for the keyboard.

And this was just the keyboard MCU part. The second part of the problem is the charging chip.

All in all this project required a lot of bootstrapping, reverse engineering, and custom tooling support. That's why you'll find many tools in my code repository, each can still be used for one of the purposes I used them for originally.

I don't measure time spent on my hobbies, but this easily took ~100h of time over the last ~2.5 months and I quite like the result and all the new stuff that I've learned. That's also why I haven't done much on Pinephone kernel last few months. ;)

Lessons learned

I didn't know much about using USB from Linux and writing USB code for microcontrollers. Nor did I use USB in any of my hobby HW projects in the past, and that changed after this project. I thought USB is complicated. I found that using USB as a dumb transport is quite comparable to using a serial port. It needs more code, but USB solves issues that you have to solve by hand in some upper protocol layer you layer on top of serial port communication, so the complexity is comparable.

On Linux side you can access USB devices with about 4–5 ioctl calls, and in most microcontrollers that I have access to, you can write basic USB device code in about 200 lines (no need for large USB stacks supplied by the vendor when you're just using USB as a dumb transport instead of trying to implement some of the standard USB device classes).

In fact I wrote 2 USB device implementations during the development of the keyboard. One for the controller inside the keyboard, and one for FX2 board I used to simplify testing. And it was not that hard. FX2 is especially easy and well documented, with hardware helping the firmware writer quite a bit at each step.

It was a fun ride, mostly, except for some stressful frustrations towards the end. I don't mind frustration with trying to fix a broken HW, reverse engineer stuff, etc. There was a lot of that in this project after all. Interrupt driven MCU code is especially tedious to write, since you have to re-read the code several times, searching for potential cirical sections, etc. Compiler does not help at all here.

What I do mind is being put under time pressure on a project where I'm just volunteering my free time. That's very easy way to get bitter feelings, which I'd like to avoid.

I've damaged the last prototype trying to figure out some way to connect I2C of the keyboard MCU to the charger I2C so that phone can monitor the keyboard battery charge and control the charger.

At the moment I have no way to do anything with the keyboard, until I fix my prototype to revert the various HW modifications I attempted, or get the final keyboard, so the feeling of accomplishment is a bit bittersweet. Oh well. :)

Some random photos from my efforts

First effort to give access to charger I2C to the phone (using TXS0108 level converter):

Second attempt to give access to charger I2C to the phone (via direct connection of I2C A MCU interface to the charger chip). A plan:

Execution:

And giving up after weird voltages on I2C pins and no more time to try things: :(

Final untested suggested changes for having I2C access to the charger proxied by the keyboard, that will be present in the final keyboard design, to have a chance to figure out the charger I2C communication later on (as a firmware update):

My new favorite USB dev board (Cypress FX2), that I used to test I2C interface of the keyboard:

FX2 controlling the keyboard over I2C from the PC:

2021–07–26: The latest Pinephone keyboard prototype

The last prototype arrived today, so I uploaded my firmware to it and tested it. It looks good! There are no shorted or interrupted columns and rows this time, the two PCBs were merged into one, and all keys are easy to press and react well. Even spacebar now reacts to key press along all it's surface.

(„stuck“ keys is just some issue with debugging USB interface code I'm using, some reports don't get through)

There's one thing that I didn't realize before though…

Phantom keys

My original enthusiasm about being able to register arbitrary key combinations has to be tempered a bit by reality. :)

The general issue with matrix keyboards can be described like this:

If you press 3 keys in this specific pattern on the matrix (electrically), the fourth corner of the box pattern will also appear as shorted, because it's specific column and row wires are shorted electrically via the other three keys. I think this is called a phantom key, and is a general problem of keyboard matrices.

Here is the illustration of the issue on the real keyboard:

It's worked around by placing typical modifier keys on separate columns and rows:

The rest of the keys are placed on the matrix in such a way, that pressing Ctrl+Alt+Shift and one other regular key will not create any phantom key effect. And when it does, the phantom key is on the unused combination of the row and column, so it can be ignored.

This makes it so, that you can use modifier keys freely, without ever noticing this fundamental issue. In fact, I haven't noticed this on normal PC keyboards in the last 20 years. But they have the same issue!

For example I can't press C+V+G on my desktop keyboard without the keyboard ignoring the last pressed key, whichever one it is. I never noticed, and I don't think I'll ever notice that again. :) Three letter combos are not common, except for government agencies.

And this is actually the workaround for the issue, too. The keyboard's software checks after each new pressed key, if the key would produce a phantom key effect, and if it would, it ignores both the actually pressed key and its phantom counterpart.

Of course this gets even more complicated if you press more than 3 keys, depending on which combo it is.

Summary

The above issue just has to be handled in the driver/userspace software. There's not much else that can be done.

I've checked the actual configuration of the pinephone keyboard matrix, and it is designed in such a way that you can use any combination of Ctrl+Alt+Shift + one other key (that is not Fn) safely.

Fn key is on the same row as Alt key, so you can't combine those two. You can combine Ctrl+Shift+Fn arbitrarily though.

You can also use any combination of any two keys on the keyboard.

Enter and Backspace keys are on their own column, so one of those can also be combined with any other safe key combination as described above.

So that's all rather good, in the end! :)

Some more photos

Schematics:

This is how to connect USB to flash the basic firmware over USB. Some rather easy soldering:

(Final version will allow flashing over I2C.)

Next steps

2021–06–28: Pinephone keyboard's final summary

I finished most of my testing of the pinephone keyboard, and also completed the firmware, and this post should serve as a summary of my observations.

There are some open questions around the charger, that need further investigation.

Next steps is completing the TODO list, which contains only minor things at this point.

Suggested HW changes

After quite some testing, this is a final set of suggestions I have:

2021–06–20: Pinephone keyboard's firmware I2C interface

I've implemented a USB stack for the pinephone keyboard's firmware, to be able to perform printf debugging and tracing of what the MCU does. It's now possible to see MCU's operation using a simple USB debugging tool that shows the output from debugging code on the MCU in real-time.

I've used the new debugger it to get to understand precise behavior of I2C slave peripheral. How interrupts are fired, and what's the contents of various status registers during various I2C transfers from the pinephone SoC over POGO pins.

This understanding can be used to write a more proper I2C control interface for the keyboard, that will be modelled as is typical with various other I2C devices a set of I2C „registers“ that can be read from and written to. The proposed specification for the register set is here. It will be possible to configure the keyboard's scanning behavior in various ways over the I2C, and perform self-tests for QA purposes.

I've also finalized the USB flashing tool. It now has a very nice interface and is quite useable and better documented.

Samuel shared his keyboard kernel driver recently. So you'll have a choice between kernel and userspace drivers for the keyboard. Both have their strengths. Userspace one will be easier to customize and play with, and the kernel one will be more available and better integrated into kernel's suspend/resume functionality, I assume.

User firmware

From factory the keyboard will come with the stock keyboard firmware. This firmware will occupy code memory from addresses 0x2000 to 0x3fff. The stock firmware will also have a I2C flashing interface, whose proposed workings are described here.

The superpower users will be able to write their own firmware, and flash it to the keyboard over I2C directly from the phone, without having to disassemble the keyboard and solder on a USB cable. ;) This firmware will not overwrite the stock firmware, but it will be placed at addresses 0x4000 to 0x7fff, after the stock firmware. If the user firmware is flashed, it will run instead of a stock firmware.

There will be a short time window where the stock firmware will wait for a I2C transactions prior to defering execution to user firmware after MCU reset. This window can be used to prevent the jump to user firmware, and stay in the stock one. This will be the mechanism for recovery in case the user flashes incorrect or borken user firmware.

This mechanism can be used to:

Power user based hardware or firmware modding the keyboard is quite a likely possibility when everyhting is developed in the open using FOSS licensing, with focus on making this relaitvely easy.

I'll propose exposing the free GPIOs as solder pads for the final revision of the keyboard's PCB.

There's just enough free space in the keyboard's body for a little drilling and eg. adding proximity sensor (to detect whether the lid is closed or not), to add notification LEDs visible when the KB is closed, etc.

You can also make much more space inside the keyboard by removing the battery, and replacing it with your own hardware. You may also expose GPIOs externally by adding a new connector to the keyboard body, and have some simple pluggable peripherals.

Next steps

Next, it's necessary to implement the proposed I2C interface. Most complicated part of which is the flashing interface, which will require further reverse engineering of the original USB bootloader, because code ROM flashing registers are not documented.

Once that is done, it will be necessary to again look at making the MCU operation as low power as possible, by using the power down and green modes of the MCU.

Charger interface

Charger's I2C interface will still need quite a bit of thought. There's an issue that the charger chip multiplexes I2C interface with the LED notification pins, and detects what kind of interface is used by checking the voltage on the I2C pins whenever it wakes up, which is not predictable.

I2C interface is shared with the keyboard's MCU, so if charger wakes up when the MCU is communicating with the SoC, it may see low voltage on I2C pins, and misdetect the interface as LED one, and start driving it as such and break communication between MCU and SoC.

Samuel also identified another way the charger can break I2C interface, and that's when the phone is off, and thus it does pull I2C SDA/SCL lines low via 10 kOhm pull-ups to DCDC1 on the SoC side, the voltage on I2C lines will be just ~1.5V, which is not enough to make the charger think I2C interface is used, and it again switches to LED driving mode.

We need to figure out some way to prevent this from happening.

The last proposal is to add mosfets between charger I2C pins and the POGO I2C pins, with gate connected to INT pin. When the INT pin is low (because it's driven low by MCU, or because DCDC1 is off, because the phone is off), these mosfets would disconnect the charger I2C pins from the POGO ones, and this would make the charger always detect the I2C interface mode correctly. It would also be possible for the SoC to disconnect the charger from I2C interface at any time by driving the INT pin low during I2C communication with the MCU.

Lastly when the phone is off, this would save some power by cutting off the path between „grounded“ DCDC1 and pull-ups to VREG (always on 3.1V linear regulator inside the keyboard charger chip).

2021–06–16: Pinephone keyboard input daemon released

For those hungry for more information about the pinephone keyboard's software, I've written up some notes:

https://xff.cz/git/pinephone-keyboard/tree/HACKING

and updated a TODO list:

https://xff.cz/git/pinephone-keyboard/tree/TODO

I've also cleaned up the firmware and released a basic input daemon, that can be used to operate the keyboard. It already supports all key combinations on the default keyboard map.

https://xff.cz/git/pinephone-keyboard/tree/inputd/factory-keymap.txt

https://xff.cz/git/pinephone-keyboard/plain/inputd/factory-keymap.jpg

With this, it's possible to actually start using the keyboard prototype day to day.

My HW keyboard is broken, and is not very usable to test multi-key combinations. Next revision will hopefully be better.

2021–06–14: Hints on debugging HDMI output on Pinephone

Some people have issues using external monitor with pinephone.

There are some ways to debug it. The most reliable one is to check the state of DRI in debugfs.

If you expect output on the monitor and there's none, you can check contents of file /sys/kernel/debug/dri/0/state or /sys/kernel/debug/dri/1/state to see what it says about the current state of the display pipeline as configured by your compositor. With only internal display turned on it will contain:

...

crtc[45]: crtc-0
        enable=1
        active=1
        self_refresh_active=0
        planes_changed=1
        mode_changed=0
        active_changed=0
        connectors_changed=0
        color_mgmt_changed=0
        plane_mask=2
        connector_mask=1
        encoder_mask=1
        mode: "720x1440": 60 72000 720 750 778 808 1440 1458 1468 1485 0x48 0xa
crtc[54]: crtc-1
        enable=0
        active=0
        self_refresh_active=0
        planes_changed=0
        mode_changed=0
        active_changed=0
        connectors_changed=0
        color_mgmt_changed=0
        plane_mask=0
        connector_mask=0
        encoder_mask=0
        mode: "": 0 0 0 0 0 0 0 0 0 0 0x0 0x0
connector[56]: DSI-1
        crtc=crtc-0
        self_refresh_aware=0
connector[58]: HDMI-A-1
        crtc=(null)
        self_refresh_aware=0

If you have in your dmesg:

...
DP state changed to 0x03
...

as the last DP state changed message, you should be able to just see your monitor's modes and enable the HDMI output via xrandr if you're using Xorg, or equivalent tool if you use something else.

It's the role of the compositor to enable the output to HDMI, so if you see connector[58]: HDMI-A-1 crtc=(null) and not connector[58]: HDMI-A-1 crtc=crtc-1 and some reasonable mode configured on crtc-1 it means that your compositor simply did not enable the HDMI output, so your monitor stays off.

If this happens on Xorg, xrandr --auto is the quickest way to at least get some output.

You need to consult documentation for your distribution's compositor to see how you can configure display output.

If /sys/kernel/debug/dri/*/state configuration looks correct, but your monitor is still not on, it's either a kernel or perhaps a HW issue.

If you sometime get to DP state changed to 0x03 after plugging in the dock, and sometimes not, you should check if you see some received SRC_CAP messages or received SVID or DP_ALT_ENTER messages in your dmesg after the last cable plugin event. If not, it means communication with the dock over the CC pins doesn't work or didn't happen for whatever reason, and the phone was not thus able to configure DP-Alt mode in the dock.

It may be due to some connector issue. Pinephone Type-C connector is a bit too much recessed into the body of the phone, and some Type-C cables don't plug in completely.

2021–06–14: First pinephone keyboard typings with my FOSS firmware

https://youtu.be/Kx6B_OL4OJ4

:)

2021–06–14: Pinephone keyboard firmware

So I managed to write and test my USB flashing tool for Pinephone keyboard.

I've also managed to write FOSS firmware for the keyboard, that can provide 12×6 bitmap of all currently pressed keys. So any combination you'll manage to press will be properly detected, without limit.

Currently this interface is consumed using a userspace daemon that waits for H->L transition on pin PL12 (POGO interrupt pin) and reads out the current bitmap of all pressed keys.

In the future this daemon can provide a userspace implementation of Linux input device. This is a way to easily support any kind of keymap without having to touch the firmware. All the mapping could be done in userspace and can be easily modifiable by users, without any re-flashing of the firmware. Userspace input devices behave just like the ones provided by the kernel, and work everywhere incl. Linux console.

I've also investigated the charging controller inside the keyboard, and it can be controlled over I2C from the phone itself. I've demonstrated reading of battery voltage/current and status of the button on the side of the keyboard. This button is connected to the charging controller.

Here is a repository with some of the code: https://xff.cz/git/pinephone-keyboard/

And here you can see the code in action: https://youtu.be/hj8DIqD74IM

Hardware-wise, the prototype I have has some issues. Some keys don't react well to being pressed (all the keys on the top row), and enter also sucks. Control and C keys are merged (pressing one, presses the other one too), and Z key is stuck (+ one other row/column combination is shorted). Shorted rows/columns on my prototype are R4/C2 and R5/C4. It may be better if enter key had two membrane contacts, because it doesn't press/react well due to its size.

All in all, things look rather bright on the software/firmware front. :) I'll probably continue hacking on the keyboard's software next weekend.

2021–06–11: PinePhone keyboard – HW testing

Pinephone keyboard arrived a workday earlier than advertised by DHL, so I started looking at it today.

Looking inside you see two boards. Top one is for the battery charger, the bottom one is for the keyboard controller:

This prototype has I2C interface from the charger controller wired ad-hoc to the I2C interface on the POGO pins. This will be useful to test how and if it really works and if the charging chip is controllable. :) I haven't checked whether the components on the other side of the board are not populated, to make the I2C interface available, but hopefully they are.

This prototype also has POGO pins and USB interface nicely exposed as solderable pads on the USB controller board:

You'll need to be careful when soldering the USB cable to the keyboard controller board, because battery wires are directly under the through hole pads you'll need to solder the USB cable to. If you do it with the board mounted as is, you'll most likely burn the insulation on the battery wires.

Soldering is necessary if you'll want to flash your own keyboard firmware.

To avoid burning the insulation, I've first unscrewed the board and padded it with a paper tape.

I've used jumper wires instead of soldering the USB cable directly, because I plan to add a USB flashing connector to the chasis, once everything is verified to work, and no further HW mods are necessary.

All that remains is to solder the jumper wires to the USB cable temporarily…

… and plugging in the keyboard to my PC to verify it works:

input: HID 04f3:1812 as /devices/pci0000:00/0000:00:01.2/0000:02:00.0/usb1/1-2/1-2.1/1-2.1:1.0/0003:04F3:1812.0038/input/input58
hid-generic 0003:04F3:1812.0038: input,hidraw5: USB HID v1.11 Keyboard [HID 04f3:1812] on usb-0000:02:00.0-2.1/input0
input: HID 04f3:1812 Mouse as /devices/pci0000:00/0000:00:01.2/0000:02:00.0/usb1/1-2/1-2.1/1-2.1:1.1/0003:04F3:1812.0039/input/input59
input: HID 04f3:1812 System Control as /devices/pci0000:00/0000:00:01.2/0000:02:00.0/usb1/1-2/1-2.1/1-2.1:1.1/0003:04F3:1812.0039/input/input60
input: HID 04f3:1812 Consumer Control as /devices/pci0000:00/0000:00:01.2/0000:02:00.0/usb1/1-2/1-2.1/1-2.1:1.1/0003:04F3:1812.0039/input/input61
input: HID 04f3:1812 Wireless Radio Control as /devices/pci0000:00/0000:00:01.2/0000:02:00.0/usb1/1-2/1-2.1/1-2.1:1.1/0003:04F3:1812.0039/input/input62
hid-generic 0003:04F3:1812.0039: input,hiddev99,hidraw6: USB HID v1.11 Mouse [HID 04f3:1812] on usb-0000:02:00.0-2.1/input1

And it does. :) This is USB interface exposed by the main firmware. It's probably possible to press keys and control the PC using the keyboard over USB HID, as is.

Entering the flashing mode requires issuing some command over one of these HID interfaces, that will cause jump to the bootloader, which will expose altogether different USB device interface meant for flashing.

Next step is writing and testing a FOSS flashing tool. It should also be possible to test the charger I2C interface. These two tasks are independent.

I'll start with the flashing tool.

2021–05–30: PinePhone keyboard – more observations

After further eyeballing of the firmware in Ghidra, with the help of a linker symbol map, I've reverse engineered the USB flashing protocol completely.

I'll not publish complete details at this time, but here's a general idea of how it works:

All the details of the flashing protocol are known to me now, and I have a flashing tool in the works. It should be possible to finish most of it even prior to getting my hands on any hardware, but I guess there's no hurry at this point with all the details known. :)

Out of all this research I personally plan to do these HW modifications on the keyboard to help make it more useable/robust:

Overall, the situation around flashing arbitrary firmware to the keyboard controller reliably seems quite reasonable, without too many gotchas for the end-users. With a known good keyboard firmware users should be able to recover from flashing failures without the need to open the keyboard or a huge risk of bricking the keyboard.

The only thing that will be troublesome to end-users is the possible non-confirmance of the charging circuit to USB specification, as mentioned in the previous post. It will be probably challenging to insert the keyboard into the USB ports on the computer without the port shutting down due to overcurrent.

I'll be also looking at the actual keyboard firmware (main app). Most of the existing code deals with exposing the USB HID interface, which is not used by Pinephone. Actual keyboard interface used by Pinephone is HID over I2C. Code for that is much simpler. There's a lot of dead code in the existing firmware provided by the vendor. All that's needed is just reading out currently pressed keys from the key matrix, and providing updates over I2C on changes. That should be a few hundred lines of code tops, not the current ~6000 lines. USB interface is only used for initiating the switch to flashing mode of the bootlader from the main app, but we can instead use a fixed key combination to do that (Pine key+F+M, for example) and drop all that other dead weight from the firmware. This will make firmware also much smaller and faster to flash (currently it's 19 KiB, or 16 KiB when compiled with optimizations, but it can be ~2 KiB).

If you'd like to support the effort to make FOSS firmware flashing tool and customizable firmware for the Pinephone keyboard, donations are welcome.

2021–05–26: PinePhone keyboard

I did some analysis of Pinephone keyboard prototype's schematics, components and firmware. Here are some of my observations.

Parts

First let's look at major parts of the keyboard, and what features they have.

Charging controller – IP5209

The chip is designed for use in power banks with optional LED flashlight.

It takes input from a 5V power supply or the battery and provides 5V output using a boost step-up DC-DC converter.

It has some other minor features, like LED indication, and timed key input (long/short press of the key do different things).

It also has rather detailed control of its functionality over I2C bus:

Charger/output control:

GPIO:

ADC:

This is quite a lot of useful functionality for Pinephone keyboard's use case!

Resources:

USB microcontroller – EM85F684A

This is rather generic and simple microcontroller with USB bus support. There is nothing really remarkable about it.

It uses 8051 instruction set, and has 32768 bytes of flash memory for code with 10,000 write/erase cycles. 2048 bytes of XRAM and 256 bytes of RAM.

Pin change wakeup is available on ports 5, 6, 9. There's a I2C slave peripheral, that is connected via POGO pins to Pinephone.

It has low power mode, that allows disabling CPU clock. Remaining features are not useful for the keyboard use case.

Documentation for the basic features is available on the Elan microelectronics website.

Resources:

Pinephone keyboard schematic

Keyboard schematic is rather simple:

Resources:

Proprietary firmware written in C was sent to some developers, incl. me, and I was able to compile it using Keil C. Firmware is half C code, half „bootloader“ distributed in binary object form. So it's possible and rather easy to modify the firmware. The issue will be with flashing it.

At this time, only sure way to program this microcontroller is to have a HW programmer from Elan microelectronics, install a windows driver and software, and use some windows GUI tool to flash the firmware over pads exposed on the PCB inside the keyboard.

There's also a possibility of flashing the firmware over USB, but the protocol is not documented, and there's no non-proprietary program available to do it.

This flashing method currently depends on proprietary bootloader, and cooperation of the existing keyboard firmware with entering the flashing mode. So if a broken application code is flashed, it will not be possible to re-flash it again using this method.

At my request, TL Lim sent me a Windows flashing tool for the keyboard's firmware. I've started reverse engineering it, and at this time I have the basic protocol reverse engineered, and I should be able to communicate with the keyboard using /dev/hidraw# device. I don't have the keyboard prototype HW, so I can't do much more at this time.

Next step is to get a capture of USB communication of the original flashing tool for Windows using usbpcap and cross-check the reverse engineering data with the actual communication and then write a Linux port of the flashing tool. This will allow me to create a better firmware updating mechanism for the keyboard controller, and experiment with key scanning routines. Looks like there should be no limit to how many keys can be pressed at the same time. :)

It would be nice if people with the keyboard prototype did the capture, otherwise the development will stall until I receive the prototype, too.

Pinephone charging/battery circuit schematic

Charging circuit can not be controled over I2C bus, because I2C bus from Pinephone is not connected to it. This is rather limiting.

Resources:

Suggested modifications

  1. Connect IP5209 to the i2c bus, instead of using 4 LEDs for charge indication.
  2. Revise the charger schematic according to the suggested reference circuit from the manufacturer of the chip. Add the current sensing resistor.
  3. (keyboard v2?) Add support for a lid switch.
  4. (myabe with optional mod/soldering bridge) Allow to share interrupt lines with keyboard controller and charge controller (not share by default)

Open questions

2021–05–22: PinePhone kernel news

There are a bunch of new notable things in my kernel since the last update 2 months ago. I've added some patches from others:

And I also continue to tune up and improve my and other drivers:

I've also done more power draw measurements a few weeks ago (that's where the before/after pictures linked above are from), and pinephone's power consumption in suspend is now around 60mW with modem off. Modem usually adds another 10–15mW. So with 11Wh battery, that means around 6 day standby, if that's all you do with the phone. :)

Overall 5.12 and 5.13 in my tree should be working quite well at this time! Samuel's time travel fix („clocksource/arm_arch_timer: Improve Allwinner A64 timer workaround“) seem to have solved the last stability issues I had with some of the pinephones I have access to. Things look pretty bright!

There are still some HDMI issues some people keep reporting, that manifest themselves by anx7688 firmware not getting past „DP status = 0×03“ (it should go from 0 all the way to 6 normally). When this happens it's caused by HDMI controller inside the SoC using incorrect clock frequency, and anx7688 fails to acquire the HDMI signal from the SoC. It should be easy to fix, but I have trouble reproducing it with my kernel and userspace, so I can't identify the root of the problem, yet.

I've also worked on musb DMA support, and I got quite far (my current work is here https://megous.com/git/linux/log/?h=musb-sunxi-dma-wip). It seems that DMA controller works (I can see DMA completion interrupts getting triggered), but integration of DMA driver with mainline musb driver is a bit complicated, and the only documentation is the existing BSP code. Musb seemingly has nice abstraction for supporting various DMA drivers, until you look closer and realize that musb core has many code paths that change how things are done based on what DMA driver is used. So in the end the interface between musb core and DMA drivers is quite messy. :(

2021–03–17: PinePhone eMMC measurements

I wrote a simple io_uring based benchmarking tool for block devices.

It's very simple. It creates a fixed size queue of random reads in the io_uring and then just keeps the queue full.

I wanted to see what my devices can do. It maxes out at 640k random read 4KiB IOPS on my 500GB Samsung 980 PRO nvme SSD, with no special Linux block subsystem tuning. Which is about in the middle of the supposed 500k and 800k values in the datasheet. It's hard to interpret various small print notes, so it's not clear what to expect with reads scattered across the whole drive. I may try tuning the kernel configuration to see if I can get higher, but this may already be close to the HW limit. So the tool can produce quite a bit of IO activity. :)

I used this tool on various uSD cards and eMMCs I have in my Pinephones.

Here are my results:

Pinephone 1 (early dev sample)

eMMC: (date=07/2019 manfid=0×0000da oemid=0×0101)

uSD: (32GiB Sandisk Ultra A1 08/2019)

Pinephone 2 (UBports endition)

eMMC: (date=07/2019 manfid=0×0000e1 oemid=0×0116)

uSD: (32GiB Sandisk Ultra A1 08/2019)

Pinephone 3 (Fixed USB-C UBports endition)

eMMC: (date=07/2019 manfid=0×0000e1 oemid=0×0116)

uSD: (32GiB Sandisk Ultra A1 09/2018)

Pinephone 4 (3GiB UBports endition)

eMMC: (date=12/2019 manfid=0×000045 oemid=0×0100)

uSD: (32GiB Sandisk Ultra A1 10/2019)

The tool, in case you want to check out the code.

Queue depth doesn't seem to matter much with eMMC/uSD cards in PinePhones, while it does with high performance SSDs on my desktop.

2021–03–15: Some PinePhone updates

There were a bunch of smallish news/fixes in my kernel since my last post:

Kernel log during boot is now a bit less noisy with non-important errors. This is mostly due to use of dev_err_probe() function that hides the temporary probe errors in case probe of some driver fails due to missing dependencies. If you want to see drivers that failed to probe due to a missing dependency, those are listed in /sys/kernel/debug/devices_deferred.

I've fixed the ANX7688 flashing timeout, so that flashing the firmware to ANX7688 works again on my 5.10 and 5.11 kernels.

There was some interest in mainlining the device tree bindings for ANX7688 this month, so I've added my observations to the discussion. Hopefully this will lead to getting ANX7688 driver mainline, eventually. Though the discussion has stalled.

There's an interesting new feature in my Linux kernel. Samuel proposed a nvmem based reboot driver. So I've took the idea, and updated the p-boot implementation so that it works with a modified variant of the proposal. It's now possible to reboot into a different boot option, or into eMMC, or into a p-boot boot menu, or even into FEL just by calling reboot(LINUX_REBOOT_MAGIC1, LINUX_REBOOT_MAGIC2, LINUX_REBOOT_CMD_RESTART2, "something"), where something can be one of emmc-egon, fel, menu, sd1, sd2, emmc1, emmc2, … and more. This will make the kernel inform p-boot to perform boot in a particular mode after the reboot, regardless of the default boot menu selection. Neat, eh? You can pass this argument to systemctl like this: systemctl reboot --reboot-argument sd3, and the phone will reboot to p-boot's 3rd boot menu option for SD card.

I've enabled USB_QUIRK_RESET for the modem's USB device in my kernel tree. This is supposed to help with modem USB interface resets during resume from sleep. more information With this it is not necessary to change power/persist: 0 and avoid_reset_quirk: 1, because that's done by the kernel automatically when the modem's USB device is probed.

With the help from #linux-sunxi/Peetz0r, I've enabled remote wakeup via WiFi packets in the 8723cs driver.

You can now configure the wifi driver to receive and react to some packets and wake the phone up if it's sleeping.

On PinePhone:

iw dev # and write down mac address, eg. 02:ba:7c:9c:cc:78
echo clean > /proc/net/rtl8723cs/wlan0/wow_pattern_info
iw phy0 wowlan enable any

# ... suspend to ram ...

On other device:

wol -i pinephone_ip_address 02:ba:7c:9c:cc:78

# ... pinephone wakes up over wifi ... :)

And the last change of note is a new patch that enables PMIC to wake the SoC from sleep when the battery is running low (<10% capacity), so that the OS may shut down gracefully, instead of just running out of power while asleep. This needs some userspace support, which is probably already implemented by some desktop environments. In case it isn't, the phone will just wake up from sleep and die sooner. I don't think the PMIC will keep sending interrupts, so the OS will need to be careful to use this opportunity to power off. It may still be more reliable to just perform periodic RTC based wakeups for various book keeping tasks, incl. battery monitoring, instead of relying on this one shot low battery interrupt from PMIC.

You can follow these kinds of changes and news in my kernel by looking at the latest orange-pi-$VER-$DATE tags in my kernel tree.

If you click at the tag name, you'll see the NEWS log. It's just not as pretty as this blog. :)

The latest tag also gets a build here: https://xff.cz/kernels/ for all the devices that I maintain my kernel for.

I didn't do much new work on my previously announced modem userspace project, because while I was away on holidays biktorgj seems to have poured a lot of effort into the same general idea, so there's no real need.

I have a bunch of things I'd like to try to tackle next. One of those is imprvoing the performance of USB peripheral mode. Currently when connecting PinePhone to a PC directly via USB cable, the performance of the USB is limited to speeds around 10 MiB/s. This is due to missing DMA implementation in the driver, so CPU has to push every byte individually into the FIFO, which is not very optimal. I have a work in progress implementation of the DMA support that I'd like to finish. Hopefully this will improve the speeds to be closer to the theoretical limit of ~40MiB/s.

2020–12–19: Away from keyboard for the rest of the month

I'll not be able to respond to e-mails until January. No worries, everything is fine. :)

2020–12–15: Battery thermistor measurements

A Mobian wiki user nivea made some interesting measurements of the thermistor present in the Pinephone battery https://wiki.mobian-project.org/doku.php?id=hardware:battery and the results show that the current safety values configured in PMIC by the kernel are roughly correct. Great!

2020–12–05: Alternate EG25G userspace project

I've been reverse engineering EG25G userspace on and off since mid July 2020.

I really dislike the Quectel's code. They took relatively clean (if a bit overengineered) Qualcomm code and made a total mess, that somehow barely holds together.

They manage to do things like sprintf(buf, "echo %s > /dev/kmsg", barely_checked_user_input); system(buf); in a superuser process, or system("rm /something") instead of unlink("/something"), etc. None of their code inspires any confidence in the modem's software as a whole.

Biktor has a nice bootloader/kernel cleanup project, so I decided to start a fully FOSS userspace rewrite project to complement it.

How it will look/work

Alternate FOSS userspace code will be very simple and will not depend on anything in the original root filesystem of the modem's firmware.

You'll be able to install the FOSS userspace along with the existing rootfs content and choose which one to boot to via my modem power manager:

# to power up the modem and boot the original userspace code
echo 1 /sys/class/modem-power/modem-power/device/powered

or:

# to power up the modem and boot the FOSS userspace code
echo N /sys/class/modem-power/modem-power/device/powered

Modem power manager will configure DTR and #W_ENABLE gpios to one of the 4 possible configurations, and then it will power up the modem. These two gpios can be relatively easily read from the modem's userspace during boot, and can be (a)bused to make boot decisions very early on after the kernel starts the /sbin/init process.

The existing firmware will need just a single simple and fairly safe modification. The rest of the changes will be just addition of new files to the root filesystem to a /foss/ directory.

In order to make boot decisions, the FOSS userspace will include a small /sbin/init binary that will be executed by the kernel instead of the original one, and will check the status of the above mentioned gpios and then execute either /foss/init.2, /foss/init.3, /foss/init.4, or the original firmware's /sbin/init.sysvinit. One of these will become the PID 1 process.

This will be the foundation for easy experimentation with the FOSS userspace for the modem's OS running on it's ARM CPU.

Anyone will be able to modify and test the FOSS userspace code quite fearlessly. Recovery will be as simple as booting to the original userspace, or keeping one of the /foss/init.* binaries in an always known-good state, and only experimenting with the others.

Plan / progress

This should hopefully be enough to be able to perform voice calls without having to use any part of the original Quectel's code that is present on the root filesystem of their firmware.

I expect the FOSS code to shrink the firmware quite a bit. Most of the heavy lifting is done by the Qualcomm/Quectel Linux kernel and their modem's DSP code. Userspace code just has to perform some setup tasks by modifying a few files in sysfs and keeping track of a few special needs the modem's DSP code may have, like configuring the codec during the call, and some other minor stuff.

This can probaly be easily handled all by a single 200kB static binary instead of a huge 80MB busybox based system with 15 interacting daemons, hundreds of shell scripts and hard to capture behavior.

And there will be 80MiB of newly released space for some actually interesting additional code to be run on the modem, if you choose to delete the original proprieatry code. :)

2020–12–02: Fixing fallout from the new WiFi driver

While testing caching and latency of call indication messages, I've noticed that the updated WiFi driver broke suspend/resume cycle, and also that it increased the resume times significantly.

Basically the driver has a compile option to either do the whole of WiFi resume synchronously during resume, or to punt it to the separate thread and don't block the resume process. The latter option only works with Android wakelock API. Though the only thing that the driver needs to be able to do this is to block further suspend to RAM until the asynchronous resume job finishes. That was fairly easy to achieve with mainline API, so now WiFi driver will do its 850ms of resume tasks without blocking the resume process as a whole, which makes userspace responsive just 400ms after resume is initiated, instead of after 1.2s.

Also WoWLAN support code path made the driver return -ENOSYS (-38) from the suspend callback, which aborted the suspend to RAM. I disabled WoWLAN in this patch https://megous.com/git/linux/commit/?h=wifi-5.10&id=6959135b030f7b3bb4db4c42824ab1743436889a I don't think anyone uses WoWLAN support, so that seems like an acceptable solution for now.

One thing that I noticed about the WiFi on Pinephone with the updated driver and newly enabled power management is that it can take some time to ping through to the phone from my PC, after there was no network activity for some long period of time. This is the same behavior that the broadcom wifi on Pinebook Pro exhibits, and it's probably just some property of improved power management.

The fixes are in this branch: https://megous.com/git/linux/log/?h=wifi-5.10

2020–11–30: Improved ANX7688 driver and increased I2C speed

I've looked again at the mud stained hair ball that is ANX7688 firmware and extended my driver to better work with the USB-C docks when the docks are powered. The driver now works better also with other USB-PD enabled devices.

The changes mostly relate to making sure Pinephone will not overload a PD charger or a dock when powered from said dock. USB-C, USB-PD, and BC 1.2 specs specify various methods for determining the the current that can be used by the device, and it's quite complicated to put it all together reliably.

The approach the driver takes now is to wait for 3s after the cable is plugged in, and if no power source announces itself via a PD message, it will fallback to determining the maximum input current from the resistors connected to the CC pins. If the power source announces itself via PD message, the driver will wait 0.5s after the ANX7688 FW forwards the announcement and ramps up the input current limit to the one hopefully negotiated with the power source by the ANX7688 firmware.

After quite some testing, this approach seems to work for most of the USB-C devices and their combinations that I have at home, except for:

Most of the new patches are available here https://megous.com/git/linux/log/?h=anx-5.10

I've also bumped the I2C bus speed to 400kHz, which should make communication with I2C devices faster. This may be noticeable when using the touchscreen (~4ms shorter input latency).

One last thing I've changed was configuring PMIC via p-boot to limit power consumption, so that VBUS voltage never drops below 4.5V. This does away with USB-PD hard resets I've seen when using my USB-PD 5V/3A charger with Pinephone. USB-PD controllers probably do some VBUS voltage monitoring (ANX7688 does for sure), and try to recover when VBUS drops bellow certain threshold.

The patch is in p-boot for now: https://megous.com/git/p-boot/commit/?id=a80cac6bdea170ddeaee9475c976ba76558839ed

With this I could charge my Pinephone from USB-C PD charger without a single hard reset. Previously there were many.

2020–11–25: New multi-distro and p-boot released, WiFi power savings

I've put together a new version of the multi-distro image. More details and what's new can be found on https://xnux.eu/p-boot-demo/

p-boot also got a new release with some sought after UI improvements. Menu is more easily navigable thanks to wraparound and volume keys auto-repeat. :)

WiFi power savings on PinePhone

I've found a very recent rtl8723cs driver in some rockchip kernel tree on github. The driver is just slightly more than a month old, and I was able to enable power saving in it. This saves 350mW when the WiFi is idle, without any negative side effect that were present with the old driver.

The new driver is integrated into my 5.10 kernel.

2020–10–31: New p-boot released, new pre-built/updated kernels

New p-boot is released. It has the 60 FPS fix as mentioned previously.

It also has some usability improvements (it shows battery status in the footer and information about where the p-boot was booted from and from where it loaded the boot filesystem as these can mix and match quite a bit). It will also report if the battery is being charged or not.

When the phone is off and you insert the charger, this will normally lead to powering up the phone. It's not avoidable. p-boot will detect this situation and force the display of the menu followed by a regular 10s poweroff timeout, regardless of any default boot option setting. This makes it easy to connect a turned off phone to a charger without booting the OS.

If you run my multi-boot SD card image, you can update it with the new p-boot to get a reboot to eMMC feature, so that you don't have to remove the SD card to boot to your normal eMMC OS, that came with the phone.

New kernel releases

I've pulled in Martijn's front camera auto-focus support into my kernel branches – both 5.10 and 5.9. I've also added the 60 FPS fixes to both, too.

Release notes/NEWS are part of the new tags I'll be making from now on in my git repos each time there's a new kernel release/update from me:

Updating p-boot

You can find some notes on how to update multi-distro's p-boot on pine64 forum:

https://forum.pine64.org/showthread.php?tid=11347&page=8

2020–10–29: New p-boot features – reboot to FEL/eMMC, 60FPS fix

The most common request I get for p-boot is to make it possible to skip the boot from SD card and boot from eMMC.

I decided to implement this along with an option to reboot to FEL, and it works now. :)

I'll make a new p-boot release soon.

I've also finally found the fix for the PinePhone LCD panel issue where the panel was running at wrong refresh rate of 36.6Hz. With the fix, it run at expected 60Hz.

With panel running at 60Hz watching 30 or 60 FPS video should be possible without annoying stuttering, that was there previously.

I've made the fix for this available both in p-boot and in my kernel.

I've also implemented HW accelerated scaling in p-boot, that allows to do flashy animations, and use of smaller than full-res textures.

With all these new features available, it's probably time to look at making a new release of my multi-boot image. It will be interesting to see how smooth things are at 60 FPS. :)

2020–10–12: Backlight changes when swithcing between USB-C power modes

I noticed that backlight brightness changes significantly when connecting a VBUS powered device to Pinephone 1.2, like USB-C dock or USB-C hub.

I tracked down the issue to a VBUS_CTRL signal that ANX7688 provides to control VBUS power direction. This signal is connected to AXP803 to tell it to not try to charge from VBUS, when the VBUS is generated from the battery. (Perpetuum mobile is still not a thing, even in 2020)

Unfortunately this signal is also needlessly connected to PL9 pin of the SoC. VBUS_CTRL is 3.3V signal, and PL port is 1.8V port, so this is a rehash of WiFi chip IO port voltage issue, that was fixed some time ago, that was also causing backlight instability, previously.

In this case, there's no SW fix for this issue. But I've informed TL Lim, and was told that HW team will still manage to fix the issue for the Manjaro edition. The fix is to remove a 0 Ohm resistor that connects PL9 to VBUS_CTRL.

Compared to WiFi IO voltage overdriving the PL port, this issue is less annoying, because VBUS_CTRL is stable. So you'll have lower brightness when Pinephone sources power to USB-C device and higher brightness when it sinks power, but the brightness will not vary. I guess some power will also be saved, because overdriving the PL9 probably causes some small amount of power to be lost needlessly.

If you like DIY fixing your Pinephone, you can fix existing 1.2 devices by removing R1318 resistor. This should be the easiest of the HW fixes so far. ;-) Pinephones 1.0 and 1.1 don't have this issue.

2020–10–05: Reboot to FEL, skipping SD card boot, external display fixes

I've noticed that A64 BROM (Boot ROM) contains a special code path for booting via a specially formatted image that can be stored in SRAM, if some registers in CPUCFG range contain a magic value and an address to the image.

I prepared the image that would toggle Pinephone LEDs and hang, and a loader program that would load the image to SRAM A1, setup the magic registers, and reset the SoC.

Well it didn't work. :)

What probably happens is that the magic registers are reset when SoC is reset.

I did all this with hope that this could be used to reboot to FEL from Linux.

Alternative options to achieve reboot to FEL from Linux require cooperation from the bootloader. For example Linux can set some value to one of the RTC data registers, and bootloader will detect it early on and jump back to BROM FEL (allowing to use USB to boot the kernel/initramfs).

One other fun thing that can be tried would be to jump into the middle of the BROM code, right after the SD card boot code, in effect skipping the SD card boot and making BROM boot from eMMC. Let's say p-boot would boot into GUI menu, and if user would choose to boot from eMMC, p-boot would write a magic value to RTC data register and reboot. p-boot would get loaded again by BROM from SD card, and it would check the flag in RTC data registers, and based on it it would jump to eMMC boot code in BROM, forcing BROM to try booting from eMMC.

This would be a fairly cheap implementation of skipping boot from SD card, while having SD card inserted.

External display fixes

With help from ollieparanoid, I noticed that whether external display works in my kernel or not depends on whether the user uses p-boot display mode or not (p-boot without display support, or without splash screen, or u-boot).

I fixed this issue. It related to what parent the MIPI-DSI clock had on boot.

There's still one issue where external display only works if the phone is booted with dock connected but not when the dock is first connected after boot. That one still remains to be investigated.

The symptoms are that TCON1 clock is not enabled when the dock is connected only after boot, for some reason.

After this is fixed, the use of external display should become much more reliable.

Hopefully, I'll manage to figure out the fix prior to 5.9 release next week.

2020–09–29: New codec driver in my 5.9 kernel

Samuel updated his codec patches a few days ago, noting in the dev chat:

I have a branch ready for anyone wanting to test the new audio patches: https://github.com/smaeul/linux/commits/pine64-5.9 three notes:

  1. Set PulseAudio to use a 48 kHz sample rate and an 8 kHz alternate rate. Both must be multiples of 8 kHz or modem/BT audio won't work.
  2. You must use a QDAI config with the codec as master (modem as slave) and an 8 kHz sample rate. Use AT+QDAI=1,1,0,1,0,0,1,1 unless you have a very good reason not to.
  3. You must use a BT config with the codec as master (BT as slave) and 2 PCM slots. Use len=2 offset=00f6 { 85 10 }.

This does include jack detection and headset mic button detection. I don't have any three-button headsets, so I can't test the volume up/down functionality. it has the „80% solution“ to call audio quality – it will sound fine unless something tries to play/record audio from/to the CPU at a sample rate >8 kHz during a call

To clarify note 1 above: you can use both 44.1 kHz streams (music) and 8 kHz streams (modem), but not at the same time. so unless you have some way to kill all 44.1 kHz streams before starting a call (can UCM do that?), the sample rates have to be compatible

And some time later:

I pushed some more changes, including a fix for the high-pitched whine from the speaker it was a harmonic from the audio PLL, and I believe it's leaking from VCC-PLL → AVCC, since the noise is the same on the headphones and speaker.

So I made two changes:

  1. triple the frequency of PLL_AUDIO, which needed to be done anyway for using the sample rate converter. now the harmonic at 44100 (previously the whine) is inaudible, and the harmonic at 48000 is lower frequency, so less annoying
  2. decreased the PLL bias current, which drastically decreases the volume of both the harmonics and the white noise if you want to try out a different series of frequencies (1×, 4×, 6×), adjust the last two hunks in https://github.com/smaeul/linux/commit/7b51d5b09136

So I spent some time pulling the new codec patches to my kernel and rebasing my patches for X-Powers AC100 codec support on top of them, so that I can merge them into my kernel.

This went fine, and the results are as described by Samuel. Modem audio is nice and smooth, as long as you don't use alsa PCM interfaces during a call with any other sample rate than 8kHz. And the new jack detection works too.

One issue that people may hit with the new codec support is that some ALSA controls were renamed. I made the list of changes from the code diffs:

#!/bin/sh

sed -i 's/AIF1 AD0 Mixer AIF1 DA0 Capture Switch/AIF1 Slot 0 Digital ADC Capture Switch/' "$1"
sed -i 's/AIF1 AD0 Mixer AIF2 DAC Capture Switch/AIF2 Digital ADC Capture Switch/' "$1"
sed -i 's/AIF1 AD0 Mixer ADC Capture Switch/AIF1 Data Digital ADC Capture Switch/' "$1"
sed -i 's/AIF1 AD0 Mixer AIF2 DAC Rev Capture Switch/AIF2 Inv Digital ADC Capture Switch/' "$1"
sed -i 's/DAC Mixer AIF1 DA0 Playback Switch/AIF1 Slot 0 Digital DAC Playback Switch/' "$1"
sed -i 's/DAC Mixer AIF1 DA1 Playback Switch/AIF1 Slot 1 Digital DAC Playback Switch/' "$1"
sed -i 's/DAC Mixer AIF2 DAC Playback Switch/AIF2 Digital DAC Playback Switch/' "$1"
sed -i 's/DAC Mixer ADC Playback Switch/ADC Digital DAC Playback Switch/' "$1"
sed -i 's/AIF3 DAC Playback Route/AIF2 DAC Source Playback Route/' "$1"
sed -i 's/AIF3 ADC Capture Route/AIF3 ADC Source Capture Route/' "$1"

exit

# and then manually:

AIF2 DAC Source Playback Route:
        "None", "AIF2 Left", "AIF2 Right" 
        "AIF2", "AIF3+2", "AIF2+3"        

AIF3 ADC Output Mux:
        "None", "AIF2 Left", "AIF2 Right" 
        "None", "AIF2 ADCL", "AIF2 ADCR"  

Remove "AIF3 Loopback Switch"
       "AIF1 Loopback Switch"
       "AIF2 Loopback Switch"


##########################
# Direct mapping from old to new names (semantics are the same):
##########################

- "AIF1 AD0 Mixer AIF1 DA0 Capture Switch"
+ "AIF1 Slot 0 Digital ADC Capture Switch"

- "AIF1 AD0 Mixer AIF2 DAC Capture Switch"
+ "AIF2 Digital ADC Capture Switch"

- "AIF1 AD0 Mixer ADC Capture Switch"
+ "AIF1 Data Digital ADC Capture Switch"

- "AIF1 AD0 Mixer AIF2 DAC Rev Capture Switch"
+ "AIF2 Inv Digital ADC Capture Switch"

- "DAC Mixer AIF1 DA0 Playback Switch"
+ "AIF1 Slot 0 Digital DAC Playback Switch"

- "DAC Mixer AIF1 DA1 Playback Switch",
+ "AIF1 Slot 1 Digital DAC Playback Switch",

- "DAC Mixer AIF2 DAC Playback Switch"
+ "AIF2 Digital DAC Playback Switch"

- "DAC Mixer ADC Playback Switch"
+ "ADC Digital DAC Playback Switch"

- "AIF3 DAC Playback Route"            "None", "AIF2 Left", "AIF2 Right"
+ "AIF2 DAC Source Playback Route"     "AIF2", "AIF3+2", "AIF2+3"

- "AIF3 ADC Capture Route"             "None", "AIF2 Left", "AIF2 Right"
+ "AIF3 ADC Source Capture Route"      "None", "AIF2 ADCL", "AIF2 ADCR"

- "AIF3 Loopback Switch"
- "AIF1 Loopback Switch"
- "AIF2 Loopback Switch"

You can run the above script to patch up your UCM profile or whatever. I've updated my call audio setup script, for the new controls too.

If you like to test new audio patches you can grab my kernel at usual places: https://xff.cz/kernels or at https://megous.com/git/linux/

Modem reset

It turns out that Quectel managed to break modem reset via AT+CFUN=1,1, while adding support for fast poweroff. This manifests itself by modem powering off instead of rebooting when fast poweroff is enabled.

I had to patch my modem driver to disable fast poweroff during a reset after QDAI update, so that QDAI updates work smoothly with the new QDAI necessary for the above mentioned codec changes.

2020–09–20: Downsizing the multi-boot image

If the multi-boot image doesn't fit your 8GiB uSD card, because it's a tad too big, you can downsize it a bit by using this script:

#!/bin/bash

set -e -x

mkdir -p m
L=`losetup -P --show -f multi.img`
mount -o compress-force=zstd ${L}p2 m
btrfs filesystem resize 7000M m
echo ",7000M" | sfdisk -N 2 ${L}
umount m
losetup -d "$L"

truncate -s $((128+7000))M multi.img

If something fails in the middle, you may need to recover by calling umount and losetup -d yourself. losetup -l can tell you if the image is still exported as a loop device, and which one.

The image will have the size of 7128 MiB after resize and this should fit more „8“ giga something uSD cards, as there are obviously some other giga units than gigabyte and gigibyte used by some manufacturers.

2020–09–20: Some ways to improve Pinephone safety

This is a follow up on some issues from the previous article. On surface, solutions to some of the previously presented issues can seem simple. Toggle a few registers in PMIC, and we're mostly done. Trouble is that safety mechanisms are barely ever triggered, by definition. Safety events occur rarely. That means that the mechanisms are not regularly tested, and it is not known that they work.

Also it's not clear which code's responsibility fixing the issues should be. Bootloader, or kernel, or userspace? Finally, there are a bunch of devils in the details, that complicate the upstreamability of any solution. And having fixes upstream is necessary to make sure they reach the most users.

Necessary minimum

Nevertheless, at least enabling some pre-existing PMIC functionality blindly is better than nothing, so that's precisely what I decided to do in p-boot. It's the easiest place to start resolving these issues for me personally, and for other p-boot users.

I fixed two issues:

I didn't measure the 3kOhm NTC used in Pinephone battery and third party batteries I bought. I just used a table from some random 3kOhm NTC spec on The Internet, that seemed like it could match. Hopefully it's close enough.

Trouble with the second fix is that it's a hard power cut-off, so data loss may occur when PMIC overheats. There are three fixed temperature levels in AXP803. On level 1 the charging is limited, on level 2 the interrupt is sent to SoC, on level 3 the PMIC shuts down if configured to do so (by default it keeps running, and this is what my p-boot fix changes). Ideally, the crossing of level 2 would be handled by Linux to make it safely shut down the system, and level 3 forced power cut-off would never happen. Arguably, if charging was source of the heating, crossing level 1 will lead to resolving the issue, so the next level will not be reached.

Suggested fixes elsewhere

These fixes will reach a very limited audience. It would be nice to have these fixes in U-Boot too, but that's not possible at the moment, because U-Boot doesn't have access to PMIC.

Other places to put the fix is to ATF or Linux kernel. That can reach more people faster, but there would have to be some generic mechanism to make the fix upstreamable, otherwise it will not reach people using the mainline Linux kernel or mainline ATF.

There are some ways to approach this:

First, the most generic solution would be to have a description of the battery in DT describing the Pinephone. Sadly, the current bindings don't include battery temperature limits.

Also converting from temperature to NTC resistance (which is necessary to determine the code word from ADC for the limits used by the temperature monitor logic in PMIC) is not straightforward. It is usually defined in NTC datasheet as a table. Do I have NTC datasheet? No. I bought the batteries online from some mobile phone service shop.

There are also equations that approximate the temperature – resistance relationship for the NTC, which could be used instead of a fixed table, if one knows the relevant coefficients. These can be calculated after measuring the NTC's resistance at a few temperature points when we lack the datasheet.

So generic solution may look like this:

Kernel also has support for NTC devices, so maybe NTC can be described outside of the battery node (even though it's part of the battery).

This may all fail to be upstreamed on one thing: the battery is user swappable, so it's arguably not part of the Pinephone, and describing it inside the pinephone DT will not be appropriate.

I don't have any plans implementing any of the above, atm. Maybe with the exception of adding a 4th approach to the fix to my Linux kernel (the easiest one ;)). I'd like to work on my multi-boot image. So these are mostly pointers for somebody else who'd like to tackle this.

Other issues

Fast charging is not necesary in many situations, so having it as a default is not great. User should be able to decide if he wants to trade off slower, safer charging and battery longetivity over speed. This tradeoff can be realized in many ways.

All this is already controllable from userspace via sysfs. Ideally there would be some charging monitoring daemon that would take into account users's wishes and select proper strategy for charging, based on preference for battery longetivity or speed.

There are several trade offs the deamon would be able to handle:

All this is decision making that doesn't belong to the kernel.

Similar daemon could monitor power usage of the phone and try to limit it to safer levels, or warn the user if that's not possible.

2020–09–18: Let's talk about safety of Pinephone

My gf read me some articles about exploding phones today. :) I think there needs to be some serious conversation about Pinephone safety. Safety needs to become an important concern now, when more and more people are getting their Pinephones every month. It's just a matter of time before the first major safety incident hits this community, and it may be more than just a hacked store. It's just a numbers game.

Pinephone is an interesting device in one way. You can run whatever software you like on it (and you do!), and this software comes almost universally with zero guarantees. Read the license to any of the program you run on your Pinephone and it will almost certainly tell you:

THIS SOFTWARE IS PROVIDED ``AS IS'' AND WITHOUT ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.

or

THIS SOFTWARE IS PROVIDED BY <COPYRIGHT HOLDER> AS IS AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL <COPYRIGHT HOLDER> BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES

or

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

etc.

In case of Pinephone you have to take these warnings very seriously, because this software is not provided by the manufacturer (Pine64), and as far as I know, there's no software related safety testing going on at all.

Some skeletons, hiding at the lower levels…

I'll give you a few reasons why things may not be so rosy, when it comes to safety.

There's no unchangeable well tested guardian angel management engine that safely manages battery, power supplies, thermal behavior, that is provided by the manufacturer, and that is independent of the operating system.

Pinephone's SoC is quite bare when it comes to software/firmware (that's why FOSS enthusiasts like it, no blobs, you know!). This has a dark side, too. All the safety critical parts are written (or rather were not written, yet) by some random people on The Internet.

You can already choose among more than 10 Linux distributions to run on your Pinephone. How do you know any one of these is safe to run on your Pinephone?

How well safety critical parts of Pinephone function (or if at all), depends on how well Linux distribution vendors understand the platform, and how well they test its safety features. Pinephone safety depends on the software they do put together, afterall.

Is device safety even a thing distribution vendors test for or plan for at the moment? I don't know, but I doubt it.

For many months pretty much all distributions used misconfigured „official“ kernel (it's a meaningless moniker, btw, pine64 doesn't do software), that didn't regulate CPU temperature at all. It could go up as high as the thermals of the surrounding environment allowed. There were indications something is wrong. People reporting their phones felt too hot, displays showing burn artifacts, temperature indicator in sysfs returning error. Regardless, it persisted for many months, until I discovered the root cause while helping one user with thermal issues on her Pinephone.

Maybe I'm a bit biased, because I was one of the people working on the sunxi thermal driver over the years, and thus understand this element of the SoC quite well, but I was surprised nobody figured out such a serious issue, or realized what it meant. So let's go through some other issues that I'm aware of, of which other people may not be:

1. Pinephone battery uses a 3 kOhm NTC to monitor the temperature. Power management chip in Pinephone expects 10 kOhm variant by default. So early on, when the times were adventurous, someone decided to patch the kernel to disable battery thermal monitoring completely. Quick and dirty fix for Pinephone not charging due to false under-temperature alarm.

Now guess what… up to now, all distributions run with battery temperature sensing and regulation disabled. If you're unlucky and use a dud battery that will heat up more easilly during fast charging, you can burn down your house.

There will be nothing to stop it going past 50, 60, 70, 600°C. Phone will happilly provide current to the battery until it explodes and burns. Unlikely scenario for any single user, but this safety mechanism that's present on regular customer phones is missing here, just because nobody cared to configure it, yet.

2. And you know what, I also suspect that all distros fast charge all the way through the constant-current phase of charging, by default. Other than contributing to making the above mentioned house burning scenario more likely to happen, this also contributes to overheating issues on Pinephone.

3. Another thing. PMIC has an emergency thermal shutdown feature, for a situation when the chip itself overheats. It's disabled by default. It's also not well documented. ¯\_(ツ)_/¯

4. And, another one! Battery is rated for some sustained continuous discharge current (0.5C, 1500mA, ~6W). I guess it overheats if it is dicharged at a significanlty faster rate for prolonged periods of time, and potentially becomes a safety hazard again. Maybe again only if it's a suboptimal piece that passed QA.

Now, Pine64 sells convergence edition Pinephone meant for use with a dock and a monitor. When you connect the phone to the dock with monitor connected over HDMI, power consumption jumps by 4W. That alone is close to this specified limit of the battery. Baseline power consumption is 2.8W (with phone display on). Now if you actually use the phone the CPU alone can add further 2.5W during full load (likely, when trying to use the pinephone as a desktop machine). That is >9W total. That is well over the specified safe limit. Of course, the rest of the phone heating up a lot from this huge power draw don't help the at all, either.

So it's quite possible for the user to load the battery well out of spec, with just normal use. This needs to be managed somehow.

And those are just things that I'm aware of.

So what…

I'm trying to be a bit inflamatory here, to start the conversation. Nevertheless, the above issues are real. It's really just a numbers game. When the distros will not take safety seriously, by planing for it, testing for it, and verifying the mechanisms they are supposed to ensure are in place for the safety of their users, the odds somehting will happen will stay needlessly high.

Also, don't be a person whose house burns down for FOSS. Ask your favorite distribution's authors what they're doing to make their OS safe. Pine64 itself can only go so far to ensure safety of Pinephone, software you put on your Pinephone matters a lot, too!

2020–09–17: Video acceleration experiments with PinePhone

Recently, gstreamer gained support for utilizing v4l2-requests API, for h264 acceleration, so I compiled it for Arch Linux, patched the kernel with the help of ndufresne from #cedrus to fix kernel panics, figured out how to make gstreamer play to kms, and made a bunch of experiments with these results:

https://www.youtube.com/watch?v=dHOgVmxH_dA

https://www.youtube.com/watch?v=6P76lQX70iI

https://www.youtube.com/watch?v=yyRm8kccyG4

External display video playback, even to a fairly large display works very nicely. Quite a feat for an entry-level phone. :)

CPU utilization during playback is miniscule. Comparing power consumption, between idle and video playback, the video decoding takes just additional 0.2W of power, compared to CPU based decoding which could take up to 2.4W.

There's an issue with internal display being locked to 36 FPS, which makes playback of 30FPS video a bit choppy.

I also hacked my kernel to workaround gstreamer kmssink bugs, to test playback to an external 1440p@60fps monitor. 1440p@30 video decoding works fine. 1440p@60 video is too much, and drops frames. 1080p@60 video decodes and scales to 1440p fine, too.

Another issue is a heavy power consumption (4W) from the dock when HDMI output is enabled. Also dock's display output doesn't seem to work when powering the dock from the USB-C power supply.

Overall, it's nice. :)

2020–09–16: PineBook Pro and Levinboot again

Crystalgamma released Levinboot 0.7.2 yesterday so I decided to update, and try to make my Pinbook Pro boot times even faster. With his help I found that I accidentally disabled optimizations in my build and that was the reason for 4s decompression times for my kernel/initrd payload, and some stack overflows in levinboot's decompression code.

Unoptimized code turns out to be really slow. What a surprise.

I switched to lz4, enabled -O3, and now levinboot decompression and load times changed from 4s to 600ms. Significantly better! :)

With the latest levinboot my Pinebook Pro Arch Linux setup lets me to enter LUKS password in 4s since I start pressing the power button:

Perfect!

2020–09–14: Putting 13 PinePhone distributions on a 8GiB uSD card

This is a summary of how I was able to make a 5 GiB multi-boot 13-distro image, that easily fits on a 8GiB SD card, and can be used to test-drive most of the OSes that are available for Pinephone today.

Starting with a 32GiB image and an old-school approach

At first I did the obvious, and started with traditional partitioning. I created 10 partitions on a 32GiB SD card, and was barely able to fit 9 Linux distributions into that image. Each paritition needed its own filesystem and some free space to allow for proper operation of the OS. It was also hard to mange. Each time I needed to change anything, I had to figure out which partition held which OS, mount it, fix things and unmount again.

It was not good.

Swithching filesystems and sharing free space

The breakthroug came with a realization that I can mount subvolume in btrfs as if it was a filesystem in its own right. This allowed me to have just a single partition with a btrfs filesystem on the SD card, and have each distribution contained in its own subvolume. This way, all distributions could share from the same pool of free space for the data. There was no longer any need to plan partition sizes for each distro. This was the first major win.

Subvolumes are also much easier to manage than partitions. Need a new subvolume? Just create one and copy some files to it. Don't need the subvolume anymore? Just delete it.

Subvolumes appear as normal directories in the filesystem when the root subvolume is mounted, so one mount operation is all that's needed to have access to files of all included Linux distributions.

I also used another feature of subvolumes to debug startup issues in some distributions. Snapshotting. It's possible to create a snapshot of a current state of any of the distributions indivudally, and restore it in the future. Ubuntu Touch gave me a headscratch, when it didn't boot during the first boot, but it did on the second one. So I made a snapshot of the initial state of the filesystem, and kept comming back to it and tweaking it until I found the reason. (One of the boot scripts checked for the presence of a file named userdata/.writable_image and if it was missing it tried to reboot the system and failed, crashing the lightdm process.)

In fact, the latest multi-boot image keeps the initial state of all included distributions in a snapshot, so it's possible to selectively restore state of any of the distributions to the original.

Compression

Looking at btrfs, I found that it supports transparent compression for the stored files. Enabling zstd compression and decompressing rootfs tarballs of 9 distributions resulted in a filesystem that used 5.8 GiB of space with a compression ratio of 50%. This was exciting, because it looked like having a 8 GiB SD card image will be possible.

Using compression is also good for antoher reason. SD card access in Pinephone is limited to 24MiB/s max. At this speed, 4 core Cortex-A53 CPU can easily keep up, and this makes the loading of data from the SD card faster.

Some interesting features of COW filesystems

So far the optimization steps were fairly trivial. Just enabling features of existing filesystem and using them in a smart way.

Getting from 5.8GiB image with 9 Linux distributions to a 5 GiB image with 13 Linux distributions was harder. It was likely that those distributions have a lot of files in common. I made a simple test, and found that there's a space for further savings in the range of 12–13% if I could make the filesystem share data for duplicate files among the distributions.

COW filesystems, like btrfs, can sometimes allow to share file data among multiple files without resorting to hardlinking. In fact, there's now a generic VFS-level Linux API to make a filesystem share file data between files if it supports this feature.

Side note: Hardlinking would not be a great solution for reducing file data duplication in the image anyway, because it would have meant that if user changed a file in Ubuntu Touch, the change might be visible in other distros that may share that file.

The next hurdle was figuring out how to effectively use this API in my specific scenario. There are tools that allow scanning the existing filesystem, search for duplicities and use this API to remove them. Then it would be possible to scrub the filesystem, discard empty space, and compress the resulting block device image for easy re-distribution. I checked a bunch of those tools, and they seemed exceedingly complicated, or buggy when used on btrfs subvolumes.

Instead I decided to write a extraction tool using libarchive that takes multiple tarballs on the input (one per Linux distribution) and decompresses them while keeping track of content of already extracted files and using the above mentioned FICLONE API to share data whenver it finds file that was already extracted before. It does deduplication on the fly, so there's no need to apply any further cleanup steps to the filesystem afterwards. This approach to deduplication is also very fast and efficient, because there's no extra IO necessary.

At this point I tried to add 4 more Linux distributions to the image. I did this just to stress test the new extraction tool. In particular, I added several variants of postmarket OS, which were bound to have a lot of the files in common.

The resulting image had the same size as the previously made 9 distribution variant.

Final space savings

The final opportunity for size optimization was a cheap one. Remove files that are not used/needed. I didn't want to tweak the distributions too much, so that they are as close to their official state as possible. I had to make one change, though. I was not able to use distributions' own official kernels, because neither had a driver for btrfs built in. I also was not very fond of supporting outdated EOLed kernels many of the distributions use, or re-building them. So I used my own kernel and made all the distributions share it.

Due to this I was able to remove the modules, firmware, and kernels that distributions package themselves. With 13 distributions this led to another 0.8GiB of saved space.

The result is a 5 GiB 13-distro image, that easily fits on a 8GiB SD card, and can be used to test-drive most of the OSes that are available for Pinephone today.

The image also serves as a demo of the GUI bootloader I wrote for Pinephone.

2020–09–11: Adding postmarket OS to multi-distro image

So I finally tried the pmbootstrap tool to create a bunch of rootfs tarballs for several postmarket OS UI variants: GNOME, Phosh, Plasma Mobile, fbkeyboard.

I've added them all to my multi-distro image without any increase in size since the previous public build of the image. So now it's a 13 distro multi-boot image with pretty much every major and a bunch of minor PinePhone supporting distributions, and it still fits on a 8GiB SD card!

2020–09–11: Ways to help improve Pinephone kernel

There are a bunch of open questions about Pinephone's HW. When answered, these could help improve the kernel behavior. Here's one of them.

Minimal display brightness is too bright

This gets asked quite a bit, but just maybe 2 people helped do anything about it, so here's a more detailed guide on how to help.

Sadly fixing this issue requires gathering a lot more samples than 2, regardless of whether your Pinephone has the problem or doesn't. Trouble is that if only people who have this issue do the following test, the results will be useless. Even people who's backlight seems to work correctly need to take part. So it's a hard problem to solve. :)

If enough people report at what brightness level the backlight turns on on their Pinephone, we could improve the default minimum brighness level, so that most users can reduce their Pinephone screen brightness to lower levels.

There's an open question on how to set the backlight brightness values on post 1.0 revision phones, since lower PWM duty cycles lead to backlight being basically off. It would be nice if more people can test the various backlight levels on 1.1 and 1.2 revision with this change in dts in the backlight node:

/backlight {
        brightness-levels = <0 1000>;
        num-interpolated-steps = <1000>;
};

The above change can be made by fdtput, which is part of dtc package:

fdtput -t u your.dtb /backlight brightness-levels 0 1000
fdtput -t u your.dtb /backlight num-interpolated-steps 1000
fdtput -t u your.dtb /backlight default-brightness-level 500

You don't need to recompile the kernel or anything. Just identify what dtb file is your OS using (it will be in /boot directory) backup it up and apply the above commands to the original (just replace yout.dtb with the name of your dtb file).

This change will make it so that PWM backlight driver sets up 1000 linear backlight steps, so you can enter values 1–1000 to /sys/class/backlight/backlight/brightness and these are directly mapped to PWM duty cycle. So level 1 is 0.1% level 100 is 10%, and so on.

What's improtant is to report:

2020–09–11: PinePhone multi-boot image deduplication tool complete

Since the last post, I was able to create the deduplicating tarball extractor tool exactly as I described it, and everything seems to work as expected.

The image size after compression shrank from 5.8 GiB to 5.2 GiB. :)

Now I have to figure out how to build a few different postmarket OS rootfs variants, to give this shiny new tool a real stress test.

I'm pretty sure I can add maybe 3 postmarket OS mobile UI variants without increasing the image size significantly. After all, the image already contains sxmo, which is based on pmOS, so most of the base data is already in the image.

Then, I'll implement p-boot DTB auto-selection, and the image will be ready to enter beta state.

New kernels

I've released new 5.9 kernels with a few more new tweaks:

2020–09–10: PinePhone multi-boot image deduplication

Btrfs has a nice feature where you can tell it when creating a file to initially use content of some already existing file. This saves space on the filesystem, because the duplicit files can share the same data space without needless data copies. This is almost like hard links, except that data stop being shared, when the file is written.

This can be used in the multi-boot image to save even more space. The multi-distribution image offers many opportunitites for this kind of space saving, due to distributions sharing quite a lot of identical files.

I've written a tool that takes all the rootfs tarballs on input, and goes through all of them, finds duplicities, and calculates space used in total by all files, and space used by unique files. For the current multi-distro image, the numbers look like this:

So the needless overhead of duplicate files accross all included distributions currently is 12.6%, btrfs compression ratio is 50% and the wasted space on the compressed filesystem by the duplicate files is 762 MiB.

That's fairly significant. I can fit another distribution in the wasted space alone. :)

If the included distributions would be much less diverse. For example when including multiple UI variants of postmarket OS, the savings could be even greater. I can potentially add 3 to 4 variants of pmOS if I was able to share duplicate files content, without the current image size changing at all.

I'll try to add FICLONE support to my extraction tool. The tool will basically put content hashes of the files and their paths into a hash table as they are being extracted, and if there's a duplicity, instead of extracting the content of a file, it will use FICLONE to share the data of previously extracted copy of the file.

Resizing the partition after flashing the image

To get more free space when using the multi-image, you can resize it to the full size of your SD card. You can do this from a PC using a SD card reader.

Warning! You'll need to run all the commands as root.

Multi-distro main data partition block device can be found with:

blkid -lt PARTUUID="12345678-02"

For me it prints:

/dev/mmcblk0p2: UUID="0cb50b0b-77a3-45bd-a605-857472b88281" \
   UUID_SUB="a1d75445-1ffe-4e6f-860a-1acdbb4aad2a" \
   BLOCK_SIZE="4096" TYPE="btrfs" PARTUUID="12345678-02"

So for me, btrfs partition is /dev/mmcblk0p2 and the device containing the partition table is /dev/mmcblk0. For you it may be different. Make absolutely sure you're using the correct block device!

Resize the main partition

First, dry run:

echo ", +" | sfdisk -n -N 2 /dev/mmcblk0

Outputs:

Quote:
Disk /dev/mmcblk0: 29.74 GiB, 31914983424 bytes, 62333952 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x12345678

Old situation:

Device        Boot  Start      End  Sectors  Size Id Type
/dev/mmcblk0p1 *      8192  262143  253952  124M 83 Linux
/dev/mmcblk0p2      262144 15359999 15097856  7.2G 83 Linux

/dev/mmcblk0p2:
New situation:
Disklabel type: dos
Disk identifier: 0x12345678

Device        Boot  Start      End  Sectors  Size Id Type
/dev/mmcblk0p1 *      8192  262143  253952  124M 83 Linux
/dev/mmcblk0p2      262144 62333951 62071808 29.6G 83 Linux
The partition table is unchanged (--no-act).

Now, verify changes look sane, and do a real resize:

echo ", +" | sfdisk -N 2 /dev/mmcblk0

Resize the main filesystem

Now that the partition is stretched to the end of the SD card, let's resize the contained filesystem too. Btrfs needs to be mounted during resize. The steps thus are:

mount /dev/mmcblk0p2 /mnt
btrfs filesystem resize max /mnt
df -h
umount /mnt

(Make sure when substituting your device for /dev/mmcblk0p2 that you keep the partition number at the end – p2. Thanks to Gazza for pointing out this pitfall.)

You should see:

Resize '/mnt' of 'max'
/dev/mmcblk0p2  30G  6.6G  24G  23% /mnt

And that's all. :)

Forum post

I've posted about my image on the pine64.org forum. That post contains some additional information:

https://forum.pine64.org/showthread.php?tid=11347

2020–09–08: PinePhone multi-boot finishing touches / modem improvements

So I dealt with a few of the „annoying details“ from the last post, and created a repository for my build scripts and overrides for each distro.

The last thing to do prior to final testing is to add support for selecting dtb for PP 1.1 or 1.2 automatically.

Modem power manager

I've also improved my modem power manager a bit, and added support for fast poweroff mode that's available in the newer modem firmwares. (I believe it's those firmware that are in Braveheart and onwards.) Modem can now be powered off in ~2 seconds.

I've also fixed issues with RDY not being received. Looks like one of the persistent configuration options makes the modem not send RDY. I just poll for the successfull empty AT command, as a way to detect when the modem is ready, and that seems to be more reliable.

All these changes are avialable in my 5.8 and 5.9 kernel branches.

2020–09–08: PinePhone multi-boot image optimizations

Since I'll be using my kernel for all the distros anyway, I thought I might spice it up a bit, and use something a bit more modern than ext4.

Instead of complicated partitioning scheme that wastes space and is not easy to optimize, I decided to use a single btrfs filesystem for everything with each distribution having its own subvolume, and all files being compressed by zstd. This made it possible to create a 8GiB SD card image, with 9 distributions on it.

The image creation process is now much simpler. First I create a distros/ directory with a subdirectory for each distribution. Then I download and prepare rootfs.tar.zst tarballs, for each distribution I want to have in the image, as previously described in this blog.

And then I create the image:

#!/bin/sh

if [ "$(whoami)" != "root" ] ; then
        exec sudo sh "$0" "$@"
fi

set -e -x

. ./config

rm -f $IMG
truncate -s $IMGSIZE $IMG

sfdisk -W always $IMG <<EOF
label: dos
label-id: 0x12345678
unit: sectors
sector-size: 512

4M,124M,L,*
128M,,L
EOF

L=`losetup -P --show -f $IMG`

mkfs.btrfs ${L}p2

mkdir -p m
mount -o compress-force=zstd:15 ${L}p2 m

for ddir in distros/*
do
        test -f $ddir/config || continue
        name=${ddir#distros/}
        btrfs subvolume create m/$name
        bsdtar -xp --numeric-owner -C m/$name -f $ddir/rootfs.tar.zst
done

./mkimage-apply-fixes.sh

umount m
losetup -d "$L"

./mkimage-boot.sh

This does away with the need for having to deal with partition numbers. Subvolumes have the name of the distro, and are easily referenced in the boot arguments.

During boot I just mount the appropriate subvolume instead of the main volume of the btrfs filesystem. All that's needed is to pass rootfsopts=subvol=distro-name.

Modifying and accessing files in all distributions at once also becomes trivial. All I need to do is to mount the main volume, and the subvolumes for each partition appear nicely available there as subdirectories in the main volume. Previously I had to juggle with mounting 10 arbitrarily named partitions.

It's also very easy to add/remove distributions without the need for re-partitioning, and all distributions share the remaining free space on the SD card.

I find this setup excellent! Btrfs partition with 9 simultaneously installed distributions takes just 6.5GiB of space. Previously just mobian alone took 3GiB of used space, and a 4 GiB partition.

Making the image bootable

To make the image bootable, I just build the boot.conf file for p-boot, based on a per-distribution config file, and use it to format the boot partition.

Config file looks like:

version=2021-05-27
name="Ubuntu Touch"
bootargs="logo.nologo vt.global_cursor_default=0"

And the script to format make the image bootable:

#!/bin/sh

if [ "$(whoami)" != "root" ] ; then
        exec sudo sh "$0" "$@"
fi

. ./config

# loglevel=15
serial="console=ttyS0,115200 earlycon=ns16550a,mmio32,0x01c28000"
silent="quiet loglevel=0 systemd.show_status=false"
bootargs_base="$serial $silent cma=256M console=tty1 consoleblank=0 panic=3 rw rootwait root=PARTUUID=12345678-02 rootfstype=btrfs rootflags=compress-force=zstd,nodatacow,subvol"
kbuilds=../builds

echo "device_id = Distro Demo Image 2021-10-01" > boot.conf

no=0
for ddir in distros/*
do
        test -f $ddir/config || continue
        dist=${ddir#distros/}

        (
                . ./$ddir/config
                        
                echo "no = $no"
                echo "  name = $name $version"
                echo "  atf = ../p-boot/dist/fw.bin"
                echo "  dtb = $kbuilds/ppd-5.14/board-1.1.dtb"
                echo "  dtb2 = $kbuilds/ppd-5.14/board-1.2.dtb"
                echo "  linux = $kbuilds/ppd-5.14/Image"
                echo "  bootargs = $bootargs_base=$dist $bootargs"
                if test -f files/$dist.argb ; then
                        echo "  splash = files/$dist.argb"
                else
                        echo "  splash = files/xnux.argb"
                fi
        ) >> boot.conf

        no=$(($no+1))
done

# JumpDrive is special
if test -d distros/jumpdrive
then
        (
                echo "no = $no"
                echo "  name = Jumpdrive 0.8"
                echo "  atf = ../p-boot/dist/fw.bin"
#               echo "  dtb = distros/jumpdrive/sun50i-a64-pinephone-1.1.dtb"
#               echo "  dtb2 = distros/jumpdrive/sun50i-a64-pinephone-1.2.dtb"
#               echo "  linux = distros/jumpdrive/Image"
                echo "  dtb = $kbuilds/ppd-5.14/board-1.1.dtb"
                echo "  dtb2 = $kbuilds/ppd-5.14/board-1.2.dtb"
                echo "  linux = $kbuilds/ppd-5.14/Image"
                echo "  initramfs = distros/jumpdrive/initramfs"
                echo "  bootargs = loglevel=0 silent console=tty0 vt.global_cursor_default=0"
                echo "  splash = files/jumpdrive.argb"
        ) >> boot.conf

        no=$(($no+1))
fi

set -e -x

../p-boot/.build/p-boot-conf-native . boot-part.img
zstd boot-part.img
exit

L=`losetup -P --show -f $IMG`
../p-boot/.build/p-boot-conf-native . ${L}p1
losetup -d $L

dd if=../p-boot/.build/p-boot.bin of=$IMG bs=1024 seek=8 conv=notrunc

And that's really all there's to it, as far as making a bootable multi-distro image is concerned.

Some annoying details are still remaining

Support

If you'd like to support this effort, you can contribute at https://xnux.eu/contribute.html

2020–09–07: PinePhone multi-boot image boot testing

So let's say that distros are really finicky about where they're booted from if their initramfs image is used to boot them. Initramfs images mostly get in the way and don't offer much for the multi-boot image, so the simple solution is to not use them at all. All distros should be bootable fine without them.

The process should be quite simple: Linux mounts root partition provided via root= boot parameter at / and runs /bin/init. The distro itself should not care about what the real partition is, because it's already running from it. It should just not do anything stupid to the current rootfs mount, and just run.

Actually most modern distros should run fine with an empty /etc/fstab, in this situation.

Using my kernel

To have some unity/sanity, I decided to use my own kernel with all necessary drivers built in. This way I don't need to copy my kernel's modules to rootfs of each distro. I did also built firmware binaries into the kernel image, so it's all self-contained.

This approach has some other benefits:

Problems encountered so far

Other than this, each distro seems to boot fine with my 5.9 kernel.

The last step is figuring out how to make modem initialize properly on each distro. This will involve finding the script that powers up the modem and replacing it with echo 1 > /sys/class/modem-power/modem-power/device/powered.

See https://megous.com/dl/tmp/multi2.mp4 for another preview. :)

2020–09–05: PinePhone multi-boot image

I made a further progress on the multi-boot image:

The result is:

I have yet to boot test all the distributions. Prior to that, I have to check whether the distros are not doing something destructive on boot, like resizing or deleting partitions, etc.

So far my boot.conf for p-boot looks like this:

#!/bin/sh

#serial="console=ttyS0,115200 earlycon=ns16550a,mmio32,0x01c28000 loglevel=15"
bootargs="cma=256M console=tty1 consoleblank=0 quiet loglevel=1 panic=3 rw rootwait root=PARTUUID=12345678"

dists=/mnt/dists

#/dev/mmcblk0p2: PARTUUID="12345678-02"  mobian (f2fs, 5.7 kernel, PP 1.0-1.2)
#/dev/mmcblk0p3: PARTUUID="12345678-03"  KDE neon (ext4 only, 5.7 kernel, PP 1.1 only)
#/dev/mmcblk0p5: PARTUUID="12345678-05"  arch (f2fs, my 5.9 kernel, , PP 1.0-1.2)
#/dev/mmcblk0p6: PARTUUID="12345678-06"  sxmo (ext4 only, 5.7 kernel, PP 1.1, 1.2)
#/dev/mmcblk0p7: PARTUUID="12345678-07"  lune (ext4 only, 5.5 kernel, PP 1.1 only)
#/dev/mmcblk0p8: PARTUUID="12345678-08"  maemo (ext4 only, 5.6 kernel, PP 1.1 only)
#/dev/mmcblk0p9: PARTUUID="12345678-09"  ut  (ext4 only, 5.6 kernel, PP 1.1 only)
#/dev/mmcblk0p10: PARTUUID="12345678-0a" sailfish (ext4 only, 5.6 kernel, some custom DTB with partial support for up to PP 1.2)
#/dev/mmcblk0p11: PARTUUID="12345678-0b" pureos (f2fs, 5.8 kernel, PP 1.0-1.2)

zcat $dists/part2/boot/Image.gz > $dists/part2/boot/Image
#zcat $dists/part3/boot/vmlinuz-5.7.0-pine64-g3823929aa > $dists/part3/boot/Image
#zcat $dists/part8/boot/Image.gz > $dists/part8/boot/Image
#zcat $dists/part9/boot/vmlinuz > $dists/part9/boot/Image

cat << EOF
device_id = Multi-Distro Demo Image

no          = 0
  name      = Mobian
  atf       = fw.bin
  dtb       = $dists/part2/boot/dtb/allwinner/sun50i-a64-pinephone-1.1.dtb
  linux     = $dists/part2/boot/Image
  initramfs = $dists/part2/boot/initrd.img
  bootargs  = $bootargs-02 splash plymouth.ignore-serial-consoles vt.global_cursor_default=0
  splash    = files/mobian.argb

no          = 1
  name      = KDE Neon
  atf       = fw.bin
  dtb       = $dists/part3/boot/dtb
  linux     = $dists/part3/boot/Image
  initramfs = $dists/part3/boot/initrd.img-5.7.0-pine64-g3823929aa
  bootargs  = $bootargs-03 splash
  splash    = files/neon.argb

no          = 2
  name      = Arch Linux ARM
  atf       = fw.bin
  dtb       = $dists/part5/boot/dtbs/allwinner/sun50i-a64-pinephone-1.1.dtb
  linux     = $dists/part5/boot/Image
  initramfs = $dists/part5/boot/initramfs-linux.img
  bootargs  = $bootargs-05
  splash    = files/arch.argb

no          = 3
  name      = Sxmo
  atf       = fw.bin
  dtb       = $dists/part6/boot/sun50i-a64-pinephone-1.1.dtb
  linux     = $dists/part6/boot/vmlinuz-postmarketos-allwinner
  initramfs = $dists/part6/boot/initramfs-postmarketos-allwinner
  bootargs  = $bootargs-06 init=/init.sh PMOS_NO_OUTPUT_REDIRECT PMOS_FORCE_PARTITION_RESIZE pmos_root=/dev/mmcblk0p6
  splash    = files/sxmo.argb

no          = 4
  name      = Lune OS
  atf       = fw.bin
  dtb       = $dists/part7/boot/sun50i-a64-pinephone.dtb
  linux     = $dists/part7/boot/Image
  initramfs = $dists/part7/boot/initramfs-uboot-image-pinephone.uboot
  bootargs  = $bootargs-07 bootmode=normal LUNEOS_NO_OUTPUT_REDIRECT
  splash    = files/lune.argb

no          = 5
  name      = Maemo Leste
  atf       = fw.bin
  dtb       = $dists/part8/boot/allwinner/sun50i-a64-pinephone-1.1.dtb
  linux     = $dists/part8/boot/Image
  bootargs  = $bootargs-08 fbcon=rotate:1
  splash    = files/maemo.argb

no          = 6
  name      = Ubuntu Touch
  atf       = fw.bin
  dtb       = $dists/part9/boot/dtb
  linux     = $dists/part9/boot/Image
  initramfs = $dists/part9/boot/initrd.img
  bootargs  = $bootargs-09 systempart=/dev/mmcblk0p9 devnum=0 logo.nologo vt.global_cursor_default=0
  splash    = files/ut.argb

no          = 7
  name      = Sailfish
  atf       = fw.bin
  dtb       = $dists/part10/boot/sun50i-a64-pinephone-1.1.dtb
  linux     = $dists/part10/boot/Image
  bootargs  = $bootargs-0a
  splash    = files/sailfish.argb

no          = 8
  name      = Pure OS
  atf       = fw.bin
  dtb       = $dists/part11/boot/dtb/allwinner/sun50i-a64-pinephone-1.1.dtb
  linux     = $dists/part11/boot/Image
  initramfs = $dists/part11/boot/initrd.img
  bootargs  = $bootargs-0b init=/sbin/init splash plymouth.ignore-serial-consoles
  splash    = files/pureos.argb


EOF

2020–09–04: Pinebook Pro and Levinboot

Pinebook Pro is a very nice laptop. Software support just has a few annoying quirks:

Thankfully, a discord user CrystalGamma took to creating Levinboot, which is a specialized bootloader for Pinebook Pro and other similar RK3399 based computers. Its goals are almost identical to my p-boot project: boot the kernel as fast as possible from local storage and get out of the way. Its limitations are similar too. No or very limited support for partition schemes, no support for traditional filesystems, etc.

Version 0.7.1 was released a few days ago, so I decided to give it a try on my Pinebook Pro, and I'm happy to report that it works quite nicely, and it looks like it's going to have a bright future. :)

Boot process now shows a feedback. At first, red LED turns on for 1 sec, then green LED turns on for another sec, then red LED turns off and in another 2–3 seconds the tty shows up, and the system is interactive. That's with a small 7MiB gzip compressed kernel payload. I also tried a large 20MiB zstd compressed payload from eMMC, but that doesn't yet work as expected. I get load speeds of about 5MiB/s with that.

With lack of support for system suspend, fast boot times help make the laptop more useable.

My Levinboot setup

I use eMMC to load the payload, and SD card to load Levinboot itself. This way I can easily upgrade and test new Levinboot versions, and not worry about it breaking, because I just pop out a SD card, and update the bootloader from my PC, in case something fails.

To get Levinboot, I compile it from source code:

CROSS=aarch64-linux-gnu
export CC=$CROSS-gcc
export OBJCOPY=$CROSS-objcopy
export LD=$CROSS-gcc

git clone https://gitlab.com/DeltaGem/levinboot.git
cd levinboot
# currently contains fixes for reading GPT partition table
git checkout dev
./configure.py --payload-emmc --payload-zstd --payload-initcpio --with-tf-a-headers $ATF_DIR/include/export
ninja

# Copy levinboot-sd.img to my Pinebook Pro
scp levinboot-sd.img root@pbp:/boot/levinboot-emmc.img

Then I install it to SD card from Pinebook Pro itself, along with flashing payload to eMMC:

#!/bin/bash
set -e -x

ROOT_UUID=6795bcbb-fba8-4d22-81ef-dc628324b021

BOOTOPTS=(
  console=tty1
  video=eDP-1:1920x1080@60
  rd.luks.uuid=$ROOT_UUID
  rd.luks.name=$ROOT_UUID=root
  rd.luks.options=timeout=0,discard
  root=/dev/mapper/root
  rootfstype=f2fs
  rootflags=x-systemd.device-timeout=0
  rootwait
  rw
  mitigations=off
  quiet
  loglevel=2
)
BOOTOPTS="${BOOTOPTS[@]}"

cp -f board.dtb board-lv.dtb
fdtput -pt s board-lv.dtb /chosen bootargs "$BOOTOPTS"

truncate -s 0 payload-lv.img
zstd -zc bl31.elf >> payload-lv.img
zstd -zc board-lv.dtb >> payload-lv.img
zstd -zc Image >> payload-lv.img
zstd -zc initramfs.img >> payload-lv.img

dd if=payload-lv.img of=/dev/mmcblk2p1
dd if=levinboot-emmc.img of=/dev/mmcblk1 seek=64
#dd if=levinboot-sd.img of=/dev/mmcblk1 seek=64

sync

When booting from eMMC, Levinboot expects to find payload in a special GPT partition. I just use a single partition with GUID e5ab07a0-8e5e-46f6-9ce8-41a518929b7c.

This is all documented in detail in the Levinboot README file.

2020–09–02: Progress on the multi-boot image

Basic process of creating a multi-boot image is:

At this point each distro will be bootable, but may fail if it has some hardcoded expectations for the partition table structure, filesystem labels, etc. This will need to be fixed manually.

Getting the rootfs tarballs

Many distros just publish a block device image. Getting the rootfs files from the image requires mounting the filesystems contained in the image and backing up all files with bsdtar. Something like this can be used in most cases:

L=`losetup -P --show -f "$1"`
BOOT_PART=1
ROOT_PART=2

mkdir -p m
mount ${L}p${ROOT_PART} m
test -n "$BOOT_PART" && mount ${L}p${BOOT_PART} m/boot
bsdtar -cvf - -C m --numeric-owner . | zstd -z -4 - > rootfs.tar.zst
test -n "$BOOT_PART" && umount m/boot
umount m

losetup -d "$L"

I excluded the distros that don't publish rootfs tarballs or block device images, because they're too much hassle, at the moment. That is pmOS and nemo mobile.

Images/tarballs can be downloaded at:

Excluded:

Partitioning

p-boot requires MBR partitions, so we need to use that with an extended partitioning scheme, to be able to have more than 4 partitions. Creating the partition table is quite simple. We just need to decide how much space each distro will need. zstdcat rootfs.tar.zstd | wc -c can give a clue. There are some larger than 3GiB, but most fit within 2GiB, so let's create a table like this:

At least 32GiB SD card is needed.

Warning: The following script is broken, and leads to first two partitions overlapping. Fixes welcome. :)

sfdisk -W always /dev/mmcblk0 <<EOF
label: dos
label-id: 0x12345678
unit: sectors
sector-size: 512

4M,1020M,L,*
,4G,L
,4G,L
,21G,Ex
,3G,L
,3G,L
,3G,L
,3G,L
,3G,L
,3G,L
,,L
EOF

Filesystems

Now let's create filesystems. My kernel supports f2fs, so if I boot any of the above distros with my kernel, I can just use f2fs everywhere. Unfortunately, I don't know which of the above distros are able to boot from f2fs with their own kernel.

They all probably support ext4. F2FS is much better for SD cards, though. A conundrum :-)

After an investigation, I've decided on this layout (PARTUUIDs will be useful later on).

The table also has the current status of the distros and their assignment to the partitions.

/dev/mmcblk0p1: PARTUUID="12345678-01"  p-boot
/dev/mmcblk0p2: PARTUUID="12345678-02"  mobian (f2fs, 5.7 kernel, PP 1.0-1.2)
/dev/mmcblk0p3: PARTUUID="12345678-03"  KDE neon (ext4 only, 5.7 kernel, PP 1.1 only)
/dev/mmcblk0p5: PARTUUID="12345678-05"  arch (f2fs, my 5.9 kernel, , PP 1.0-1.2)
/dev/mmcblk0p6: PARTUUID="12345678-06"  sxmo (ext4 only, 5.7 kernel, PP 1.1, 1.2)
/dev/mmcblk0p7: PARTUUID="12345678-07"  lune (ext4 only, 5.5 kernel, PP 1.1 only)
/dev/mmcblk0p8: PARTUUID="12345678-08"  maemo (ext4 only, 5.7 kernel, PP 1.0-1.2)
/dev/mmcblk0p9: PARTUUID="12345678-09"  ut  (ext4 only, 5.6 kernel, PP 1.1 only)
/dev/mmcblk0p10: PARTUUID="12345678-0a" sailfish (ext4 only, 5.6 kernel, some custom DTB with partial support for up to PP 1.2)
/dev/mmcblk0p11: PARTUUID="12345678-0b" pureos (f2fs, 5.8 kernel, PP 1.0-1.2)

The status is not great, only maybe 3 distros really support PinePhone 1.2, and only one uses a kernel that's not EOLed upstream. Triste!

Anyway, this leads to:

for bd in /dev/mmcblk0p{3,6,7,8,9,10}
do
  mkfs.ext4 $bd
done

for bd in /dev/mmcblk0p{2,5,11}
do
  mkfs.f2fs $bd
done

After this we can just mount all the partitions one by one and extract the prepared rootfs contents there, with:

mount /dev/mmcblk0p# /mnt
bsdtar -xp --numeric-owner -C /mnt -f rootfs.tar.zst
umount /mnt

That's about it for today. I'll do first boot tests in the following days, and you can look forward to a bootloader setup guide. :)

2020–09–01: More p-boot cleanups and an example configuration

With some suggestions from Yoda, I added a few more sanity checks to the boot partition configurator, and more importantly, I prepared an example boot configuration directory with some ready-made p-boot themes, splashscreens and scripts.

This should make it much easier to get people started using p-boot, even if they will not read the (currenlty somewhat outdated) README file.

The sample config is in the example/ directory in p-boot git repo.

2020–08–31: Releasing p-boot GUI bootloader

Today I finally decided to release display support for p-boot. Along with it, p-boot's code also got quite a bit of cleanup. At this point main.c is fairly readable. There's also a new support for 3GiB variant of PinePhone, that TL Lim sent me some time ago.

I've copied my megatools.megous.com website template, and made it work better on mobile phones, to make a nice landing page for it, too: xnux.eu/p-boot

Since previous publicly released version, p-boot can now be configured to show the name of the phone on the boot screen, which is quite useful if you're like me and TL Lim keeps sending you PinePhones for development every other full moon. :)

I also use the name to describe HW mods I did to the particular phone.

There are some things I'd still like the p-boot to do. Since p-boot has direct access to PMIC, I thought of using it for following things:

And more:

In the other news, TODO

I tried to cleanup and organize my kernel upstreaming TODO list, and it's still too long. So instead I decided to give p-boot a little boost in exposure by preparing a multi-boot image that will feature a range of diverse distributions. With help of #pinephone channel users, I came up with this list:

Looks like making some of these share space and play nice with the others will be quite a challenge. If you're a distro maintainer, publishing a ready-made rootfs tarball is a great way to simplify your inclusion into a multi-boot image.

The image will be great for people new to PinePhone to try various distros with very little hassle. Just flash one image to SD card, and select the distro on boot.

I'll try documenting the image creation steps, but it will most probably be a one shot experiment for me. So if anyone will want to maintain this image further into the future, they'll be very wellcome.

I'm still undecided whether I'll force my kernel on all the above distros, or let them use their own. Probably the latter.

2020–08–31: Getting started

I already maintain xnux.eu which is a regular website that provides summarized information about my projects organized by device/topic. I decided to start this log to provide more of a in-progress information to people interested in my PinePhone related work. That is mostly kernel drivers/bootloader development, and performance optimizations.

Send any feedback you like to x@xnux.eu.