megi's PinePhone Development Log RSS

2023–02–17: Pinephone keyboard firmware update (1.3)

I revisited the pinephone keyboard firwmare with a goal to further optimize the power consumption of the keyboard MCU. Originally I managed to optimize it down to 9 mW power consumption when no key was pressed, and 20 mW when the key was pressed. Recently I noticed that when the phone is left in the keyboard, the phone's battery discharges at faster rate than expected – a few weeks and the battery is dead instead of expected 2 months.

Not nice. Ideally one should not worry about phone battery discharging while leaving the phone in the keyboard.

The datasheet for the keyboard MCU tells us on page 150 that achievable power consumption in power down mode should be ~100 uA, but the MCU is consuming 1700–3000 uA with the current firmware.

There's no guide for optimizing the power use of the MCU, so I had to play around and try various things. It turned out most of the power consumption was due to leaving MCU inputs floating. Enabling pullups and/or switching unused pins to output mode made it so that the MCU started consuming expected 100uA in so called „power down“ mode. In this mode, the MCU oscillators are shut off (not even timers run in this mode), and the MCU can only recover from this via external signal.

While it's nice that the MCU can follow the supply current specification, I noticed during the debugging session that at times I was able to get even lower power consumption. I tried to isolate the source of the reduction and it turned out to be a 3.3V regulator used for USB interface of the MCU. MCU USB is not exposed to users of the keyboard, and the user firmware turns it off. Nevertheless, the firmware left this regulator enabled. Turning it off gained us extra 50 uA of power savings.

After all this optimization, with the latest firmware, keyboard was consuming 50% less power than minimum achievable level given in the datasheet. :D So about 250 uW. Great. Looks like I unlocked some new achievement, …

Except the problem with this is that I2CA bus doesn't seem to work when USB 3.3V regulator is disabled, so we can't control the keyboard from the phone (MCU works though, and can communicate over other GPIO pins…). I2CA seems to be marked as 3.3V only in the datasheet, so I guess I2CA not working with internal 3.3V regulator disabled meakes some sense. We'll have to live with 700 uW.

There was also another bug in the firmware, that prevented entry to „power down“ mode after the user pressed the first key after keyboard powerup. So basically the MCU never even got to that original 9 mW powerdown mode power consumption level in practice and it was just constantly consuming 20 mW. This was the cause of unexpectedly fast battery draining.

So in summary, firmware v1.3 brings in ~30× reduction in power use. That translates to theoretical increase in how long the phone can stay powered off in the keyboard case (on full battery) from 23 days to 654 days (1.8 years). This means that you can leave the phone connected to the keyboard without any worries and without it draining the phone battery in any meaningful way.

Firmware upgrade howto

To upgrade the firmware, you need to download the updated firmware from my website, extract it and run:

./arm64/ppkb-i2c-flasher -e i2c -i fw-user.bin write reset

The tool will output this on success:

Opening keyboard I2C device
Flashing code ROM
Finishing flashing
Restarting the MCU

And that's all there's to it. :)

Enjoy!

Active mode

Video from testing/debugging:

Addendum (2023–03–07)

For some reason firmware 1.3 doesn't have very reliable I2C connection. Probably because the MCU power down mode affects the I2C slave modeule/code somehow, now that the chip actually goes into powerdown way more often.

A lot more testing will be needed. :(

2023–01–12: Pinephone cpuidle

I received an e-mail about non-working CPUIDLE on Pinephone on pmOS and Arch (presumably Danct variant). So this is a bit of a PSA, since working cpuidle is a very nice to have power saving feature, and it would be sad for major Pinephone distros to be missing it due to a simple build config issue in TF-A.

Hello Chris,

On Wed, Jan 11, 2023 at 09:57:28PM +0000, Chris Xxxxxxxxxx wrote:
> Dear Megi,
>
> I have noticed when working with the Pinephone that cpuidle does not seem to
> be operational on distributions using your kernel, either (Postmarketos)
> with a slightly diffferent config, or Arch.
>
> Looking at  /sys/devices/system/cpu/cpuidle/current_driver  the content is
> 'none' where normally you would expect it to be something like arm_idle or
> similar.

https://megous.com/dl/tmp/9bb7d2f832ddd547.png

> This shows up in the apparently short battery life and if cpupower is
> installed you get:
>
> $ sudo cpupower idle-info
> [sudo] password for alarm:
> CPUidle driver: none
> CPUidle governor: menu
> analyzing CPU 0:
>
> CPU 0: No idle states

https://megous.com/dl/tmp/7f2d4b461e59d8e0.png

> Is this what you would expect for an Allwinner CPU? I don't have much
> experience of the Sunxi architecture, maybe C-states are not supported? The
> Config looks OK, with (Postmarketos config)

No, it's not what I'd expect. I suspect pmOS is not using a TF-A and/or crust
version that supports cpuidle, which is why cpuidle does not work on it.

Check that you build TF-A's bl31 with SUNXI_AMEND_DTB=1

For reference, my build script for TF-A/crust are here:

  https://megous.com/git/p-boot/tree/fw

The whole sunxi_prepare_dtb() function is disabled if you don't explicitly
specify SUNXI_AMEND_DTB=1 during TF-A build, and this function must be
enabled for cpuidle to work:

https://github.com/ARM-software/arm-trusted-firmware/blob/master/plat/allwinner/common/sunxi_prepare_dtb.c#L38

Default is SUNXI_AMEND_DTB=0:

https://github.com/ARM-software/arm-trusted-firmware/blob/master/plat/allwinner/common/allwinner-common.mk#L77

kind regards,
        o.

> |CONFIG_CPU_IDLE=y CONFIG_CPU_IDLE_MULTIPLE_DRIVERS=y #
> CONFIG_CPU_IDLE_GOV_LADDER is not set CONFIG_CPU_IDLE_GOV_MENU=y #
> CONFIG_CPU_IDLE_GOV_TEO is not set CONFIG_DT_IDLE_STATES=y
> CONFIG_DT_IDLE_GENPD=y # ARM CPU Idle Drivers # CONFIG_ARM_PSCI_CPUIDLE=y
> CONFIG_ARM_PSCI_CPUIDLE_DOMAIN=y The Arch Arm distribution for the Pinephone
> seems to use your defconfig unchanged, and the running result is the same.
> Best regards Chris |

2022–12–02: Pinephone kernel news and some bits about the keyboard, too

During 6.1 Linux development cycle there were several largish changes in my kernel, which are worth sumarizing a bit. So here you go:

sun4i-drm driver fixes

Mainline sun4i-drm diver (display driver for original Pinephone's SoC display engine) has multiple known bugs in the DRM plane handling. I had some workarounds for some of the issues (drm: sun8i-ui/vi: Fix layer zpos change/atomic modesetting, drm: sun4i-drm: Recover from occasional HW failures), but these workardounds still didn't catch every corner case. For example during some UI app development for my multi-boot image, I noticed that when the DRM app turns off the crtc, when it shuts down, the driver will fail to configure the DRM plane for Linux kernel console correctly, when Linux switches to the console later on.

So I decided to rewrite the plane z-position and update handling from scratch, and move plane position setup to CRTC component of the DRM driver, where it's much easier to iterate over configuration of all planes of that CRTC and figure out what values to write to display engine's registers to make planes display in proper z-order on the screen.

The new patches are simpler to think about, mostly delete code, and are smaller. They are also implementing a solution to the issues that was suggested by the driver maintainer, when some workarounds were submitted to upstream a few years back, so they should be more upstreamable.

The new patch is drm/sun4i: Fix layer zpos change/atomic modesetting. It needs some preparatory work in the driver, which is split out to drm/sun4i: Unify sun8i_*_layer structs and drm/sun4i: Add more parameters to sunxi_engine commit callback.

sun6i-csi driver multi-camera support rewrite

Mainline sun6i-csi driver got a lot of changes during 6.1 development cycle. So much changes, that my multi-camera support patch was not at all aplicable anymore.

So I ended up rewriting the patch from scratch, with focus on making as little changes as possible. The new patch is considerably smaller and is ready for upstreaming. I still need to update DT binding documentation, which is also required for upstreaming.

Pinephone Pro display fix for 60 Hz refresh rate

Pinephone Pro LCD display driver had several issues, among them the most elusive so far was that the true refresh rate was 53 Hz instead of expected 60 Hz.

Fix proved to be somewhat similar to correcting a similar issue on original Pinephone LCD. First, clock constraints and possibilities had to be taken into account. RK3399 clock subsystem/PLLs can't generate arbitrary clock frequencies, but LCD requires a precise pixel clock frequency to achieve precisely 60 Hz of refresh rate.

If you ask the Linux clock subsystem to get you 72 MHz for the display clock, because that's what you need, it may silently instead set up the clock for 66.6MHz, and be all happy about it. Needless to say, this will not result in having any precision in the refresh rate at all.

Solution to the problem is to pick some frequency we can actually get from the clock sybsystem, and massage the „factory“ display mode settings in such a way, so that our chosen display clock frequency would lead to 60 Hz refresh rate. This is achieved in practice by padding or shortening the horizontal and vertical synchronization timings.

So this is what I did a long time ago, but it resulted in one of my Pinephone Pro's having a correct 60 Hz refresh rate, and the other to have corrupted output to display (horizontal lines were wobbling around as if pixel data were sometimes skipped and sometimes added during the line scan out – but vertical synchronization was fine, and the display always show the correct number of lines, and the lines were aligned well vertically).

I though this was some issue with display panel controller configuration, which is done each time the display panel is powered up. I wrote a patch to allow modifying panel mode and configuration sequence from userspace, so that experimenting with various changes was quick and efficient, but nothing I did affected the corruption issue.

Eventually I looked again at what could be wrong on the RK3399 SoC side, and in particular on DSI DPHY code and noticed that the bandwidth calculation is different in Rockchip BSP code compared to the one used mainline. I switched the calculation to what Rockchip uses in their kernel, which fixed the corruption issue on one of my Pinephone Pros. This change will affect every device using the RK3399 with a MIPI DSI panel, so I doubt it's acceptable mainline, but alternatively, it's possible to implement direct configuration of bandwidth from device tree, so that it doesn't need to be calculated, and can be controlled more precisely per board DT file.

In any case, Pinephone Pro LCD driver should now be working fully and correctly, so after some testing in the wild, it should be possible to start mainlining work for it, too.

Pinephone keyboard power consumption

I've noticed that my pinephone's that are connected to the keyboard are discharging a bit more quickly than expected. It's expected for pinephone keyboard to be constantly drainging Pinephone's main battery until it dies (yeah, not great but that's how it is), but it should be happening at ~8 mW. So with a fully charged battery, if Pinephone is left in the keyboard unused, it will last for about 2 months,… maybe slightly less.

The keyboard seems to be discharging the phone in a few weeks, though. So I've cut up the keyboard, so that I can measure the power consumption of the keyboard part of the circuit…

… and it seems to consume:

So the kernel driver does something to the keyboard firmware that makes it permanently consume 20mW and not go into low power mode, after the first key press.

Something for a TODO, to improve in the future. :) For now, it's best to remove the phone from the keyboard if you're not be using the phone for more than a few days, otherwise the keyboard will needlessly dicharge the phone's battery in about 3 weeks.

According to keyboard MCU datasheet, power consumption of <1mW should be possible in power down mode. But I was not able to achieve it last year, when writing the original firmware. That may be something to revisit when looking at the above bug, too, now that I've cut up the keyboard again anyway.

Pinephone keyboard power manager

I have integrated my power manager for the pinephone keyboard into my kernel and changed it a bit compared to where it was early in the year. Early this year, I thought I can use the capacity reported by the phone and by the keyboard driver to drive the algorithm behind this driver, but values reported by these power supply drivers are wildly inaccurate and tend to vary/jump around a lot, especially in response to control actions performed by the power management driver. Reported capacity is also wildly inaccurate. In this new version of the driver, I stopped relying on driver reported capacity of batteries, and switched to relying on battery voltage. All the driver needs for control purposes is „is the battery almost dis/charged?“ and using battery voltage to determine that is much more robust than some „capacity“ value dreamed up by the drivers. To avoid control loop oscillations, the driver now uses timers to keep some decision for a reasonable length of time, during which voltage changes are ignored.

Anyway, this driver manages charging cycle of both batteries so that the energy is used more wisely and provides LED triggers, uevents and combined cached state of the whole keyboard/phone combo in debugfs/kbpwr/state which is usable for summary capacity/discharge rate reporting in userspace. This is non-standard interface, so it will only work if you write some scripts for getting the values out of there. Standard userspace can still see individual batteries and display combined capacity for them somehow. For that purpose I've also added reporting of design total energy of the batteries to sysfs, so that userspace programs can better report combined remaining state of charge by taking into account that the batteries have different „sizes“. :)

The driver optimizes for the highest power efficiency. Power is used more directly without needless recharging of phone's battery from the keyboard battery.

The driver also performs calibration of kb battery internal resistance to improve reliability of capacity reporting of keyboard battery. Current value of internal resistance of the battery is critically important to guess the open circuit voltage (OCV, voltage when the battery is not under load) of the battery, which is used to lookup the battery's remaining capacity from a OCV->capacity lookup table.

The driver optionally provides emergency shutdown when both batteries are almost discharged.

You can enable and disable the driver's control loop via sysfs like this:

echo 1 > /sys/devices/platform/keyboard-power/disabled

The driver is still a bit experimental (it has verbose debugging enabled, and lacks testing in some corner case situations), and will probably be disabled by default in my final release of my Linux 6.1 branch.

Lots of other smaller things

My Linux 6.1 branch now uses latest mainline keyboard driver from Samuel Holland. I added some patches on top, to be able to experiment with various xkb settings in userspace. In particular, it's possible to disable the keyboard part of the driver, and also the Fn overlay layer (and implement it in userspace instead). All these changes can be seen in ppkb-6.1 branch.

There was also an issue with power key interrupt handling code on original Pinephone reported in pinephone chat, where if you released power key while the kernel was still resuming from sleep, the power key would get stuck in pressed state, possibly causing some issues to userspace, which is now fixed.

I've also split device tree for pinephone 1.2 and pinephone 1.2b, because I've not found other way to make magnetometer work on both of these Pinephone variants. This will likely not work great for distributions using U-Boot and U-Boot based Pinephone variant autodetection, which is only capable of determining Braveheart from other Pinephone variants. Detecting what magnetometer is used and selecting what 1.2 DT variant to use should be possible though. But it will require some U-Boot patches.

I've tried to make some of my internal tooling to work with the new magnetometer used in Pinephone 1.2b and noticed that buffered capture is not implemented in af8133j driver, so I've added support for that, too.

Overall 6.1 is looking out to be a very nice kernel release for Pinephone and Pinephone keyboard.

2022–10–08: Pinephone UART HW Issue

A few months ago I discovered a HW issue with Pinephone UART (the one that is used for serial console access). I discovered this when trying to use PP UART for a personal HW project, to communicate with some circuit on my Pinephone Breadboard (former keyboard).

The issue manifests as weird signal corruption of Pinephone UART signals when there's a low → high transition on RX line (say when you type something into a serial console, which causes activity on the RX line), which looks like this (zoomed in to a rising edge of RX signal):

Now those are repeated drops from 3.3V down to almost 2V for some weird reasons. It should be a nice square flat signal, not this mess. Also RX is being actively driven high during the L->H transition, it's not some open drain bus like I2C, so there must be something strongly pulling the RX line low on the phone side, for this to happen.

So I hooked up both signals to the scope, and saw even larger effect on the TX line (magenta):

Now that's quite something. :) Both signals are being corrupted, even when there's no activity on the TX line from the Phone side. So I tried to capture a few more instances of this issue, and managed to capture the interference during the time when TX line was driven low by the Phone:

So I disconnected the RX signal completely (so that it was floating), and this resulted in some cross-talk from TX to RX, but otherwise TX signal was completely clean:

At this point it was clear that the issue related to L->H changes on RX, and it started looking like the multiplexer that is used to switch the headphone jack receptacle between UART and audio modes via a kill-switch was somehow involved. Perhaps the mux was being temporarily switched to audio outputs. This would explain why RX input would be able to overdrive the TX output of the PIC chip on the breadboard and pull it down almost to 2V. It would also explain the simultaneous TX signal corruption. Headphone amplifier would be able to cause that.

The schematic around the multiplexer looks like this:

The multiplexer is siwtched to UART when the IN1/IN2 inputs are low (< 1V). Some signal must be getting to IN1/IN2 from RX/HP_L when RX is transiting from L->H. There's no other way to flip the mux to audio.

So the question is, how? First, IN1/IN2 are pulled low quite weakly with just a 47 kOhm resistor when SW1-F (headphone kill switch) is turned off, so it does not take much current to drive this input over 1V. Just 20 uA would do it.

Second, the switching of the mux to audio mode is short and temporary, so that suggests some capacitor being involved. We can even guesstimate the capacity from the timings on the scope captures. The first switch to audio lasts for 300 ns. The L->H transition is 0->3.3V. The switching threshold on IN1/IN2 is 0.5–1V, so the capacitor would have to charge from 0V to 2.3V in 300ns over a 47 kOhm resistor connected to 3.3 V, so about to 69% of the maximum voltage. What capacitance it has? :)

See eg. https://www.student-circuit.com/studentzone/transient-response-of-rc-circuit/

Charging to 69% of the max voltage takes about τ long. τ = RC so, C = τ / R = 300e-9 / 47e3 = ~6 pF. That's not much. So where could this be? Then I noticed that jack plugin detection is done by shorting HP_DET1 signal to HP_L via some mechanism in the receptacle, and that means that HP_L/RX → HP_DET1 → Q801 → R816 → IN1/IN2 now becomes a plausible path for the interfering signal. Q801 purpose seems to be to disconnect the plug-in detection when headphones are kill switched (I guess so that UART RX signal would not be connected to audio codec's plug-in detection input pin, which would lead to misdetection of RX activity for headpones plug-in). The problem though is that MOSFET transistors have significant parasitive capacitances between their leads. These are well specified in their datasheets. And guess what?

The parasitic capacitances for SSM3K35MFV are in the ballpark of what we guesstimated. So we have found the culprit at last. :)

H->L transitions when coupled to IN1/2 do not cause any problems, because they just further push the voltage on IN1/2 down below 0V and thus do not switch the MUX direction.

So to sum it up: Q801 is acting like a 9pF capacitor between its source and gate leads which passes signal from HP_L/RX to multiplexor switching input and thus causes it to switch to audio mode repeatedly for about 1.7us before it settles back to UART mode each time there is L->H transition on UART RX.

The mess lasts 1.7us, so the maximum usable baudrate on pinephone is around 115200 baud.

The repeated switching back and forth between audio/UART during those 1.7us is being caused by a sort of a positive loop, where a swith back to UART causes change in voltage from 2V → 3.3V which is again coupled to the IN1/IN2 and causes it to cross the switching threshold for audio mode again. This happens several times and we're lucky that it eventually stops and does not cause indefinite oscillation.

In the end, Pinephone's UART is reliable only in half-duplex at 115200. Any other use may result in corrupted transmission or reception of data.

If you make sure RX signal is always high, you may be able to use UART TX at high baudrates.

If there's TX and RX activity at the same time, even if RX timing is reasonably slow for RX to not be sampled by the receiver during those 1.7us of interference, concurrently happening TX communication will be affected, and may be corrupted when the TX output is sampled by the receiver during the ongoing interference.

Funny how one FET's parasitical capacitance can degrade uart from easily reaching 3 Mbaud to being cripled to 115.2k and half-duplex.

It's not usually an issue for a typical serial console use, because that's sort of half duplex. You type a character, the phone receives it and then echoes it back. This is a sequential process so TX and RX transmission is not happening at the same time. Of course, that's not a guarantee.

I tried to connect Pinephone to PIC for full-duplex communication over UART, and that doesn't work reliably at all for all the above reasons.

HW workaround

Unsoldering R816 would allow PP to use much higher baud rates, because it disconnects the parasitic signal.

Pinephone Pro

Pinephone Pro does not have this issue, because it doesn't try to gate the HP_DET signal via a FET connected to mux control signals.

HP_DET is be connected directly to UART RX on the jack receptacle, so UART RX signal changes will be misdetected as headphone plugin/plugout events.

To mitigate this, Pinephone Pro is using a simple first order low pass filter with cutoff frequency of 1.6kHz. With high enough baudrate and not too much bytes that consist of mostly 0 being transmited at once, this misdetection of plug-in should not happen all that often in UART mode.

2022–08–21: Sound on Pinephone Pro

I've gotten back to playing with audio codec on Pinephone Pro, after writing the initial device tree support for it last year. Last year I just did a few quick tests, without delving deep into the codec.

This time, I've analyzed it in detail, and mapped ALSA controls to codec audio paths in the process. I noticed several issues with the codec, while trying various configurations:

I've made patches to fix all of the issues I've found so far.

2022–08–14: USB recovery/hacking tool for Pinephone Pro

I noticed in Pine64 chat that some people lack uSD card or uSD card reader when they receive the phone, and are thus prevented from flashing an OS to eMMC until they buy it. It's kinda wasteful to buy these things when you may only need them for such a specific one-time use case.

I decided to write a tool that works somewhat like https://github.com/dreemurrs-embedded/Jumpdrive but does not require a uSD card. The tool itself and more infromation can be found here: https://xff.cz/kernels/pinephone-pro-recovery/

It's a simple to use binary executable with no options and no extra files to keep around that boots my Linux kernel build using Levinboot over USB and runs a program (from initramfs) that exports eMMC and SD block devices over USB.

Everything that is necessary to achieve this is included in the binary itself. The tool itself does not rely on any modifiable code that may be present on the phone, to function.

There's a space for a recovery tool that does not rely on any modifiable software located on the phone itself and thus works on all Pinehone Pros regardless of what is flashed on them.

Tow-Boot may have JumpDrive like functionality integrated, but that's kinda useless if you don't have Tow-Boot flashed, yet, or if you want to use a different bootloader, like Levinboot for example. This tool also relies on Linux, so the phone will have the Type-C port's functions properly managed during flashing, and in the future some UI may be easily added to perform various HW testing tasks, or to support direct installation of images from the internet over WiFi.

I also hope this will be a useful tool for distributing experimental kernel builds that can be quickly tested on anyone's Pinephone Pro without affecting their normal OS installation at all, to get a predicatble environment for distributed kernel testing and development. :)

2022–08–06: Pinephone keyboard bugs and new findings

There is some speculation out there about Pinephone keyboard charger chip (IP5209) getting damaged simply by plugging in something into Pinephone's USB-C port, because that's supposed to be like connecting two chargers into a single port, which somehow causes damage.

I decided to measure this, and see what really happens:

The charging situation with the keyboard looks like this:

There are a few things that can be seen from the picture:

So that's a bit annoying. Another few things can be seen about how charging the batteries works:

Many fun combinations are possible. The question is if some of them are problematic. For example what happens when you enable VOUT from keyboard and plug in USB PSU to the phone at once. Not much really. I've measured voltage transients and current from VOUT pin, and everything is withing reasonable parameters. The load presented by the phone is split between USB PSU and the keyboard's VOUT BOOST regulator, so part of the power is provided by the keyboard and part by the USB PSU. That is somewhat wasteful use of the keyboard battery, but not harmful. It's good to disable VOUT when you want to charge the phone battery using a phone's USB-C connector, to not needlessly discharge the keyboard battery.

What happens when VOUT is disabled? Nothing. IP5209 lives on, and there's no current flowing into VOUT from USB PSU connected to VBUS. All good.

Now the surprising thing is what happens when you plug USB PSUs to both the keyboard and the phone. One would think that the keyboard will continue supplying power to the phone via VOUT, like in the first scenario, except that the keyboard battery would not be getting discharged this time. Nope. What I measured was 450mA current flowing into VOUT, and not out of it, like before. That is NOT expected, and not wanted at all. At best we'd expect no current at all. IP5209 would nicely charge the battery from it's own VBUS input, and phone would charge it's own battery from its own VBUS input and no current would flow to VOUT. This is a situation that is problematic and needs to be avoided. To avoid it, it's enough to not connect anything to phone's USB port when a USB PSU is connected to the keyboard. All other combinations seem to be safe. In particular, it should be safe to have VOUT from the keyboard enabled, and to charge the phone via phone's USB-C port, or to use the dock with the phone's USB-C port.

So the only thing to worry about is to not connect anything to phone's USB port when keyboard's USB port is occupied. That is unsafe, and it behaves weirdly. Turning off VOUT when charging the phone directly is a good idea, but forgetting to turn it off will only lead to needless discharging of the keyboard's battery, and should not damage the keyboard. There's no backflow of the current into VOUT. The short periods of energy backflow into VOUT did not lead to IP5209 damage in my tests. But I don't know if that energy went into charging the keyboard battery somehow, or was dissipated as extra heat. That would be something to measure next. :)

2022–06–28: Pinephone Pro Type-C support is now complete

Final thing that was irritating me about Type-C support on Pinephone Pro and Pinebook Pro was that some things worked only in one orientation of the Type-C plug. There are two possible orientations (unless you're really forceful, then you may create a third one, and I'll not be adding support for that one).

The problem was that Rockchip Type-C PHY driver needs to powerdown and powerup the PHY while the DWC3 driver holds USB controller in reset, in order to be able to reconfigure the PHY for the detected Type-C plug orientation. This power cycling and reconfiguration was not being done.

Rockchip modified the DWC3 driver, and abused its runtime PM implementation to perform the reconfiguration. I didn't like this implementation's assumptions, and decided to figure out something better myself.

In the end I came up with these 3 patches to DWC3:

Which should make both orientations work regardless of data role modes, or runtime PM state. The patches are fairly small and unobtrusive, too. Complete explanations are in the patch descriptions.

And I'm happy to report that Type-C support now works on PPP and PBP without any obvious major issues, with Superspeed available in device and host modes regardless of connector orientation, with Alt-DP working in both orientations, too. Witch charger detection working for both PD and legacy BC1.2. PBP is non-conformant due to lacking input current limitation on USB-C port, but that's a HW issue not fixable in SW. It's just something to keep in mind when using the notebook.

There may still be some issues with particular PD docks (especially the more complicated ones without a captive cable), but it's somewhat hard to determine whether those are caused by bugs in fusb302 driver or the dock's firmware having some quirks or behaviors that interact badly with fusb302. fusb302/TCPM is a very complicated driver combination, and not easy to debug at all. But that will just be small obscure bugs.

As always, the more you pay for the dock, the more bugs you're buying. Pine's docks seem to work fine, just as well as the cheapest dock I managed to buy a few years ago locally for original Pinephone Type-C support development.

2022–06–23: Further Pinephone Pro camera development

I am continuing to work on tuning the cameras, which is quite a complicated process. I've bought some specialized tools to help me with the tuning, like a calibrated color card with 24 common colors, but regular tuning procedure would require specialized expensive and bulky tooling, like light boxes with standardized lights, opal based light difusor, and knowledge of color theory. I'm only interested in acquiring the last thing in that list.

Instead, I am trying to avoid specialized equipment where I can, using life hacks like unfocused focus lens position to perform some amount of difusion, using screen of another phone to get evenly distributed source of light of a particular „temperature“, figuring out ways to calibrate against other calibrated cameras instead of against standardized sources of light, etc. Throwing creativity at the problem instead of resources and money. Results may turn out to be subpar in general, but this is not really something properly solvable in a cheap home lab, I think, now that I have some sense of how the real calibration is supposed to be performed. OTOH, I may be able to perfectly calibrate the phone cameras to my particular real-life indoor ligting, lol. This is all a big experiment and it remains to be seen what's achievable without specialized tools.

Output of the tuning process is lens shading correction (LSC) and shielded pixel calibration (SPC) data for the sensor, and a lot of different data and parameters for the RK3399 ISP (image signal processor) for various scene types. Some data for the ISP is available for Android camera stack in a XML file present inside the Android factory image. I very much doubt this data is useful on Pinephone Pro as is, because when I use some parts of it, like LSC data for ISP (yes, LSC can also be done on ISP level), the resulting image still has very visible vignetting. Applying DPCC paramteres as present in the XML file does not remove dead pixels at all, and so on. It's almost certain that there was no camera calibration done for the factory image, so the data in the XML file are not terribly helpful, and the calibration process will need to be done again.

Normally, phone manufacturers include an EEPROM on the board and store calibration data for the sensor there. There's no such thing on Pinephone Pro, so the calibration needs to be done at both the sensor and ISP levels from scratch. It is not very helpful that for example lens shading effects differ based on a spectral characteristics of light present in the scene. So it may be necessary to upload different parameters for shooting at low light, for shooting under daylight, for shooting under tungsten lighting, etc. LSC is applied early in the process, so the rest of the calibration also needs to be re-done for each light source type.

Rockchip has a special application that allows tuning the camera ISP for dark current subtraction, lens shading correction, gamma correction, color correction, dead/stuck pixel removal, noise filtering, etc. This application is not available publicly, and not very useful anyway for any non-Android stack use.

I started writing a GTK4 based app that connects to the Pinephone Pro over WiFi and allows to modify parameters inside the sensor and ISP, while monitoring the effects of various correction in real time, inspect histograms for various color components, and in general to experiment with the cameras and the ISP fairly painlessly. This should help with the calibration process as much as possible. It will still be a very tedious manual process, though. To give an idea of the scale: the incomplete PDF guide for using the official calibration app from Rockchip for the Android stack has about 60 pages of steps to perform, with many steps also undocumented and hardcoded as algorithms in the calibration application itself, which is not available anywhere. Many of the steps need to be re-done for each standardized light source, individually. It sounds like a ton of work even if you know what you're doing, and have access to the Rockchip's calibration app.

Documentation for ISP used on RK3399 is somewhat lacking, so this process requires quite a bit of experimentation. Sensor level documentation is also quite lacking in some key areas, SPC doesn't work as advertised, in particular. This means that the grid of tens of thousands of artificially darkened pixels that's present accross the entire sensor is not possible to remove at the sensor level in real time. Newer Rockchip ISP variants allow to handle this at the ISP level, but not so the SoC inside the Pinephone Pro. I use several workaround, but it's a very annoying issue specific to PDAF variant of IMX258 in particular.

Aside from calibration efforts, I've extended my ppp-cam app to allow highly optimized live mjpeg video streaming over HTTP, I've started designing the UI for my app, thinking very much about the ergonomics of manual controls, etc. I'm a bit of an optimization nut, so the app is designed to be able to run on a single little CPU core clocked to almost minimum, and still be able to perform all of it features without any UI lag, by utilizing HW offload where possible.

It's possible to have live preview of both cameras at once on screen, encoding video to JPEG via Hantro VPU, and streaming it off over HTTP, with negligible CPU load. There's CPU offload processing HW on the SoC for pretty much every processing step. It's a lot of fun tying it all together. :)

Here's some video from my experiments https://megous.com/dl/tmp/ppp-wificam2.mp4 from a few weeks ago, streaming mjpeg video from the phone over WiFi (still without calibration).

And here's an early screenshot from the calibration app: https://megous.com/dl/tmp/a42fbb7180f34729.png if you're curious about that one. :)

My current plan is to complete enough features of the calibration app to be able to get past LSC stage, and to the actual color calibration via my 24 color card. I've added an overlay to the app to be able to line up the color boxes for readout by the app https://megous.com/dl/tmp/2ef7ab8564619185.png from known positions and some algorithm needs to be figured out to figure out color transform matrix parameters based on expected and captured color values. This will likely need some optimization algorithm, to find parameters that will optimize the overall error over all the color boxes to minimum.

Afterwards it will be possible to turn back to playing with AE/AWB/AF algorihms, and to finally start shooting pictures of reasonable quality out of the box.

This is still some months into the future. I'm thinking of doing another binary release of my ppp-cam app in the meantime that would allow others to experiment with the calibration process, too. The calibration app would be opensource and would connect to the ppp-cam app over the TCP socket interface, which is already implemented. ppp-cam app itself will stay closed source.

2022–05–29: Pinephone Pro camera pipeline testing app

I've been playing with the Pinephone Pro cameras a bit more, and found a few issues. Things mostly work though.

Here are some videos from my experiments:

The capture is happening at full resolution of the sensors at ~30fps. I had to bump up exposure time of the selfie camera to 3 frames, to get reasonable image in the dark, but it can also do preview at 30fps.

Some recent changes in my kernel

A bit of news about my camera app

Here is --help output:

Usage: ppp-photo [--sensor selfie|world] [--focus <value>] [--flash]
                 [--exposure <value>] [--again <value>] [--dgain <value>]
                 [--mode <mode>] [--test <pattern>] [--delay-shot <seconds>]
                 [--format bayer|ppm|jpg|webp|png] [--output <path>]
                 [--quality <quality>] [--gui]

Sensor options:
  -s, --sensor <type>    Which sensor to use: selfie or world
      --focus <value>    Focus (values 0.0 (far) - 1.0 (near))
      --flash            Use flash LED during a shot
      --exposure <value> Exposure time (1.0 = time it takes to scan a frame)
      --again <value>    Analogue gain (1.0 = no gain)
      --dgain <value>    Digital gain (1.0 = no gain)
      --mode <mode>      Select a different sensor output mode (see below)
      --test <pattern>   Output sensor test pattern (values 1 - 4)

  Pinephone Pro Selfie Camera:
    - mode 1: 4032 x 3024
    - mode 2: 2104 x 1560
    - mode 3: 1048 x 780

  Pinephone Pro World Camera:
    - mode 1: 3264 x 2448
    - mode 2: 1632 x 1224

UI options:
      --delay-shot <seconds>
                         Wait <seconds> before taking a shot
      --gui              Run in interactive GUI mode
      --setup-only       Setup the media pipeline and print paths to
                         various device files for the selected sensor
                         without doing any capture (useful for scripting).

Output options:
  -o, --output <path>    Specify path where you want to store the pictures
                        (When using --gui mode a ".####" number will be
                         appended to the path)
  -f, --format <format>  Specify format of picture files:
                         - bayer - raw data from the sensor (in bayer format)
                         - tiff - raw TIFF6 data from the sensor (in bayer format)
                         - ppm - debayered uncompressed PPM image
                         - jpg - JPEG (requires ImageMagick)
                         - webp - WebP (requires ImageMagick)
                         - png - PNG (requires ImageMagick)
                        (Append .zst to bayer/ppm formats for Zstandard
                         compression)
      --quality <value>  Specify jpg/webp compression quality (values 1 - 100)

Misc options:
  -v, --verbose          Show details of what's going on.
  -h, --help             This help.

Pinephone Pro photo shooting tool 0.1
Written by Ondrej Jirman <megi@xff.cz>, 2022

The app is available as a pre-built static binary in this repository The code will not be available at this time.

You can use it to test cameras on your phone, before a proper end-user app is developed based on libcamera or by extending megapixels app.

Power consumption

Sensors + ISP + image rotation consumes additional 850mW for the world sensor, and 600mW for the selfie sensor. CPU use is <1% during the tests, because my test app is written in such a way that no processing happens on the CPU (zero-copy buffer sharing between ISP<->RGA and RGA<->DRM).

Hardware issues I've found

While trying out the flash LED, I've noticed that flash mode is not really different from torch mode and that torch mode consumes excessive amounts of power (1.5W). LED is rated for 150 mA. It's clearly driven at much higher current (~3× the rated current in torch mode), which will degrade the LED and eventually kill it if used for longer than brief periods of time.

GPIO that should be switching between flash and torch modes seems to have no effect.

Also the driver chip used on PPP is not the same as on PP. It's somewhat compatible, except for supporting a feature where by toggling the enable pin enough times, it's possible to select flash intensity and duration. (except that flash mode doesn't work at all)

That may be because flash control signal is not connected or connected to a different GPIO.

Sensor driver for the selfie camera doesn't load calibration data from the OTP, because the data is probably not there. This is somewhat unfortunate.

Software issues I've found

I use Rockchip RGA to rotate the picture from the camera to match the orientation of the screen, and to change the pixel format from BGRX to RGB supproted by Rockchip's VOP/DRM driver.

I noticed that the image on the display had the RB components swapped, so I searched for the culprit, and it turned out to be the RGA driver. When converting between BGR and RGB, it will swap the colors correctly, but when converting between BGRX and RGB it will just remove the X byte without swapping the colors.

Hardware JPEG encoding

I've noticed that Rockchip's Hantro based HW video encoder can encode YUV422 to JPEG, and that mainline driver already supports this. So I decided it may be a good idea to use HW encoding for JPEG output because it may be faster and more energy efficient than using a CPU to do the same.

I've measured speed of conversion for 4032×3024 sized images, and it is 84 ms. For full HD resolution, it is 14–15ms. This makes it possible to encode FullHD mjpeg videos in real-time at 60 FPS and to significantly speed up encoding of full resolution photos and save them at the rate of ~10 photos per second continuously.

HW encoding is roughly 10× faster than using Imagemagick to encode PPM files to JPEG.

You can test HW encoding speed using gstreamer:

strace -f -e ioctl -tt \
gst-launch-1.0 videotestsrc num-buffers=2 ! \
  video/x-raw,width=1920,height=1080 ! v4l2jpegenc ! filesink location=test.jpg

Next steps

Now that I have basic live preview from the sensor, I can start playing with configuration of the ISP and dynamic updates of exposure/gains and other parameters based on statistics calculated by the ISP. Important parts to get right will be color calibration for both sensors, and lens shade correction for the selfie sensor. World sensor seems to have lens shade correction working.

Live preview is also very useful for debugging sensor controls and testing effects of various register values on sensor output. Seeing the effect of changes right away is extremely important for the ease of development and testing.

I intend to add some touch controls to the camera app's --gui mode, to manually control exposure/gain/focus/flash and various ISP options.

I'll try to integrate mp4/mjpeg recording into my app, once I figure out mp4 container format. :) And I'll definitely use the HW accelerated jpeg encoder for encoding captured images at full resolution, because the speedup is very significant.

It should be possilbe to record mjpeg videos at real-time with very little CPU use, and thus at reasonable power consumption. I've already tested JPEG HW encoding in my app, and this should be rather straightforward.

2022–05–22: Pinephone Pro camera improvements

Just a quick update that I managed to fix some bugs in IMX258 camera sensor driver to allow it to capture full resolution photos. I've also hooked up camera lens driver for controling focus on the world facing camera, so that images look better ;).

I've also implemented a command line image snapping tool for use with both cameras.

Currently world facing camera output looks like this (full):

There are still imporvements to be made, but output now looks much better than a few days ago. :)

2022–05–22: Pinephone Pro cameras kernel support

5.18 is to be released imminently, so to add a bit more motivation for people to update this release cycle, I've fixed up support for world facing camera on Pinephone Pro, and I also forward ported the BSP driver for the user facing camera and integrated it into my kernel tree, along with proper DT changes.

It involved debugging two issues. One stemming from a confusing Pinephone Pro schematic, and the other from me using Levinboot and having drivers built-in into my kernel builds that I use for development.

Pinephone Pro schematic names signals incorrectly in a bunch of places, which did hinder the development effort. From the schematic it appears that world facing camera is connected to ISP0, but it's in reality connected to ISP1. The schematic also names powerdown signal for user facing camera as active high, but it's in reality active low. So that camera was also failing when DT faithfully followed the schematic. While fun figuring that out, it took half a year for someone to notice, lol.

As for the Levinboot issue… Levinboot doesn't set I/O volatge levels for the I2C-1 bus to 1.8V, so they remained at default 3.0V which caused camera sensor probe failures on all my PPP prototypes ever since I first added the DT bindings for the cameras last year to my kernel tree. I thought my early prototype is different from the explorer batch, and left development to people who reported working IMX258 probe on their devices. U-Boot is not affected, which is why probe worked for other people.

I've patched my Levinboot branch to pre-configure the correct I/O voltage levels, and you can find camera support integrated here. It's also present in my pre-built kernels along with other changes, which you can read about in a release log.

Both cameras should now work as far as kernel is concerned.

Potential issues that remain to be fixed

Each sensor driver uses different master clock frequency while they share the same clock line.

User facing camera driver will set its desired frequency (24 MHz) when powered up, world facing driver will set its own (19.2 MHz) on powerup.

It's thus currently not possible to use both cameras at once. That would require for both drivers to use the same clock frequency. So don't enable both cameras at once. If you do, the sensor which was powered up first will have master clock frequency changed unexpectedly while it's operating. It will probably not work correctly afterwards.

Userspace

It should now be possible for interested people to start working on userspace support for the cameras in apps like megapixels, and libraries like libcamera.

Until then, you can use anteater's guide to play with the cameras from command line and discover new bugs and issues to be hunted down and solved. :)

There's also some useful documentation on Rockchip ISP in the kernel tree.

Have fun hacking! :-)

Sample photos

Selfie camera:

World camera:

2022–04–12: Pinephone keyboard keymaps

I've merged latest keyboard driver from Samuel to my 5.17 and 5.18 kernel branches. That driver has function keys F1-F10 mapped to Fn+1-Fn+0. There's no way to type alternate symbols printed on those keys, without loading a keymap into a kernel.

For use with Linux kernel virtual console, you can load the following keymap via loadkeys -d ppkb.map:

strings as usual
compose as usual for "iso-8859-1"
keymaps 0-63

plain keycode 125 = ShiftR
ShiftR keycode 125 = ShiftR
ShiftR keycode 0x02 = bar
ShiftR keycode 0x03 = backslash
ShiftR keycode 0x06 = asciitilde
ShiftR keycode 0x07 = grave
ShiftR keycode 0x08 = minus
ShiftR keycode 0x09 = equal
ShiftR keycode 0x0a = underscore
ShiftR keycode 0x0b = plus

Afterwards you'll be able to type Pine+1 to type |, Pine+2 to type \, and so on. Of course you are not limited to this simple keymap, you can read man keymaps, and implement any kind of key mapping you want. You can override arrow keys to make them active by default, you can make use of AltG or Pine key for more things, etc.

You can create similar keymaps for XKB, for use with Xorg server or wayland compositors.

You should not be modifying keymaps inside the kernel or device tree.

2022–04–02: Pinephone keyboard power manager

I noted previously on this blog, that Pinephone keyboard's charger doesn't have ideal behavior when combined with the phone (see Pinephone Keyboard – p-boot landscape mode for more details).

So that's something I've endeavored to fix the past few days. :)

Linux kernel

I decided for a kernel based solution for a few reasons. One is that I maintain the multi-distro demo image which shares one kernel between multiple Linux distributions, so any feature I implement in the kernel will be available to all distros at once, without having to integrate the same feature into 15 different distributions' varying init systems and userspace environments. The feature will also be available to all distros which use my kernel without much effort.

Other reason is that I have more control from the kernel over the system state, so I can reliably perform tasks prior to suspend to RAM, or after resume, or before system shutdown, etc. I can do this regardless of what mechanisms the Linux distribution uses for system sleep, whether it's systemd based sleep, or autosleep with wakelocks. The code is the same in all cases.

Implementing this feature in the kernel is also much simpler in general. I get direct access to states of power supply objects and other useful kernel features, like LED triggers.

How the power manager works

Power manager polls the status of various power supplies that represent the phone's and keyboard's battery charger, USB inputs and updates their status based on a simple control algorithm I've devised. This algorithm improves the charging and discharging behavior of Pinephone (and Pinephone Pro) when used together with the keyboard.

Basically, the algorithm tries to ensure that:

The driver also provides a set of LED triggers that you can associate with any of the phone's LEDs to get notified of the status of the batteries. For example, to get notified when the keyboard power is not being used via a red notification LED, you can simply:

echo kbpwr-kb-offline > /sys/class/leds/red:indicator/trigger

Or if you instead want to get notified when the capacity of both batteries is running dangerously low:

echo kbpwr-kb-capacity > /sys/class/leds/red:indicator/trigger

Power manager also handles several edge cases.

When you're turning off the phone, the driver will automatically shut the keyboard power down, so that the phone can be turned off. This would otherwise be a problem with Pinephone Pro, which would just restart if the keyboard was not powered down.

I'm still figuring out how I'd like the keyboard power supply to be handled on suspend/resume.

Other changes coming soon

In order for the power manager driver to have identical interface to chargers on Pinephone and Pinephone Pro, I had to extend the rk818 and axp20× drivers, to make them present the same power supply properties, and for those properties to have the same meaning on both phones.

One benefit of this is that I've unified the rk818-battery and rk818-charger drivers, so they are now presenting a single rk818-battery power supply interface to userspace. This will be less confusing to userspace. :)

I'll do some extensive testing, and push the changes out later next week.

I've made a tool to easily monitor the status of the phone while I test the algorithm in real usage.

The tool prints the current status reported by various power supplies in the phone, so I can see that the algorithm works as expected.

Now, I'll have to do the testing with both original Pinephone and Pinephone Pro… :)

2022–03–31: Keyboard light

Pinephone keyboard is hard to use in the dark. Easy usage in the dark is one of the benefits of on screen keyboards, apparently. With one usless Type-C port on the Pinephone that shouldn't be used for charging or USB peripherals, because of risk of damage, there's an opportunity to improve this.

I've bought a Type-C connector that is meant for home made Type-C power sources. It has resistors configured on CC pins, so that the device it's plugged into thinks the connector is part of a power supply. This is pretty good, because this connector will not risk damaging the phone or the keyboard charger when connected to the phone that is currently inserted into the keyboard.

I bought this connector to solder a LED/resistor to, so that I can have a compact LED light oriented towards the keyboard when necessary. The LED will be supplied by the keyboard battery.

What I needed was the above linked connector, a 220 Ohm resistor, a white LED unsoldered from a 12V LED strip (I measured it and it has a V-I characteristic knee at around 2.6V in forward direction, and gives enough light at about 7–10mA), UV epoxy, and a star or other more portable source of UV radiation. Some common tooling to put it all together is necessary, too.

I soldered the LED and the resistor to the connector. I tried to solder it in such a way so that LED will face the keyboard when the plug is inserted into the phone:

Other angle:

Then I added the epoxy, to make the connector more durable, and to unexpose the metalic contacts on the PCB. I pushed the epoxy with a syringe and a needle under the resistor and the LED, and all over the PCB:

A 5 minutes of exposure to UV radiation hardens the epoxy:

The benefits of this kind of epoxy are that it hardens quickly, and doesn't change shape/volume during hardening.

And the other side:

Exposure again:

And… done: :)

This is how it looks inserted into the phone:

The first test in dark conditions:

As expected, the LED light will need some shade, or a mirror to make it not shine right into my face. I've added a bit of a isolation tape for a test:

It's still a bit translucent, but good enough for making a quick video:

And the keyboard use in dark is much easier now. :)

I'll have to figure out a better shading, maybe something that will reflect the light towards the keyboard, instead of wastefully absorbing it.

Power consumption of this is ~60mW. This adds ~4% to original Pinephone power consumption and 2.5% to Pinephone Pro power consumption when in active use.

The LED is turned on only when the keyboard power is enabled with the keyboard power button, so it also serves as an indication when the keyboard power output is on.

2022–03–21: Some finer points about Alt-DP support in mainline kernel

Docks that support Alt-DP come in two flavors. With Alt-DP present either on a Type-C receptacle or on a plug (with a captive cable like the Pine dock). Pinephone just has a receptacle.

Each port supporting Alt-DP communicates its type via a VDO that is communicated using USB-PD protocol. VDO is a 32bit value where each bit has some meaning.

Pinephone Pro's VDO should be 0×c46:

Pinephone dock (and all my other docks with captive cable) reports 0×c05:

A dock with a Type-C receptacle with Alt-D support would report 0×c0045:

Now all this information is retrieved by the TCPM driver and passed to the usb/typec/altmodes/displayport.c driver for processing.

All these VDOs from Pinephone and from the connected dock should be compared against each other to figure out whether the current connection satisfies one of these valid combinations:

The code has to figure out whether both ports share one of the supported pin assignments and properly pass the selected assignments down the stack.

This is the current check in the code:

531         /* Make sure we have compatiple pin configurations */
532         if (!(DP_CAP_DFP_D_PIN_ASSIGN(port->vdo) &
533               DP_CAP_UFP_D_PIN_ASSIGN(alt->vdo)) &&
534             !(DP_CAP_UFP_D_PIN_ASSIGN(port->vdo) &
535               DP_CAP_DFP_D_PIN_ASSIGN(alt->vdo)))
536                 return -ENODEV;

#define DP_CAP_DFP_D_PIN_ASSIGN(_cap_)  (((_cap_) & GENMASK(15, 8)) >> 8)
#define DP_CAP_UFP_D_PIN_ASSIGN(_cap_)  (((_cap_) & GENMASK(23, 16)) >> 16)

It succeeds if there is at least one shared pin assignment option between bits 15:8 and 23:16 on either VDO. This is a valid check for receptacle<->receptacle combination.

For receptacle<->plug combination this check is not valid, because when bit DP_CAP_RECEPTACLE is 0, DP_CAP_UFP_D_PIN_ASSIGN and DP_CAP_DFP_D_PIN_ASSIGN swap their meanings.

So it looks like mainline code doesn't support Pinephone convergence dock at all. The only reason this check passes in my kernel is because I've set this VDO value in device tree (workaround to make the check pass, without fixing the mainline code):

altmodes {
        dp {
                svid = <0xff01>;
                vdo = <0x0c0046>;
        };
};

Instead of using proper one according to the spec. :)

Of course what happened is that someone bought a dock with a receptacle and reported to me that it doesn't work. It is detected properly, but Alt-DP mode is not entered because no valid combination of pin assignments is found.

Time for a proper fix.

2022–02–22: Adding LibreELEC.tv to an existing Pinephone multi-distro image

I wanted to try LibreELEC.tv on Pinephone, just for fun, and to see if it will make Pinephone into a nice portable TV. :)

LibreELEC.tv has support for Pine64 A64 SBC, so the system should work fine. The image can be downloaded here: https://libreelec.tv/downloads/allwinner/

None of the images are for Pinephone, though. If you download the image, and extract it, you'll find that it contains kernel build and the DTB for the target board only. :(

Anyway, that will not stop us from trying. Coincidentally, I wanted to write about how easy it is to modify existing multi-boot image, and this is a very good opportunity to demonstrate that.

Adding a new distro to existing multi-boot image

My Pinephone multi-boot image already includes a kernel that works on Pinephone. The only thing we'll have to achieve is to extract root filesystem from the LibreELEC.tv image for Pine64 SBC (see above), and copy the files from it to multi-boot image. Then we'll have to somehow add a boot entry for the new distro, and that will be it.

Getting LibreELEC.tv rootfs contents from the image

Let's start by extracting the rootfs from LibreELEC.tv image. Get the image first:

wget 'https://releases.libreelec.tv/LibreELEC-A64.arm-10.0.1-pine64.img.gz'

It's gzipped, so ungzip it:

gzip -d LibreELEC-A64.arm-10.0.1-pine64.img.gz

The image is partitioned as you can see using sfdisk for example:

sfdisk --dump LibreELEC-A64.arm-10.0.1-pine64.img

Prints:

label: dos
label-id: 0x179a0c97
device: LibreELEC-A64.arm-10.0.1-pine64.img
unit: sectors
sector-size: 512

LibreELEC-A64.arm-10.0.1-pine64.img1 : start=        8192, size=     1048576, type=c, bootable
LibreELEC-A64.arm-10.0.1-pine64.img2 : start=     1056768, size=       65536, type=83

Let's see what's on the first partition…

dd if=LibreELEC-A64.arm-10.0.1-pine64.img of=LibreELEC-A64.arm-10.0.1-pine64.img1 skip=8192 count=1048576
7z x -obootfs LibreELEC-A64.arm-10.0.1-pine64.img1
tree bootfs

Tree will print us:

bootfs
├── extlinux
│   └── extlinux.conf
├── KERNEL
├── KERNEL.md5
├── overlays
│   ├── sun50i-a64-ir.dtbo
│   ├── sun50i-a64-pine64-audio-board.dtbo
│   ├── sun50i-a64-pine64-wifi-bt.dtbo
│   └── sun50i-a64-spdif.dtbo
├── sun50i-a64-pine64.dtb
├── SYSTEM
└── SYSTEM.md5

2 directories, 10 files

Looks like some typical kernel/DTB files and booltoader configuration file,… aaand some suspect SYSTEM file. Check it out:

file bootfs/SYSTEM

And it looks like it's SquashFS image:

bootfs/SYSTEM: Squashfs filesystem, little endian, version 4.0, zstd compressed, \
  111496993 bytes, 9715 inodes, blocksize: 1048576 bytes, created: Fri Oct 29 20:07:52 2021

If we extract it, we will get what we were for: root filesystem contents. Let's leave that for later.

Add new boot entry and the distribution subvolume to the multi-boot image

Multi-boot image has two partitions. The first one contains the boot filesystem with the kernel and other files needed by the bootloader, and the second one contains BTRFS filesystem with one subvolume per distro. During boot, the same kernel is used for all boot options, and which subvolume is used is deterimned by the rootflags=subvol=name kernel boot argument. With BTRFS, you can either mount the toplevel subvolume, or any other subvolume by name if you specify a subvol=name mount option. And this is what the kernel argument above does.

That means that in order to add a new distro to multi-boot image, all we have to do is:

Let's step through that in detail. First we need to get access to the multi-distro image somehow. This can be done from the phone itself, from the PC via uSD card reader, or directly on the downloaded and image that was not flashed yet. In all these cases we will want to have the image represented as a partitioned block device. In all cases except the last one, this is already done by the kernel automatically. To modify the unflashed image, you'd need to use loop device to turn the downloaded and extracted image into a block device (for example via losetup --find --show --partscan image.img).

To find the multi-distro block devices, use sudo blkid | grep 12345678. For me this outputs (among other things):

/dev/mmcblk2p2: UUID="dfe75f46-40a7-4593-902c-1659f6c05f2d" UUID_SUB="6b4a8f9f-d8d3-4f5b-9d66-6d3dcfe9e6c0" BLOCK_SIZE="4096" TYPE="btrfs" PARTUUID="12345678-02"
/dev/mmcblk2p1: PARTUUID="12345678-01"

(In my case I'm modifying an existing multi-distro installation on an eMMC directly on Pinephone, so this found devices /dev/mmcblk2p1 with the boot filesystem and /dev/mmcblk2p2 with the BTRFS. Names of devices will vary depending on your situation.)

Now we will need to mount the BTRFS filesystem, and add LibreELEC.tv root filesystem to it.

mkdir m
sudo mount /dev/mmcblk2p2 m
sudo subvolume create m/electv

And now extract the root filesystem (SYSTEM file) to the new subvolume (remember, it's in SquashFS format):

sudo unsquashfs -f -dest m/electv bootfs/SYSTEM

That's it for the root filesystem. We can unmount it now, and move on:

sudo umount m

Modifying the p-boot's boot filesystem

Now we need to add a boot entry. p-boot uses a custom boot filesystem format that is not understood by the Linux kernel. So we'll need some special tools to modify it. The steps to modify the p-boot's bootfs are in general:

Let's get the necessary tools first. I keep the latest versions statically pre-compiled in p-boot's repository. We will download the two tools needed to do our job from there. You can also compile them yourself, if you're so inclined.

wget https://megous.com/git/p-boot/plain/dist/p-boot-conf
wget https://megous.com/git/p-boot/plain/dist/p-boot-unconf
chmod +x p-boot-{un,}conf

Now that we have all the tools, extract the boot filesystem (remember, /dev/mmcblk2p1 is the p-boot's bootfs partition as we determined earlier):

./p-boot-unconf pboot /dev/mmcblk2p1
tree pboot

Tree will show us the structure of files extracted from the boot filesystem:

pboot
|-- atf-0.img
|-- boot.conf
|-- dtb-0.img
|-- dtb2-0.img
|-- files
|   |-- arch-dreemurrs.argb
|   |-- arch.argb
|   |-- fedora.argb
|   |-- jumpdrive.argb
|   |-- lune.argb
|   |-- maemo.argb
|   |-- manjaro-phosh.argb
|   |-- manjaro-plasma.argb
|   |-- mobian.argb
|   |-- neon.argb
|   |-- off.argb
|   |-- pboot.argb
|   |-- pboot2.argb
|   |-- pmos-dplasma.argb
|   |-- pmos-fbkeyboard.argb
|   |-- pmos-gnome.argb
|   |-- pmos-mplasma.argb
|   |-- pmos-phosh.argb
|   |-- pureos.argb
|   |-- sailfish.argb
|   |-- sxmo.argb
|   |-- ut.argb
|   `-- xnux.argb
|-- initramfs-16.img
|-- linux-0.img
`-- splash
    |-- splash-0.img
    |-- splash-1.img
    |-- splash-10.img
    |-- splash-11.img
    |-- splash-12.img
    |-- splash-13.img
    |-- splash-14.img
    |-- splash-15.img
    |-- splash-16.img
    |-- splash-2.img
    |-- splash-3.img
    |-- splash-4.img
    |-- splash-5.img
    |-- splash-6.img
    |-- splash-7.img
    |-- splash-8.img
    `-- splash-9.img

2 directories, 46 files

We're only interested in boot.conf at this time. It contains the configuration for all the boot options:

device_id = Distro Demo Image 2020-11-23

no = 0
        name        = Arch Linux ARM 2020-11-21
        bootargs    = console=ttyS0,115200 earlycon=ns16550a,mmio32,0x01c28000 quiet loglevel=0 systemd.show_status=false cma=256M console=tty1 consoleblank=0 panic=3 rw rootwait root=PARTUUID=12345678-02 rootfstype=btrfs rootflags=compress-force=zstd,nodatacow,subvol=arch
        atf         = atf-0.img
        dtb         = dtb-0.img
        dtb2        = dtb2-0.img
        linux       = linux-0.img
        splash      = splash/splash-0.img

no = 1
        name        = Arch Linux ARM / dreemurrs 20201112
        bootargs    = console=ttyS0,115200 earlycon=ns16550a,mmio32,0x01c28000 quiet loglevel=0 systemd.show_status=false cma=256M console=tty1 consoleblank=0 panic=3 rw rootwait root=PARTUUID=12345678-02 rootfstype=btrfs rootflags=compress-force=zstd,nodatacow,subvol=arch-dreemurrs
        atf         = atf-0.img
        dtb         = dtb-0.img
        dtb2        = dtb2-0.img
        linux       = linux-0.img
        splash      = splash/splash-1.img

no = 2
        name        = Lune OS 2020-11-23
        bootargs    = console=ttyS0,115200 earlycon=ns16550a,mmio32,0x01c28000 quiet loglevel=0 systemd.show_status=false cma=256M console=tty1 consoleblank=0 panic=3 rw rootwait root=PARTUUID=12345678-02 rootfstype=btrfs rootflags=compress-force=zstd,nodatacow,subvol=lune bootmode=normal LUNEOS_NO_OUTPUT_REDIRECT
        atf         = atf-0.img
        dtb         = dtb-0.img
        dtb2        = dtb2-0.img
        linux       = linux-0.img
        splash      = splash/splash-2.img

no = 3
        name        = Maemo Leste 20201101
        bootargs    = console=ttyS0,115200 earlycon=ns16550a,mmio32,0x01c28000 quiet loglevel=0 systemd.show_status=false cma=256M console=tty1 consoleblank=0 panic=3 rw rootwait root=PARTUUID=12345678-02 rootfstype=btrfs rootflags=compress-force=zstd,nodatacow,subvol=maemo fbcon=rotate:1
        atf         = atf-0.img
        dtb         = dtb-0.img
        dtb2        = dtb2-0.img
        linux       = linux-0.img
        splash      = splash/splash-3.img

no = 4
        name        = Manjaro / Phosh beta2-20201119
        bootargs    = console=ttyS0,115200 earlycon=ns16550a,mmio32,0x01c28000 quiet loglevel=0 systemd.show_status=false cma=256M console=tty1 consoleblank=0 panic=3 rw rootwait root=PARTUUID=12345678-02 rootfstype=btrfs rootflags=compress-force=zstd,nodatacow,subvol=manjaro-phosh
        atf         = atf-0.img
        dtb         = dtb-0.img
        dtb2        = dtb2-0.img
        linux       = linux-0.img
        splash      = splash/splash-4.img

no = 5
        name        = Manjaro / Plasma 201122
        bootargs    = console=ttyS0,115200 earlycon=ns16550a,mmio32,0x01c28000 quiet loglevel=0 systemd.show_status=false cma=256M console=tty1 consoleblank=0 panic=3 rw rootwait root=PARTUUID=12345678-02 rootfstype=btrfs rootflags=compress-force=zstd,nodatacow,subvol=manjaro-plasma
        atf         = atf-0.img
        dtb         = dtb-0.img
        dtb2        = dtb2-0.img
        linux       = linux-0.img
        splash      = splash/splash-5.img

no = 6
        name        = Mobian 20201121
        bootargs    = console=ttyS0,115200 earlycon=ns16550a,mmio32,0x01c28000 quiet loglevel=0 systemd.show_status=false cma=256M console=tty1 consoleblank=0 panic=3 rw rootwait root=PARTUUID=12345678-02 rootfstype=btrfs rootflags=compress-force=zstd,nodatacow,subvol=mobian splash plymouth.ignore-serial-consoles vt.global_cursor_default=0
        atf         = atf-0.img
        dtb         = dtb-0.img
        dtb2        = dtb2-0.img
        linux       = linux-0.img
        splash      = splash/splash-6.img

no = 7
        name        = KDE Neon 20201123-084050
        bootargs    = console=ttyS0,115200 earlycon=ns16550a,mmio32,0x01c28000 quiet loglevel=0 systemd.show_status=false cma=256M console=tty1 consoleblank=0 panic=3 rw rootwait root=PARTUUID=12345678-02 rootfstype=btrfs rootflags=compress-force=zstd,nodatacow,subvol=neon splash
        atf         = atf-0.img
        dtb         = dtb-0.img
        dtb2        = dtb2-0.img
        linux       = linux-0.img
        splash      = splash/splash-7.img

no = 8
        name        = pmOS / Plasma Desktop 2020-11-21
        bootargs    = console=ttyS0,115200 earlycon=ns16550a,mmio32,0x01c28000 quiet loglevel=0 systemd.show_status=false cma=256M console=tty1 consoleblank=0 panic=3 rw rootwait root=PARTUUID=12345678-02 rootfstype=btrfs rootflags=compress-force=zstd,nodatacow,subvol=pmos-dplasma init=/sbin/init PMOS_NO_OUTPUT_REDIRECT
        atf         = atf-0.img
        dtb         = dtb-0.img
        dtb2        = dtb2-0.img
        linux       = linux-0.img
        splash      = splash/splash-8.img

no = 9
        name        = pmOS / fbkeyboard 2020-11-21
        bootargs    = console=ttyS0,115200 earlycon=ns16550a,mmio32,0x01c28000 quiet loglevel=0 systemd.show_status=false cma=256M console=tty1 consoleblank=0 panic=3 rw rootwait root=PARTUUID=12345678-02 rootfstype=btrfs rootflags=compress-force=zstd,nodatacow,subvol=pmos-fbkeyboard init=/sbin/init PMOS_NO_OUTPUT_REDIRECT
        atf         = atf-0.img
        dtb         = dtb-0.img
        dtb2        = dtb2-0.img
        linux       = linux-0.img
        splash      = splash/splash-9.img

no = 10
        name        = pmOS / GNOME 2020-11-21
        bootargs    = console=ttyS0,115200 earlycon=ns16550a,mmio32,0x01c28000 quiet loglevel=0 systemd.show_status=false cma=256M console=tty1 consoleblank=0 panic=3 rw rootwait root=PARTUUID=12345678-02 rootfstype=btrfs rootflags=compress-force=zstd,nodatacow,subvol=pmos-gnome init=/sbin/init PMOS_NO_OUTPUT_REDIRECT
        atf         = atf-0.img
        dtb         = dtb-0.img
        dtb2        = dtb2-0.img
        linux       = linux-0.img
        splash      = splash/splash-10.img

no = 11
        name        = pmOS / Plasma Mobile 2020-11-21
        bootargs    = console=ttyS0,115200 earlycon=ns16550a,mmio32,0x01c28000 quiet loglevel=0 systemd.show_status=false cma=256M console=tty1 consoleblank=0 panic=3 rw rootwait root=PARTUUID=12345678-02 rootfstype=btrfs rootflags=compress-force=zstd,nodatacow,subvol=pmos-mplasma init=/sbin/init PMOS_NO_OUTPUT_REDIRECT
        atf         = atf-0.img
        dtb         = dtb-0.img
        dtb2        = dtb2-0.img
        linux       = linux-0.img
        splash      = splash/splash-11.img

no = 12
        name        = pmOS / Phosh 2020-11-21
        bootargs    = console=ttyS0,115200 earlycon=ns16550a,mmio32,0x01c28000 quiet loglevel=0 systemd.show_status=false cma=256M console=tty1 consoleblank=0 panic=3 rw rootwait root=PARTUUID=12345678-02 rootfstype=btrfs rootflags=compress-force=zstd,nodatacow,subvol=pmos-phosh init=/sbin/init PMOS_NO_OUTPUT_REDIRECT
        atf         = atf-0.img
        dtb         = dtb-0.img
        dtb2        = dtb2-0.img
        linux       = linux-0.img
        splash      = splash/splash-12.img

no = 13
        name        = Sailfish 1.1-3.3.0.16-devel-20201101
        bootargs    = console=ttyS0,115200 earlycon=ns16550a,mmio32,0x01c28000 quiet loglevel=0 systemd.show_status=false cma=256M console=tty1 consoleblank=0 panic=3 rw rootwait root=PARTUUID=12345678-02 rootfstype=btrfs rootflags=compress-force=zstd,nodatacow,subvol=sailfish
        atf         = atf-0.img
        dtb         = dtb-0.img
        dtb2        = dtb2-0.img
        linux       = linux-0.img
        splash      = splash/splash-13.img

no = 14
        name        = pmOS / sxmo nightly-202011090018
        bootargs    = console=ttyS0,115200 earlycon=ns16550a,mmio32,0x01c28000 quiet loglevel=0 systemd.show_status=false cma=256M console=tty1 consoleblank=0 panic=3 rw rootwait root=PARTUUID=12345678-02 rootfstype=btrfs rootflags=compress-force=zstd,nodatacow,subvol=sxmo init=/sbin/init PMOS_NO_OUTPUT_REDIRECT
        atf         = atf-0.img
        dtb         = dtb-0.img
        dtb2        = dtb2-0.img
        linux       = linux-0.img
        splash      = splash/splash-14.img

no = 15
        name        = Ubuntu Touch 2020-11-19
        bootargs    = console=ttyS0,115200 earlycon=ns16550a,mmio32,0x01c28000 quiet loglevel=0 systemd.show_status=false cma=256M console=tty1 consoleblank=0 panic=3 rw rootwait root=PARTUUID=12345678-02 rootfstype=btrfs rootflags=compress-force=zstd,nodatacow,subvol=ut logo.nologo vt.global_cursor_default=0
        atf         = atf-0.img
        dtb         = dtb-0.img
        dtb2        = dtb2-0.img
        linux       = linux-0.img
        splash      = splash/splash-15.img

no = 16
        name        = Jumpdrive 0.6
        bootargs    = loglevel=0 silent console=tty0 vt.global_cursor_default=0
        atf         = atf-0.img
        dtb         = dtb-0.img
        dtb2        = dtb2-0.img
        linux       = linux-0.img
        initramfs   = initramfs-16.img
        splash      = splash/splash-16.img

That's quite a lot of boot options! You can modify this list to your liking. For me, I'll simply add a 17'th boot option for LibreELEC.tv:

cat <<EOF >> pboot/boot.conf
no = 17
        name        = LibreELEC.tv
        bootargs    = console=ttyS0,115200 earlycon=ns16550a,mmio32,0x01c28000 cma=256M console=tty1 consoleblank=0 panic=3 rw rootwait root=PARTUUID=12345678-02 rootfstype=btrfs rootflags=subvol=electv
        atf         = atf-0.img
        dtb         = dtb-0.img
        dtb2        = dtb2-0.img
        linux       = linux-0.img
EOF

I've removed all the kernel silencing options, because when playing with new distro, we usually want all the debugging aids we can get. The most important kernel argument is rootflags=subvol=electv that one determines what subvolume this p-boot menu option will use. Don't forget to change no option to an unique boot option index, either. In my case, this is 17.

To write the modifications to boot partition, we will run:

./p-boot-conf pboot /dev/mmcblk2p1

The tool will print a bynch of detailed information about what it did, and we can now reboot to test the new distribution. :)

Boot testing

Here's a first video of the boot test:

Pretty good! :)

And it looks like LibreELEC.tv also has a touch interface:

The only problem is that Pinephone LCD panel has native resolution 720×1440, which is not exactly great for watching videos, or using LibreELEC.tv's UI for that matter (as you can see from the videos). One would have to figure out some option for rotating the display.

Using LibreELEC.tv with external monitor

Pinephone Convergence Dock to the rescue. :) LibreELEC.tv doesn't support hotplugging the external monitor on Pinephone. It keeps displaying UI on the primary LCD. We need to nudge it a bit, to do what we want. The easiest way to do it is to disable the internal display using a kernel argument (video=DSI-1:d).

For that we can modify bootargs line in boot.conf and re-run p-boot-conf exactly as before. We don't need to run p-boot-unconf anymore from now on, as long as we don't delete the pboot directory.

The final line will be:

bootargs    = quiet loglevel=0 cma=256M console=tty1 consoleblank=0 panic=3 rw rootwait root=PARTUUID=12345678-02 rootfstype=btrfs rootflags=subvol=electv video=DSI-1:d

And let's try rebooting again, with dock connected to the phone and to the external monitor:

Pretty good! :)

Now all I need to do is connect a keyboard/mouse and play with it a bit. It looks quite snappy and the video playback should be accelerated on Pinephone!

With p-boot, you can have your Pinephone serve as a mobile OS and TV box at the same time. :) You can simply add LibreELEC.tv as one of the boot options, and reboot your phone to it wherever you need to have a TV box. :)

It looks like LibreELEC.tv is comming soon to next multi-distro demo image, after I do more testing! If you want to have it sooner than that. This article shall be your guide. ;-) You can add any distro you like, this way.

2022–02–22: Pinephone Pro – Debugging Type C port issues

Helping debug Type-C port related issues is not as simple as sending a dmesg output as on original Pinephone.

If you have issues with some USB functionality not working on Pinephone Pro, you should reboot your phone (so that the logs are clear and contain only information relevant to the issue), reproduce the issue, and immediately collect all the needed debugging information, by running this script:

#!/bin/bash
uname -a
cat /etc/os-release
dmesg
echo -e "\nfusb302:\n"
cat /sys/kernel/debug/usb/fusb302-4-0022/{regs,log}
echo -e "\ntcpm:\n"
cat /sys/kernel/debug/usb/tcpm-4-0022/log
echo -e "\npower:\n"
cat /sys/class/power_supply/*/{device/uevent,uevent}
echo -e "\nextcon:\n"
cat /sys/class/extcon/extcon*/{name,state}
echo -e "\nalt-dp:\n"
cat /sys/kernel/debug/dri/*/state
echo -e "\nregulators:\n"
cat /sys/kernel/debug/regulator/regulator_summary
echo -e "\nkconfig:\n"
zcat /proc/config{,.gz}
echo -e "\nkmodules:\n"
lsmod

The script is also downloadable from here

Run the script as:

bash ppp-typec-dbg.sh > typec.log

Then you can inspect the typec.log yourself and/or send it to me (x@xnux.eu) along with some description of the issue, the USB devices you're using and the expected behavior.

2022–02–01: Pinephone Pro – Type C port support

Since November last year, I've been working on and off on understanding and implementing support for Type-C port on Pinephone Pro. It's a complex physical interface, so it was hard to understand how everything fits together on the Rockchip platform without going through a lot of drivers and without understanding the USB Type-C specification. I already had some grasp of the Type-C spec from implementing support for the similar functionality on original Pinephone.

So my approach with Pinephone Pro was to find all existing drivers that are related to supporting all the features of the Type-C port on RK3399 and read them top to bottom.

This gave me a nice overview of how everything can fit together eventually, and also what's missing in mainline Linux compared to Rockchip BSP Linux.

I was also interested in making Type-C support (which includes power negotiation, charging and supplying power, USB data role switching, Display Port alternate mode, power delivery and BC1.2 spec support, and similar things) work as best as possible as soon as possible, so that when Pinephone Pro got to a wider audience, all this worked as well as was possible to achieve with my limited resources in a few weaks before I was going to winter holidays.

Once I understood the drivers and their supposed interactions, I needed to make them talk together with as little code changes as possible. This meant using their existing interfaces.

Mainline Linux is moving in the direction of using in-kernel Type-C interfaces for controlling the drivers that need to listen to the Type-C port manager driver. Most drivers on the Rockchip platform don't support these interfaces, and even if they did, these interfaces currently don't support controlling multiple devices at once, which is necessary on this particular platform. Instead, Rockchip drivers mostly use extcon interface (bus, really) to communicate among each other.

So as a quick stop-gap solution I decided to write the typec-extcon driver, which adapts Type-C mux/switch/role interfaces into extcon ones the existing drivers understand. This made it possible to quickly wire things things together, and be able to start testing the hardware, and finding bugs.

It works semi-reasonably now. Interdriver communication is now solid, with my latest kernel releases for Pinephone Pro.

That doesn't mean there are no issues. The drivers themselves, that are already mainline, have several issues:

USB driver will probably be reasonably easy to fix. BSP driver does things correctly, so it can serve as an inspiration.

fusb302 will be tough to fix. The chip is not horribly complicated, but TCPM driver fusb302 serves as a backend to is, and fusb302's problem is more in the fact that it's quite low level, so you're setting comparators, dealing with D/A converters, voltages, pull-up resistor values, etc. But the higher level deals with abstract states on the CC pins. So far, issues with the driver revolve around misdetection of states of CC pins, and infinite loops, or unhandled states due to that.

Once the individual drivers are made to behave properly, it will be time to think about mainlining this work. I have some plan that involves waiting for multi consumer support for in-kernel Type-C interfaces, that is being proposed on the USB mailing list to be reviewed and accepted. This patchset will help solve the driver communication issue, in an upstreamable way, by allowing TCPM to talk to multiple drivers at once. The drivers will also need to be adapted to use the proper interfaces. This will be fairly easy at that point.

Once all that is done, it will be possible to upstream the whole work.

In the meantime there are drivers to fix.

Fixing up fusb302 is the highest priority. Identifying and solving issues in the wild, from people using the interim solution will be another one.

When working on anx7688 driver I was able to stuff it with a lot of useful debug output, that was then all printed to the kernel log. This allowed me to just ask people to send me their dmesg output, when I noticed someone had issues with things related to Type-C port, and I would not need much else from them. I gradually improved the debug output as I tracked the known issues to be able to understand them better as more people reported issues and sent the logs. This was nice, and helped stabilize the Type-C support on original Pinephone.

Compared to that, Pinephone Pro seems quite hopeless. There's almost 0 useful output in kernel log itself, useful logs are hidden deep in debugfs and sysfs in several places, etc. It's very time consuming to do debugging with users, because of that. Reading the logs is hader too, becuase they are very verbose and low-level, and are split, so there's no single timeline to skim through. They need to be cross-references manually.

I decided to not add to this difficulty, and only help users who will be able to install and run my exact kernel build for debugging. It's just too much effort to also track what subset of my patches some unknown version of the downstream Linux distribution's kernel, with no existing git tree to diff against, decided to include.

It will take maybe 4 kernel releases to clean up for the upstreaming attempt.

I'll step away from this work for a while to prepare basic device tree for Pinephone Pro and submit it mainline. Otherwise things will start getting messy pretty soon with multiple kernel trees being actively used by distributions and no shared base device tree upstream.

2022–01–31: Pinephone Pro battery charging

If you have trouble chargigng battery on Pinephone Pro, this article may help you underastand what may be wrong. First, here are a few quick facts you should know about the Pinephone Pro Type-C port and the power management chip (RK818), to better understand how it all works together:

This is basically it. Most charging issues revolve around setting this one limit. How could that possible be, that there are seemingly so many issues then? :)

How does input power limitting work

So as you can see, if the phone decides it will take no power from the USB power supply, there will simple be not enough power to charge the phone's battery, or even to boot and run the phone. All the charging issues revolve around this one variable: power input limit.

How then does the phone decide what power input limit to set?

Phone startup process

You most probably have some software on the phone, right? :) Therfore:

Let's assume all went well up to this point. Next:

Input power limit negotiation

So how does this detection work?

This is where it gets complicated. Dumb charger detection and Type-C port management are two independent processes with independent hardware signals. Nevertheless, the output of the detection process has to be just one number (power input limit). So these processes have to meet somewhere to decide which one is right and which one will choose the final value for the limit.

So if you:

And this is where most of the software issues were or are.

Type-C software support issues

Let's list some of those issues that I know about at this time:

2022–01–22: Pinephone Keybaord – kernel driver merged

I have updated Samuel's WIP driver that the distros started using to make it work with his IP5209 charger driver on the final production keyboard. The result is now merged into my kernel tree and works on both Pinephone and Pinephone Pro.

This also means, that keyboard battery voltage and current can now be read from sysfs. You don't need to use my userspace tools for that anymore.

I've also updated the drivers to make them report probe errors, and improved the charger driver to make it not fail when the charger is asleep during boot. You still have to wake the charger up by pressing the button on the side of the keyboard before being able to access it via sysfs.

You can run:

dmesg | grep kb151

to see if the keyboard driver is functioning properly. If you see any errors there, you may have a bad physical connection between the phone and the keyboard. You can try improving the connection, and rebooting the phone.

You may see for example:

[    0.388320] kb151 5-0015: Found KB151 with firmware 1.0 (features=0xf)
[    0.916089] kb151 5-0015: Charger is initialized

or

[    0.828521] kb151 2-0015: KB151 was not detected on the bus (-6)

If you want to run ppkb-i2c-inputd, just add kb151.disable_input option to kernel boot arguments to disable the kernel driver. Kernel driver handles Fn layer differently from the userspace daemon. With userspace daemon, pressing Fn+~ or Fn+- will really write out those characters to the console. Userspace driver also has Pine Key layer, which can be used to invoke function keys. Kernel driver doesn't have ability to invoke F1 - F10 keys in any way directly.

Enjoy!

2021–12–12: Pinephone Keyboard – p-boot landscape mode

I've been tuning my pinephone for use with the keyboard. First thing that needed to be done was to switch and optimize everything to work well in landscape mode.

I've started with p-boot, which can now render its virtual console in rotated by 90 degrees. Everything else is the same. Background bitmaps just need to be laid out and re-rendered so that they look nice when looked at in landscape mode, but they are loaded the same as before.

I'll try adding keyboard support to p-boot in the future, too. It might be useful in case you'll want to change kernel parameters, or whatever.

The rest of the system also needs some optimizations. Linux virtual console can be rotated from get go by using kernel parameter fbcon=rotate:1. Default console font is also way too small for use with the keyboard. Due to the layout of human body :), you'll be looking at the display from further away than when using a touch interface, so text needs to be larger. Use of some larger font than default 8×16 one is warranted. I enabled SUN12×22 in my kernel, and my final boot argument for fbcon looks is now fbcon=rotate:1,font:SUN12x22.

Xorg X server to work in landscape mode needs some tweeks too. First display outout to DSI-1 needs to be rotated via xrandr --output DSI-1 --rotate right and separately touchscreen input needs to be transformed using some transformation matrix so that it matches the display output rotation via xinput set-prop 'pointer:Goodix Capacitive TouchScreen' 'Coordinate Transformation Matrix' 0 1 0 -1 0 1 0 0 1. I've put these commands to my .xsession file, so that they are applied at the start of each X session.

If you'd like to use Firefox with touch input, it's a good idea to add export MOZ_USE_XINPUT2=1 to your X session too. Firefox will then be able to recognize Goodix Capacitive TouchScreen as a touch input device, and act accordingly. Tuning Firefox is another chapter itself. By default it uses some physics based scrolling, which works really badly on Pinephone, so this needs to be disabled. Force enabling GPU acceleration of layer composition, is also not a terrible idea.

So far my system setup looks like this: https://megous.com/dl/tmp/ppkb-test.mp4

More chargers, more issues

Having two chargers and batteries chained together in a phone creates some new issues.

First is that the phone is powered from the keyboard charger internally and not over USB connector, so normal, USB based charger type detection and current limit setting does not work.

The second issue is that both chargers act as power supply and battery charger at once, and they both have some automatic behavior that may lead to suboptimal results when left on their own.

On other dual battery devices, strategy for charging and taking power from multiple batteries is likely handled by some embedded controller. On Pinephone, this is not handled by anything. Some software needs to be written to monitor the chargers and correct their settings based on their current status.

For example let's say both batteries are discharged. What should happen when you connect the keyboard to the USB power supply? What I'd like to happen is this:

  1. Phone's internal battery is charged as fast as possible
  2. When the internal battery is fully charged, keyboard battery starts charging, while still supplying enough power to the phone so that internal battery doesn't start discharging, until both batteries are fully charged.

What happens instead:

  1. Phone doesn't detect anything on USB port so the input power limit is left at default 2.5W, this leads to very slow charging
  2. Keyboard charger starts charging the keyboard battery at max speed, because it can take 10W from USB power supply, and the phone is only taking 2.5W
  3. After keyboard battery is fully charged, phone battery will still keep charging only very slowly

When you disconnect the USB cable, the keyboard charger will now supply power for both charging the phone battery, and for phone's other power needs. This is not very efficient. It would be best if the internal battery had absolute priority if it's not fully charged, because powering the phone from internal battery is more efficient. It's also not very necessary to move charge from keyboard battery to phone battery all the time. Due to pinephone design there needs to be some charge in internal battery for the modem and wifi to work, but it does not need to be kept at 100%. Shifting charge between batteries should only be necessary when the internal battery charge drops bellow 30% so that it's kept between 30% and 35% for example.

It would be more efficient to disable the phone's battery charging when the charge is high enough, but still use the keyboard battery to supply power to the phone, until keyboard battery charge gets very low. Then the system should switch to using the phone battery.

All this needs to be managed by some userspace daemon, yet to be written, because it will not happen automatically and automatic behavior will be annoying and sub-optimal.

There will also be some need for user interaction, because the keyboard battery charger is controlled by the button on the side of the keyboard. This button enables the keyboard power output to the phone. Without user pressing it it's not possible to power the phone from the keyboard. So the power management daemon will have to have some way to notify the user about the status of the keyboard charger and the need to press the button in some situations, to wake the keyboard power management circuitry.

2021–11–20: Pinephone Pro – audio and modem power up support

I've added Pinephone Pro audio codec support to my kernel and did a quick test that it works. Everything except Bluetooth should work. Modem audio too. Compared to Pinebook Pro's 37 controls, the codec on Pinephone Pro has more than 3 times as many.

One other thing I enabled yesterday is support for powering up the modem using my modem power driver. Powering up the modem on my kernel is now as simple as on the original Pinephone:

echo 1 > /sys/class/modem-power/modem-power/device/powered

And the result will show up in dmesg:

[   87.347227] modem-power serial1-0: powering up
[   87.608383] modem-power serial1-0: wakeup ok
[   90.260373] modem-power serial1-0: status ok
[   90.260584] dw-apb-uart ff1b0000.serial: failed to request DMA
[   99.666364] usb 1-1: new high-speed USB device number 2 using ehci-platform
[   99.804639] usb 1-1: New USB device found, idVendor=2c7c, idProduct=0125, bcdDevice= 3.18
[   99.804662] usb 1-1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[   99.804668] usb 1-1: Product: EG25-G
[   99.804673] usb 1-1: Manufacturer: Quectel
[   99.805882] option 1-1:1.0: GSM modem (1-port) converter detected
[   99.806238] usb 1-1: GSM modem (1-port) converter now attached to ttyUSB0
[   99.806602] option 1-1:1.1: GSM modem (1-port) converter detected
[   99.807004] usb 1-1: GSM modem (1-port) converter now attached to ttyUSB1
[   99.807496] option 1-1:1.2: GSM modem (1-port) converter detected
[   99.807836] usb 1-1: GSM modem (1-port) converter now attached to ttyUSB2
[   99.808301] option 1-1:1.3: GSM modem (1-port) converter detected
[   99.808686] usb 1-1: GSM modem (1-port) converter now attached to ttyUSB3
[  103.302076] modem-power serial1-0: ===================================================
[  103.302099] modem-power serial1-0: Project Name: EG25G
[  103.302104] modem-power serial1-0: Project Rev : EG25GGBR07A08M2G_01.002.07
[  103.302108] modem-power serial1-0: Branch  Name: 9x07_R07_NEW
[  103.302112] modem-power serial1-0: Custom Name : STD
[  103.302116] modem-power serial1-0: Build   Date: 2020-06-13,09:14
[  103.302121] modem-power serial1-0: Version:  SBLR07EG25G01_1:37500: May 30 2
[  103.302125] modem-power serial1-0: Version:  TZR07A03: Dec  2 2019
[  103.302129] modem-power serial1-0: Version:  RPMR07A03_1:37500: Dec  2 2019
[  103.302133] modem-power serial1-0: ===================================================
[  103.326183] modem-power serial1-0: ADB KEY is '24774888' (you can use it to unlock ADB access to the modem, see https://xnux.eu/devices/feature/modem-pp.html)
[  104.358333] modem-power serial1-0: QDAI is '1,1,0,1,0,0,1,1'
[  104.365510] modem-power serial1-0: QCFG 'risignaltype' is '"physical"'
[  104.374031] modem-power serial1-0: QCFG 'urc/ri/ring' is '"pulse",1,1000,5000,"off",1'
[  104.382449] modem-power serial1-0: QCFG 'urc/ri/smsincoming' is '"pulse",1,1'
[  104.389571] modem-power serial1-0: QCFG 'urc/ri/other' is '"off",1,1'
[  104.396175] modem-power serial1-0: QCFG 'urc/ri/pin' is 'uart_ri'
[  104.401964] modem-power serial1-0: QCFG 'urc/delay' is '0'
[  104.408745] modem-power serial1-0: QCFG 'sleepind/level' is '0'
[  104.415852] modem-power serial1-0: QCFG 'wakeupin/level' is '0,0' (changing to '0')
[  104.448347] modem-power serial1-0: QCFG 'ApRstLevel' is '0'
[  104.456210] modem-power serial1-0: QCFG 'ModemRstLevel' is '0'
[  104.462069] modem-power serial1-0: QCFG 'apready' is '0,0,500'
[  104.469190] modem-power serial1-0: QCFG 'airplanecontrol' is '1,1'
[  104.478157] modem-power serial1-0: QCFG 'fast/poweroff' is '1'
[  104.486540] modem-power serial1-0: powered up in 17139 ms

If you want to make sense of the 124 controls that the codec exposes, you'll have to read the datasheet.

Most of the controls are muxes and switches. There are only about 5 volume controls exposed by ALSA.

Main codec is connected via analog path to the modem's external ALC5616 codec. If you want to be able to make calls with the modem, you'll need to configure this codec properly via AT+QDAI command. I have not done that yet, and my kernel still uses the QDAI value that works only on the original Pinephone.

There should be no „robotic voice“ issue while calling with Pinephone Pro, because there's no need to do a sample rate conversion between the modem and the SoC.

2021–11–19: Pinephone Pro LCD panel

According to /sys/class/power_supply/rk818-usb/input_current_limit the panel uses the following mode:

mode: "720x1440": 53 66000 720 760 800 840 1440 1458 1468 1485 0x48 0xa

This is problematic, becase 53 Hz is not exactly the expected refresh rate. Kernel calculates this refresh rate from other timings on the line. It is not possible to just raise the pixel clock from 66 Mhz to 74.7 MHz to compensate. If we do, the panel will have corrupted output.

The following is a initialization sequence for the panel's HX8394-F controller. It's copied here from the driver written by ayufan, and was probably copied from Android kernel, originally:

dsi_dcs_write_seq(dsi, 0xb9, 0xff, 0x83, 0x94);
dsi_dcs_write_seq(dsi, 0xb1, 0x48, 0x11, 0x71, 0x09, 0x32, 0x24, 0x71, 0x31, 0x55, 0x30);
dsi_dcs_write_seq(dsi, 0xba, 0x63, 0x03, 0x68, 0x6b, 0xb2, 0xc0);
dsi_dcs_write_seq(dsi, 0xb2, 0x00, 0x80, 0x78, 0x0c, 0x07);
dsi_dcs_write_seq(dsi, 0xb4, 0x12, 0x63, 0x12, 0x63, 0x12, 0x63, 0x01, 0x0c, 0x7c, 0x55, 0x00, 0x3f, 0x12, 0x6b, 0x12, 0x6b, 0x12, 0x6b, 0x01, 0x0c, 0x7c);
dsi_dcs_write_seq(dsi, 0xd3, 0x00, 0x00, 0x00, 0x00, 0x3c, 0x1c, 0x00, 0x00, 0x32, 0x10, 0x09, 0x00, 0x09, 0x32, 0x15, 0xad, 0x05, 0xad, 0x32, 0x00, 0x00, 0x00, 0x00, 0x37, 0x03, 0x0b, 0x0b, 0x37, 0x00, 0x00, 0x00, 0x0c, 0x40);
dsi_dcs_write_seq(dsi, 0xd5, 0x19, 0x19, 0x18, 0x18, 0x1b, 0x1b, 0x1a, 0x1a, 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x20, 0x21, 0x18, 0x18, 0x18, 0x18, 0x18, 0x18, 0x18, 0x18, 0x18, 0x18, 0x18, 0x18, 0x24, 0x25, 0x18, 0x18, 0x18, 0x18, 0x18, 0x18, 0x18, 0x18, 0x18, 0x18, 0x18, 0x18);
dsi_dcs_write_seq(dsi, 0xd6, 0x18, 0x18, 0x19, 0x19, 0x1b, 0x1b, 0x1a, 0x1a, 0x07, 0x06, 0x05, 0x04, 0x03, 0x02, 0x01, 0x00, 0x25, 0x24, 0x18, 0x18, 0x18, 0x18, 0x18, 0x18, 0x18, 0x18, 0x18, 0x18, 0x18, 0x18, 0x21, 0x20, 0x18, 0x18, 0x18, 0x18, 0x18, 0x18, 0x18, 0x18, 0x18, 0x18, 0x18, 0x18);
dsi_dcs_write_seq(dsi, 0xe0, 0x00, 0x04, 0x0c, 0x12, 0x14, 0x18, 0x1a, 0x18, 0x31, 0x3f, 0x4d, 0x4c, 0x54, 0x65, 0x6b, 0x70, 0x7f, 0x82, 0x7e, 0x8a, 0x99, 0x4a, 0x48, 0x49, 0x4b, 0x4a, 0x4c, 0x4b, 0x7f, 0x00, 0x04, 0x0c, 0x11, 0x13, 0x17, 0x1a, 0x18, 0x31, 0x3f, 0x4d, 0x4c, 0x54, 0x65, 0x6b, 0x70, 0x7f, 0x82, 0x7e, 0x8a, 0x99, 0x4a, 0x48, 0x49, 0x4b, 0x4a, 0x4c, 0x4b, 0x7f);
dsi_dcs_write_seq(dsi, 0xcc, 0x0b);
dsi_dcs_write_seq(dsi, 0xc0, 0x1f, 0x31);
dsi_dcs_write_seq(dsi, 0xb6, 0x7d, 0x7d);
dsi_dcs_write_seq(dsi, 0xd4, 0x02);
dsi_dcs_write_seq(dsi, 0xbd, 0x01);
dsi_dcs_write_seq(dsi, 0xb1, 0x00);
dsi_dcs_write_seq(dsi, 0xbd, 0x00);
dsi_dcs_write_seq(dsi, 0xc6, 0xed);

The panel's controller may or may not really be HX8394-F, because the panel's datasheet does list a number of controller options, and it's not clear which one is really used. Current kernel driver does not check the controller ID (if that's even possible).

The driver defines the mode as:

static const struct drm_display_mode hsd060bhw4_mode = {
        .hdisplay    = 720,
        .hsync_start = 720 + 40,
        .hsync_end   = 720 + 40 + 40,
        .htotal      = 720 + 40 + 40 + 40,
        .vdisplay    = 1440,
        .vsync_start = 1440 + 18,
        .vsync_end   = 1440 + 18 + 10,
        .vtotal      = 1440 + 18 + 10 + 17,
        .clock       = 66000, /*XXX: only 66MHz works */
        .flags       = DRM_MODE_FLAG_NHSYNC | DRM_MODE_FLAG_NVSYNC,
        .width_mm    = 68,
        .height_mm   = 136,
};

This should match the parameters sent via the above initialization sequence. It's not really all that clear that it does match.

One of the more important initialization commands are:

They are important, because they need to correspond to the mode timings used by the kernel. They don't seem to. Vertical back/front porch values are off, other things may be off too. For example, some initialization sequence commands write beyond the end of the defined command parameter list, so it's not even clear that HX8394-F is really used, but not one of the other controllers specified as options in the LCD panel datasheet.

Hm? :(

It doesn't help that I can't find „HX8394-F application note“ that specifies how to correctly configure GIP source/gate driving circuitry to match the panel datasheet and our desired timings. The datasheet itslef glosses over all the details, just refering to unavailable application note for the details.

It may be useful to compare datasheets for other controllers listed in the panel datasheet, to check if one of them does match the initialization commands and their parameter counts better, or whether they explain relationships between GIP settings and hsync/vsync timing more properly.

I asked for more information about the panel in the Pine64 devzone. Until it's available, it will not be possible to achieve 60 Hz refresh rate. (without a lot of experimentation and research)

For easier experimentation not requiring a kernel re-compile for each display init sequence change, I'll extend the driver to allow changing and testing things over some custom debugfs interface.

Until this is solved, Pinephone Pro will have the same 60Hz video playback issues (unpleasant stutter due to mismatch between display refresh rate and the video frame rate) the original Pinephone had, until I fixed the refresh rate issue.

2021–11–16: A bunch of mostly Pinephone Pro news

fusb302 bug from the previous post

My fix for fusb302 disconnect bug from the last post was reviewed by upstream maintainers and will be accepted for mainline kernel and likely also merged into stable kernels.

Phone LCD getting all garbled after disconnecting the external monitor

I've successfully chased down one bug that caused internal display to be clocked wrong after plugging in and removing the USB dock. It turns out RK3399 can generate rather precise pixel clock for the displays, if and only if the clock tree is configured to use a fractional divider (numerator/denominator can both be any 16-bit value, which is perfect!). It was not configured to use it in Pinephone Pro DT, and thus just a very rough clock rate was set each time the display pipeline was re-configured, and it turned out the display is very senstitive to the pixel clock rate. To makes things worse the clock rate was not even set consistently, but changed based on situation. 2.4Mhz difference here or there is not something that LCD controller tolerates well. With the fix, it's possible to generate the required pixel clocks precisely almost to a single Hertz and the problems go away.

Panfrost/Lima panicking the 5.16-rc1 kernels

I've also hit an issue in a brand new 5.16-rc1 kernel, that led to kernel panics whenever panfrost was being used. I reported it and it will be fixed upstream in -rc2.

HW issues and needless de-railements

I've had problems with USB 2.0 in device mode on Pinepehone Pro. Connecting the phone to USB 2.0 port would cause USB enumeration failures, but BC1.2 detection was working. I assumed it must be some software issue, because if BC1.2 worked (it's using the same wires in the cable that are used for USB 2.0 data link).

It turned out the red Pine64 USB-A → Type-C cable was somehow half-broken. Using a different red Pine64 cable allowed the phone to enumerate correctly. Go figure.

I also have issues with the Type-C port being a bit wobbly on my Pinephone Pro unit, so I padded it a bit with some spare thermal padding that I have at home, to fix it in place. Sometimes I can still see USB re-enumeration in kernel log when I shift things around on my table and touch the cable going to the phone. It makes it hard to distinguish software and hardware issues, slowing down development with needless debugging that leads nowhere and confuses me, until I figure out it's just a bad cable or a bad connector connection, or whatever. :(

Development now moved on to 5.16-rc1

I've rebased my kernel branches on top of 5.16-rc1 and with all these issues out of the way, it now works quite nicely for development and testing of the PPP support in the kernel. I've thus moved all my PPP work to 5.16-rc1.

New charger driver for RK818

I've implemented and tested the charger driver for RK818 PMIC. It can now correctly set input power limits, based on what limit is reported by Type-C controller based on standard PD and BC1.2 negotiation protocols. I've verified this with the Power-Z USB Type-C power meter.

I've also discovered a more standard way to pass input current limits based on a link between two power supplies in the device tree. This helped me clean up the code a bit, and make it eventually a bit more upstreamable.

Cleaner implemnetation of USB OTG, Type-C and USB Power Delivery on Pinephone Pro

I've found yet easier and more proper ways to tie everything together in both drivers and the device tree for all Type-C support to work.

For example previously I used graph lookup in DT to connect fusb302/tcpm to my new typec-extcon bridge driver, which requires very wordy port/endpoint setup on both nodes in the DT. See my cameras support patch for Pinephone Pro for an example of this port/endpoint based device node relationships. Looking at the kernel code, I figured that instead of complicated port/endpoint dance, typec interfaces also allow to just use a simple references like usb-role-switch = <&typec_extcon>;, mode-switch = <&typec_extcon>;, orientation-switch = <&typec_extcon>; to refer to the providers of various typec interfaces. Much simpler! It would be nice if this was documented somewhere. :)

Because typec-extcon device driver is just a software artifact to adapt two incompatible in-kernel software interfaces together (typec mux/switch/role switch ↔ extcon), and it's not driving a real hardware, it's certainly not going to be accepted upstream. Device tree should describe hardware and not software components. typec-extcon bridge is still a useful stepping stone on the way to a propper upstream support. It makes everything work with little to no changes to existing drivers, and it's easy to support until various drivers are extended with proper typec interfaces.

USB host/peripheral mode works very well now. Display port support is also much better, but still has some issues that need fixing.

Pinebook Pro port

I've also modified the DT of Pinebook Pro to use the same approach to Type-C support as Pinephone Pro. One major issue with Pinebook Pro is that it can't limit input current based on what software wants. It consumes 12.5 W from Type-C port if it can (as long as voltage doesn't drop bellow 4.5V). This makes proper implentation of power delivery impossible. It's risky to connect PBP to regular USB-A ports on the computer due to this. The limit is 5× of what it should be for USB 2.0 port, and 2.8× more for USB 3.0 port. It will also not work well with Type-C docks, that need to reserve some power for other devices connected to the dock and thus tell PBP over power delivery protocol to limit input current to < 2.5 A.

Megi's media section :)

If you're a visual person, here's a bunch of random images/videos that I posted recently to #pinedev, that you might have missed if you don't follow the developer's chat. :)

Next steps

LCD is not running at 60Hz. It's only running at ~53Hz and it's not very easy to improve this. Just changing the pixel clock frequency breaks the display output. Timing configuration for the mode seems to not match what's configured to the display. Kernel previously set the clock rate to 66.3Mhz instead of 69Mhz requested by the mode. With precise clock being set, the display doesn't work at all. So this needs to be cleaned up a bit. Application note for configuring the display for a particular timing is not available, so this will be hard. Few hours of trying various changes so far led nowhere.

Rockchip Type-C phy probably needs some improvements with connector plug orientation handling for display port signals. Both orientations work for USB3, but not for displayport alt mode. There's some patch available from Rockchip, so maybe it improves things.

2021–11–07: Pinephone Pro – USB, Type-C, OTG, DP-Alt mode, Charger,… – some success :)

Some good news below! :)

fusb302 disconnect bug

I've been slowed down by a bug in fusb302, that manifested as inability of the driver to detect disconnects when in host mode (with dock connected). That meant that when I unplugged the dock, rest of the drivers would not be notified, and this caused issues.

I had to read through the entire driver, write a summary of what it does in detail, and add some extra debugging functionality to dump fusb302 chip registers, before I found the issue. It was caused by wrong masking of interrupts, so the interrupt that should have notified the SoC about unplug was only set in the interrupt status register, but was not notifying the SoC because it was masked.

It's a longstanding bug in fusb302 driver that was there for years, so I guess it was causing issues on Pinebook Pro, too.

Bridge driver is complete

My typec → extcon bridge driver I was writing about in the previous post is now also complete.

Now I'm in the phase of doing various tests, to check for reliability and correctness of the behavior. So far it looks very promissing, and all the USB host/peripheral mode, Charger PD negotiation, BC1.2 detection, Display port alt mode, seem to work. :)

Here's a video from the tests:

https://www.youtube.com/watch?v=PTMXgPoylJA

(Dual display output with Pine64 dock with USB keyboard and a charger connected to the dock)

My implementation is different from the one done previously by Manjaro, and does not require any patches to the mainline drivers that would be unacceptable upstream.

Comparing Pinephone Pro convergence with original Pinephone

Type-C implementation on Pinephone Pro is way less complicated than on the original Pinephone. Figuring Type-C stuff on Pinephone took me many months, and required thousands lines of complicated code. I was still uncovering HW bugs until recently.

On Pinephone Pro, it just took about 2 weeks, required just a few simple few hundred line drivers, and it already seems rather complete, with no obvious missing things. Implementation is also using all the standard kernel interfaces.

I expect the remaining issues to just be a subtle corner-case bugs, rather than a missing functionality. The hardware part is also very simple, with very little opportunity for hardware bugs.

IMO, all this looks rather good, and much more boring than the original Pinephone! :)

2021–11–05: Pinephone Pro – USB, Type-C, OTG, DP-Alt mode, Charger,…

I've been looking at how all this works on Rockchip platform and in the mainline kernel for the last few weeks, writing docs, making diagrams, … and I've come up with a solution to tying together all the 7 or so drivers that need to communicate to make this all work.

This should end up being better than the incomplete solution that's currently available on Pinephone Pro or Pinebook Pro.

This effort will likely improve the situation on PBP, too. What I've come up with is a solution that will not require modifying any mainline drivers.

I wrote two new drivers that are needed for this to work. One for rk818 charger, that is needed to update the input current limit of the rk818 based on whatever is negotiated by the Type-C port driver and USB 2.0 phy (according to BC1.2 spec). The other driver is for bridging Type-C driver's native interfaces to extcon bus.

I'm quite close to first boot tests of this new approach. So, there will likely be some exciting news from me soon. ;)

2021–10–28: Fixing broken Pinephone, aka Pinephone repairability

One of the Pinephones I use for testing started having a parasitic power consumption of additional 900mW after it blue-smoked a Pinephone keyboard MCU during development of the Pinephone keyboard.

TL Lim sent me a replacement Pinephone Beta unit, because of this, but I still dislike to throw otherwise perfectly fine HW to the trash, so I decided to try to fix the issue.

Finding the culprit

Pinephone is disassemblable with just a screw driver and some fingernails (tweezers help, too). That means extracting the mainboard was quite easy:

The presenting issue is that when the phone's mainboard is off, it will still consume 900mW. This is quite a bit of power, that will surely create noticeable heat, and it will likely be concentrated around the culprit.

With the mainboard out, I tried to find, where this heat is produced. I had some ideas beforehand, so I focused on the PMIC and the circuits around the USB-5V voltage rail.

Some unnamed youtube personalities use alcohol evaporation test to identify a component that's heating up more than the rest of the board. Being the monkey I am, I tried that test, too. I poured some alcohol on the PCB and connected it to the battery.

PCB might have gotten a bit drunk, but otherwise nothing exceptional happened. The reason was that the battery was completely dead. Multimeter showed some 2.8V across its terminals, so it was providing no power to my test. I fully charged the battery a week ago and let it sit on my table, unused, so this was a bit unexpected.

Alcohol test is messy, so I gave up on that and just used my finger to find the heating component. 1W is not easy to hide, not even from my big fingers.

I found that D600 was getting very hot. D600 serves as a rectifier for the boost DC-DC converter for USB-5V power rail. It didn't make sense to me how it could be the the failing part, because the DC-DC converter is off when the phone is off, and USB-5V should not be loaded either in that state.

Something else connected to USB-5V must be heating up, too. So I continued my search. It turned out to be U1302, which is a DCIN to USB-5V switch. I measured a resistance across its terminals and it was 14 Ohm. It was not a switch anymore. It was loading the USB-5V power rail all by itself!

Quick calculation verified that the load matches with my power meter measurements. USB-5V has roughly a voltage of VBAT – 0.6V, when the DC-DC converter is off. That may be approximately 3.6 V. This ammounts to 3.6^2 / 14 = 926 mW. That sealed the case.

Fixing the issue

Removing U1302 is safe. It will just make Pinephone unable to supply power to external USB peripherals connected to the Type-C port.

So I decided to remove it. The more leads the part has, the more annoying it is to remove with just a soldering iron. So I decided to cut the leads on one side with pliers, and unsolder the leads on the other side.

And that fixed the issue. :) This Pinephone will continue serving some tasks well into the future, instead of being thrown out. It's not the first time I managed to fix some issue on one of my Pinephones in this way. It's all made possible by having schematics and component placement maps publicly available and by Pine64 making sure disassembly and re-assembly is easy and doesn't require any special tools or risky maneuvers. If I'd want, I can buy the replacement part on Aliexpress for $5 and solder it back.

Repairability at home with common EE hobbyist tools, is one of the other things I like about Pinephone.

2021–10–26: Pinephone HDMI hot-plug-detection HW bug fixed

The HDMI HPD HW bug I discovered previously that breaks USB-C Alt-DP mode was confirmed by the product team.

I've took some time to figure out a software based workaround to signalling hot-plug detection signal (HPD) from ANX7688 HDMI bridge to HDMI PHY inside the SoC and the rest of the DRM driver stack.

If you had issues with Alt-DP stopping at DP state 0x03 and not going further and was not able to use Pinephone with a HDMI dock connected to a monitor, now you should be able to with my 5.15 Linux kernel.

2021–10–21: Pinephone Pro – levinboot payload selection

I've extended Levinboot with support for chosing which payload to boot based on status of volume keys. Levinboot will also report which payload was booted using LED color. Ordering of payloads is based on physical location of volume keys, thus:

Here's a little demo:

With this setup it's easy to quickly try new kernel changes without any worry, and without having to swap uSD cards or open the phone ever again. The feeback cycle from code change to a booted kernel can be as fast as 10s, which makes for very pleasant development experience.

5.15 kernel fixed

I've managed to identify the cause for boot hangs on 5.15 kernel. The hang happens in of_platform_default_populate when processing the newly added debug@fe610000 (arm coresight) DT nodes (specifically the ones for the big ARM cores).

Precise reason is unknown to me, but I can live without the coresight support, so I commented those nodes out, and the 5.15 kernel now boots fine.

This allowed me to start playing with camera sensors support. I tried to add as much necessary support for the cameras as possible, and found a few issues in the process:

I've pushed my changes to my kernel tree to various branches. I've added DT changes that should match the hardware, but for some reason cameras don't work yet and will need more debugging.

I've also found out that SD card power supply was wrongly defined in the DT. Interestingly a lot of regulators in the phone are enabled automatically by chaining output of one regulator to the enable pin of the dependent regulator. This was the case with SD card regulator, too.

2021–10–18: Pinephone Pro – support merged into my kernel tree

I've integrated initial support for Pinephone Pro, that was previously done by Martijn Braam and Kamil Trzciński (ayufan) into my kernel tree, and spent yesterday looking at various issues that I've found with that kernel tree and fixing them.

I've fixed some of the most annoying issues, some of them with the help of carlos from #pinedev IRC chat:

After all these fixes I've made more suspend/resume tests, and it seems that it all works much better now. Pinephone Pro now idles at ~1.7 W, which is very close to what kernel can achieve by suspending all devices it has under control currently (s2idle). With suspend to RAM, it can get as low as 400 mW.

Next time it will be useful to figure out how to reliably power down and reset the phone, because that's not working great at the moment either.

It will be also interesting to figure out how to achieve 160 mW in suspend to RAM that the Android factory image seems to be capable of, from my previous measurements.

Pinephone Pro code is now integrated into my kernel in two branches: ppp-dt and ppp-drivers. I squashed all the original DT code and split the driver code from device tree changes and started building on top. I've excluded Type-C related changes, for now, because they are not documented well and I don't trust that they don't break non pinephone kernel builds.

I've also switched to 5 Ghz wifi (40MHz channel) and Pinephone Pro's seems to be able to achieve 15 MiB/s with that, which is a nice upgrade over original Pinephone.

More to come…

2021–10–15: Pinephone Pro – A Quick Review

I received Pinephone Pro (2021–05–11 batch) at the end of August. Here's a summary of my experiences with it so far.

Obligatory boot video :)

U-Boot is terribly slow in the default configuration, so I didn't even try it. But I ported Levinboot to PPP and here's Levinboot booting up the Arch Linux ARM from eMMC: (Levinboot is on SD card and the payload is on eMMC)

Whoa, that was fast! 6.15s to desktop. Exactly just like the Pinebook Pro boot times. ;) All that is thanks to CrystalGamma, and his work on Levinboot. :)

Here's a serial console view of Levinboot starting up on Pinephone Pro.

Power consumption

It's a portable mobile device, so I guess power consumption is what people will care about a lot. It has two aspects: how long it will run on battery, and how much it will heat up under load.

I have a nice setup for measuring the load at 1000 samples per second via this fake battery setup I made for original Pinephone:

Pinephone Pro came with Android factory image, that I used to get some initial numbers to compare against in the future. This image most likely doesn't limit the maximum CPU frequency, so the peak power consumption is quite high, due to that.

So this basically says that heavy use will drain the battery in about 1.5h, idling with screen on in aboutu 3–4h, and the phone will become unstable near the lower end of the capacity, because I don't think the single cell battery will sustain high enough voltage with load peaks of 10W or more near the lower end of the capacity. (that would be 3A+) battery in Pinebook Pro probably fares better at this, due to lower internal resistance. CPU throttling based on remainging capacity will be needed to control this issue.

When suspended to RAM (standby) the phone will last for about 2 and half days with the Rockchip's TF-A blob. This is worse than Pinephone, which has power consumption in suspend around 60 mW (2.5× better).

Once I had Arch Linux working on the phone, I also measured the power consumption on mainline Linux + mainline TF-A.

I used the default backlight brightness of 2000 or so, wifi was turned on, and I varried CPU frequency limits (cpupower frequency-set --freq $FREQ) and loaded the CPU cores with openssl speed -multi $N.

Baseline idling with brightness set to kernel default and 1.42GHz limit was 3.1W.

N   FREQ
---------------------
6    408 MHz    3.5 W
6    600 MHz    3.8 W
6    816 MHz    4.2 W
6   1.01 GHz    5.5 W
6   1.20 GHz    5.5 W
6   1.42 GHz    6.9 W
2   1.42 GHz    5.1 W
1   1.42 GHz    4.1 W

N = 1 (loading one performance core to max)
N = 2 (loading two performance cores to max)
N = 6 (loading all cores)

Backlight brightness vs power:

100     2.8 W  (almost not visible)
2000    3.0 W
3000    3.4 W
3500    3.7 W
4000    4.2 W
4095    4.3 W  (max)

The scale is non-linear.

So at full brightness/full CPU load at 1.42GHz, the phone will consume about 8.3W. This test is without the GPU doing anything. GPU will probably add something, too.

Some board photos

Some things I noticed:

What works

I did not yet test the rest.

Kernel status

I'd like to add cameras to DT, but that will need a working 5.15 kernel (because it has support for multiple cameras). Linux 5.15 is currently not booting on Pinephon Pro. It hangs mid-boot possibly on some issue related to RCU.

Other than that, non-mainlined drivers for the PMIC battery and charger support are needed.

The rest of the support is very similar to Pinebook Pro, ie. non-working convergence in mainline Linux.

Closing words

Pinephone Pro is basically Pinebook Pro with more useful PMIC, modem, cameras, and a better supported wifi chip (cypress supported cyw43455 is more open than broadcom supported brcm43456 in PBP) in a much smaller form factor.

Performance is thus much better than the original Pinephone, but the power consumption is higher too. Baseline power consumption with normal backlight brightness and screen turned on is ~2.8–3W. Doing something in the browser or just moving the mouse around rises the power consumption to ~5W. This was tested in sway, with GPU acceleration turned on, and only one CPU core enabled.

That's not a huge problem for Pinebook Pro with its 36Wh battery (4–7h runtime on a notebook is nice) 2–3h runtime with screen on, on a phone, is a problem. It will be a nice mini-notebook with a pinephone keyboard, though. :)

2021–10–13: How to adapt multi-boot image to your needs

I've added a program to p-boot that allows you to extract contents of the boot partition to a directory, so that you can modify it and re-import back into the boot partition.

This allows you to change menu items in the boot image, customize graphics of the boot menu, update the kernel, change kernel boot parameters, add new distributions to the existing multi-distro image, etc.

It's as simple as:

p-boot-unconf new-dir /dev/mmcblk0p1

Now you can edit contents of new-dir in any way you want. If you don't want to change the kernel, you'll only want to modify the boot.conf file.

To save the changes back to the boot partition, you can run:

p-boot-conf new-dir /dev/mmcblk0p1

And that's it. :)

Multi-boot image has some nice builtin structure that you can use to your advantage. Each distribution is contained in its own subvolume in the main btrfs filesystem on the second partition.

For example, the subvolume strucuture may look like this if you mount the second partition of the multi-distro SD card on your PC:

manjaro-phosh
.pre-boot/manjaro-phosh
.pristine/manjaro-phosh
pmos-mplasma
.pre-boot/pmos-mplasma
.pristine/pmos-mplasma

Here manjaro-phosh is the subvolume that's actually booted, .pre-boot/manjaro-phosh is snapshot of the manjaro-phosh before it was booted for the first time and .pristine/manjaro-phosh is the snapshot of the original root filesystem of the distribution, before I applied any fixes to it, to make it not mess up the multi-distribution image with so called first boot scripts.

So if you want to restore one of the distributions to the original state, you can simply run:

btrfs subvolume delete manjaro-phosh
btrfs subvolume snapshot .pre-boot/manjaro-phosh manjaro-phosh

And that's it. Next time you boot manjaro-phosh, it will be in the original state. This is useful if some update breaks the distro, or whatever.

You can use this to your advantage, and make a snapshot before trying some new thing that can potentially make the distribution unusable.

btrfs subvolume snapshot manjaro-phosh manjaro-phosh-2021-10-13

Go wild and if anything breaks, you can recover with:

btrfs subvolume delete manjaro-phosh
btrfs subvolume snapshot manjaro-phosh-2021-10-13 manjaro-phosh

In fact you can create several snapshots of the same distribution, add boot menu for each snapshot (snapshot is itself a subvolume and snapshots in btrfs are writable) and just use both snapshots in parallel. :)

If you don't want to keep losing your data in /home or /root when you switch between OS snapshots, you can create subvolumes for those directories and mount those subvolumes via fstab. That way you can revert from failed OS update, but keep your data (or share data between distributions in the multi boot image in general).

If you want to save space you can delete distributions you don't like from the multi-distro image:

btrfs subvolume delete manjaro-phosh
btrfs subvolume delete .pre-boot/manjaro-phosh
btrfs subvolume delete .pristine/manjaro-phosh
p-boot-unconf boot-export /dev/mmcblk0p1
# remove manjaro phosh from boot-export/boot.conf
p-boot-conf boot-export /dev/mmcblk0p1

The similar way (just in revers) you can add any new distribution to the multi-boot image:

btrfs subvolume create new-distro

# get rootfs tarball for the distro from somewhere
bsdtar -xp --numeric-owner -C new-distro -f tarball.tar.gz

p-boot-unconf boot-export /dev/mmcblk0p1
# add new-distro to boot-export/boot.conf (you can get inspiration
# by looking at the existing boot entries on how to do it)
p-boot-conf boot-export /dev/mmcblk0p1

And that's pretty much it. You may even boot the distro's own kernel and initramfs, if it supports btrfs out of the box. Otherwise, you can stick with my kernel.

If you want to update my kernel, the latest version is always available at: https://xff.cz/kernels in the ppd.tar.gz package. You only need to copy the Image file over linux-0.img and board-1.1.dtb over dtb-0.img and board-1.2.dtb over dtb2-0.img in the boot-export directory.

Once you have the boot filesystem exported to some directory, you don't need to keep re-exporting it over and over again. You can skip the p-boot-unconf boot-export /dev/mmcblk0p1 from then on.

You can also have multiple boot options for the same distribution in the boot menu. I use this to try different kernels during development. If one fails, I just boot the older one that is known to work.

The same way you can experiment with different crust or TF-A versions, without ever having to remove the back cover of your pinephone again.

All this management is also doable from the booted pinephone itself.

P-boot is that good. :)

The necessary tools are pre-built in the p-boot repository.

(WARNING: All filenames in the above text are examples, you have to modify the commands to your own situation/needs!)

2021–09–04: Pinephone HDMI hot-plug-detection HW bug

I've figured out what is the cause of an issue that some Pinephone owners experience, when using the convergence dock. It's a HW bug in the pinephone's design of level shifting circuitry for the hot plug detect (HPD) signal between HDMI bridge and A64 SoC.

Relevant circuit is here: https://megous.com/dl/tmp/452023169b0cce41.png

If you have this issue, and you're using my kernel v5.13 or later, which includes HPD change logging patches, you will:

NOTE: If you're using Mobian, the above may not apply to you, because Mobian cherry-picks patches from my tree, and may not include the HPD debugging one. If you want to help me debug kernel or hardware issues, debug with a distro that uses my kernel tree more directly.

This may also explain why for some people HDMI works only at certain battery charge level, because the issue depends on precise voltages and part tolerances in the particular phone, and these may shift a bit depending on the current voltage of the battery.

It's fixable in software. Quick workaround may be to ignore the HHPD pin's state, and try forcing the HDMI-A-1 connector to be enabled. You may also need to dump EDID data from your monitor and force load them on Pinephone. How to do it is described below.

It may be enough to make it work (I haven't tried the workaround myself), for people who experience this issue.

Proper software workaround will be a bit more involved, requiring to perform hot plug detection in software by querying the HDMI bridge chip's status over I2C bus, and passing it to DRM driver via a software path.

How to test with a quick workaround

  1. Dump your monitor's EDID binary data on some other Linux computer using cat /sys/class/drm/card-[connector_name]/edid > edid.bin
  2. Copy edid.bin to your pinephone and pass it to the kernel using cat edid.bin > /sys/kernel/debug/dri/[card_id]/HDMI-A-1/edid_override
  3. Force enable enable HDMI output by echo on > /sys/kernel/debug/dri/[card_id]/HDMI-A-1/force

Now you should be able to get past the dreaded DP state 0x03.

You need to figure out correct values for [connector_name] and [card_id] yourself by browsing the parent directories /sys/kernel/debug/dri/ and /sys/class/drm/card, because these can differ netween systems.

2021–08–10: Wrapping up Pinephone keyboard firmware development

The keyboard production seems to start soon.

I'm ending the firmware development for now. The final code is here:

https://xff.cz/git/pinephone-keyboard/

Samuel Holland helped with the final testing, and this code will be flashed in factory. You can read about the design of the code here:

https://xff.cz/git/pinephone-keyboard/tree/README

and in other readme files in that repository.

What I've achieved since June

Initially I received the vendor's flashing tool, vendor's proprietary firmware code for the keyboard and some schematics for the keyboard.

And this was just the keyboard MCU part. The second part of the problem is the charging chip.

All in all this project required a lot of bootstrapping, reverse engineering, and custom tooling support. That's why you'll find many tools in my code repository, each can still be used for one of the purposes I used them for originally.

I don't measure time spent on my hobbies, but this easily took ~100h of time over the last ~2.5 months and I quite like the result and all the new stuff that I've learned. That's also why I haven't done much on Pinephone kernel last few months. ;)

Lessons learned

I didn't know much about using USB from Linux and writing USB code for microcontrollers. Nor did I use USB in any of my hobby HW projects in the past, and that changed after this project. I thought USB is complicated. I found that using USB as a dumb transport is quite comparable to using a serial port. It needs more code, but USB solves issues that you have to solve by hand in some upper protocol layer you layer on top of serial port communication, so the complexity is comparable.

On Linux side you can access USB devices with about 4–5 ioctl calls, and in most microcontrollers that I have access to, you can write basic USB device code in about 200 lines (no need for large USB stacks supplied by the vendor when you're just using USB as a dumb transport instead of trying to implement some of the standard USB device classes).

In fact I wrote 2 USB device implementations during the development of the keyboard. One for the controller inside the keyboard, and one for FX2 board I used to simplify testing. And it was not that hard. FX2 is especially easy and well documented, with hardware helping the firmware writer quite a bit at each step.

It was a fun ride, mostly, except for some stressful frustrations towards the end. I don't mind frustration with trying to fix a broken HW, reverse engineer stuff, etc. There was a lot of that in this project after all. Interrupt driven MCU code is especially tedious to write, since you have to re-read the code several times, searching for potential cirical sections, etc. Compiler does not help at all here.

What I do mind is being put under time pressure on a project where I'm just volunteering my free time. That's very easy way to get bitter feelings, which I'd like to avoid.

I've damaged the last prototype trying to figure out some way to connect I2C of the keyboard MCU to the charger I2C so that phone can monitor the keyboard battery charge and control the charger.

At the moment I have no way to do anything with the keyboard, until I fix my prototype to revert the various HW modifications I attempted, or get the final keyboard, so the feeling of accomplishment is a bit bittersweet. Oh well. :)

Some random photos from my efforts

First effort to give access to charger I2C to the phone (using TXS0108 level converter):

Second attempt to give access to charger I2C to the phone (via direct connection of I2C A MCU interface to the charger chip). A plan:

Execution:

And giving up after weird voltages on I2C pins and no more time to try things: :(

Final untested suggested changes for having I2C access to the charger proxied by the keyboard, that will be present in the final keyboard design, to have a chance to figure out the charger I2C communication later on (as a firmware update):

My new favorite USB dev board (Cypress FX2), that I used to test I2C interface of the keyboard:

FX2 controlling the keyboard over I2C from the PC:

2021–07–26: The latest Pinephone keyboard prototype

The last prototype arrived today, so I uploaded my firmware to it and tested it. It looks good! There are no shorted or interrupted columns and rows this time, the two PCBs were merged into one, and all keys are easy to press and react well. Even spacebar now reacts to key press along all it's surface.

(„stuck“ keys is just some issue with debugging USB interface code I'm using, some reports don't get through)

There's one thing that I didn't realize before though…

Phantom keys

My original enthusiasm about being able to register arbitrary key combinations has to be tempered a bit by reality. :)

The general issue with matrix keyboards can be described like this:

If you press 3 keys in this specific pattern on the matrix (electrically), the fourth corner of the box pattern will also appear as shorted, because it's specific column and row wires are shorted electrically via the other three keys. I think this is called a phantom key, and is a general problem of keyboard matrices.

Here is the illustration of the issue on the real keyboard:

It's worked around by placing typical modifier keys on separate columns and rows:

The rest of the keys are placed on the matrix in such a way, that pressing Ctrl+Alt+Shift and one other regular key will not create any phantom key effect. And when it does, the phantom key is on the unused combination of the row and column, so it can be ignored.

This makes it so, that you can use modifier keys freely, without ever noticing this fundamental issue. In fact, I haven't noticed this on normal PC keyboards in the last 20 years. But they have the same issue!

For example I can't press C+V+G on my desktop keyboard without the keyboard ignoring the last pressed key, whichever one it is. I never noticed, and I don't think I'll ever notice that again. :) Three letter combos are not common, except for government agencies.

And this is actually the workaround for the issue, too. The keyboard's software checks after each new pressed key, if the key would produce a phantom key effect, and if it would, it ignores both the actually pressed key and its phantom counterpart.

Of course this gets even more complicated if you press more than 3 keys, depending on which combo it is.

Summary

The above issue just has to be handled in the driver/userspace software. There's not much else that can be done.

I've checked the actual configuration of the pinephone keyboard matrix, and it is designed in such a way that you can use any combination of Ctrl+Alt+Shift + one other key (that is not Fn) safely.

Fn key is on the same row as Alt key, so you can't combine those two. You can combine Ctrl+Shift+Fn arbitrarily though.

You can also use any combination of any two keys on the keyboard.

Enter and Backspace keys are on their own column, so one of those can also be combined with any other safe key combination as described above.

So that's all rather good, in the end! :)

Some more photos

Schematics:

This is how to connect USB to flash the basic firmware over USB. Some rather easy soldering:

(Final version will allow flashing over I2C.)

Next steps

2021–06–28: Pinephone keyboard's final summary

I finished most of my testing of the pinephone keyboard, and also completed the firmware, and this post should serve as a summary of my observations.

There are some open questions around the charger, that need further investigation.

Next steps is completing the TODO list, which contains only minor things at this point.

Suggested HW changes

After quite some testing, this is a final set of suggestions I have:

2021–06–20: Pinephone keyboard's firmware I2C interface

I've implemented a USB stack for the pinephone keyboard's firmware, to be able to perform printf debugging and tracing of what the MCU does. It's now possible to see MCU's operation using a simple USB debugging tool that shows the output from debugging code on the MCU in real-time.

I've used the new debugger it to get to understand precise behavior of I2C slave peripheral. How interrupts are fired, and what's the contents of various status registers during various I2C transfers from the pinephone SoC over POGO pins.

This understanding can be used to write a more proper I2C control interface for the keyboard, that will be modelled as is typical with various other I2C devices a set of I2C „registers“ that can be read from and written to. The proposed specification for the register set is here. It will be possible to configure the keyboard's scanning behavior in various ways over the I2C, and perform self-tests for QA purposes.

I've also finalized the USB flashing tool. It now has a very nice interface and is quite useable and better documented.

Samuel shared his keyboard kernel driver recently. So you'll have a choice between kernel and userspace drivers for the keyboard. Both have their strengths. Userspace one will be easier to customize and play with, and the kernel one will be more available and better integrated into kernel's suspend/resume functionality, I assume.

User firmware

From factory the keyboard will come with the stock keyboard firmware. This firmware will occupy code memory from addresses 0x2000 to 0x3fff. The stock firmware will also have a I2C flashing interface, whose proposed workings are described here.

The superpower users will be able to write their own firmware, and flash it to the keyboard over I2C directly from the phone, without having to disassemble the keyboard and solder on a USB cable. ;) This firmware will not overwrite the stock firmware, but it will be placed at addresses 0x4000 to 0x7fff, after the stock firmware. If the user firmware is flashed, it will run instead of a stock firmware.

There will be a short time window where the stock firmware will wait for a I2C transactions prior to defering execution to user firmware after MCU reset. This window can be used to prevent the jump to user firmware, and stay in the stock one. This will be the mechanism for recovery in case the user flashes incorrect or borken user firmware.

This mechanism can be used to:

Power user based hardware or firmware modding the keyboard is quite a likely possibility when everyhting is developed in the open using FOSS licensing, with focus on making this relaitvely easy.

I'll propose exposing the free GPIOs as solder pads for the final revision of the keyboard's PCB.

There's just enough free space in the keyboard's body for a little drilling and eg. adding proximity sensor (to detect whether the lid is closed or not), to add notification LEDs visible when the KB is closed, etc.

You can also make much more space inside the keyboard by removing the battery, and replacing it with your own hardware. You may also expose GPIOs externally by adding a new connector to the keyboard body, and have some simple pluggable peripherals.

Next steps

Next, it's necessary to implement the proposed I2C interface. Most complicated part of which is the flashing interface, which will require further reverse engineering of the original USB bootloader, because code ROM flashing registers are not documented.

Once that is done, it will be necessary to again look at making the MCU operation as low power as possible, by using the power down and green modes of the MCU.

Charger interface

Charger's I2C interface will still need quite a bit of thought. There's an issue that the charger chip multiplexes I2C interface with the LED notification pins, and detects what kind of interface is used by checking the voltage on the I2C pins whenever it wakes up, which is not predictable.

I2C interface is shared with the keyboard's MCU, so if charger wakes up when the MCU is communicating with the SoC, it may see low voltage on I2C pins, and misdetect the interface as LED one, and start driving it as such and break communication between MCU and SoC.

Samuel also identified another way the charger can break I2C interface, and that's when the phone is off, and thus it does pull I2C SDA/SCL lines low via 10 kOhm pull-ups to DCDC1 on the SoC side, the voltage on I2C lines will be just ~1.5V, which is not enough to make the charger think I2C interface is used, and it again switches to LED driving mode.

We need to figure out some way to prevent this from happening.

The last proposal is to add mosfets between charger I2C pins and the POGO I2C pins, with gate connected to INT pin. When the INT pin is low (because it's driven low by MCU, or because DCDC1 is off, because the phone is off), these mosfets would disconnect the charger I2C pins from the POGO ones, and this would make the charger always detect the I2C interface mode correctly. It would also be possible for the SoC to disconnect the charger from I2C interface at any time by driving the INT pin low during I2C communication with the MCU.

Lastly when the phone is off, this would save some power by cutting off the path between „grounded“ DCDC1 and pull-ups to VREG (always on 3.1V linear regulator inside the keyboard charger chip).

2021–06–16: Pinephone keyboard input daemon released

For those hungry for more information about the pinephone keyboard's software, I've written up some notes:

https://xff.cz/git/pinephone-keyboard/tree/HACKING

and updated a TODO list:

https://xff.cz/git/pinephone-keyboard/tree/TODO

I've also cleaned up the firmware and released a basic input daemon, that can be used to operate the keyboard. It already supports all key combinations on the default keyboard map.

https://xff.cz/git/pinephone-keyboard/tree/inputd/factory-keymap.txt

https://xff.cz/git/pinephone-keyboard/plain/inputd/factory-keymap.jpg

With this, it's possible to actually start using the keyboard prototype day to day.

My HW keyboard is broken, and is not very usable to test multi-key combinations. Next revision will hopefully be better.

2021–06–14: Hints on debugging HDMI output on Pinephone

Some people have issues using external monitor with pinephone.

There are some ways to debug it. The most reliable one is to check the state of DRI in debugfs.

If you expect output on the monitor and there's none, you can check contents of file /sys/kernel/debug/dri/0/state or /sys/kernel/debug/dri/1/state to see what it says about the current state of the display pipeline as configured by your compositor. With only internal display turned on it will contain:

...

crtc[45]: crtc-0
        enable=1
        active=1
        self_refresh_active=0
        planes_changed=1
        mode_changed=0
        active_changed=0
        connectors_changed=0
        color_mgmt_changed=0
        plane_mask=2
        connector_mask=1
        encoder_mask=1
        mode: "720x1440": 60 72000 720 750 778 808 1440 1458 1468 1485 0x48 0xa
crtc[54]: crtc-1
        enable=0
        active=0
        self_refresh_active=0
        planes_changed=0
        mode_changed=0
        active_changed=0
        connectors_changed=0
        color_mgmt_changed=0
        plane_mask=0
        connector_mask=0
        encoder_mask=0
        mode: "": 0 0 0 0 0 0 0 0 0 0 0x0 0x0
connector[56]: DSI-1
        crtc=crtc-0
        self_refresh_aware=0
connector[58]: HDMI-A-1
        crtc=(null)
        self_refresh_aware=0

If you have in your dmesg:

...
DP state changed to 0x03
...

as the last DP state changed message, you should be able to just see your monitor's modes and enable the HDMI output via xrandr if you're using Xorg, or equivalent tool if you use something else.

It's the role of the compositor to enable the output to HDMI, so if you see connector[58]: HDMI-A-1 crtc=(null) and not connector[58]: HDMI-A-1 crtc=crtc-1 and some reasonable mode configured on crtc-1 it means that your compositor simply did not enable the HDMI output, so your monitor stays off.

If this happens on Xorg, xrandr --auto is the quickest way to at least get some output.

You need to consult documentation for your distribution's compositor to see how you can configure display output.

If /sys/kernel/debug/dri/*/state configuration looks correct, but your monitor is still not on, it's either a kernel or perhaps a HW issue.

If you sometime get to DP state changed to 0x03 after plugging in the dock, and sometimes not, you should check if you see some received SRC_CAP messages or received SVID or DP_ALT_ENTER messages in your dmesg after the last cable plugin event. If not, it means communication with the dock over the CC pins doesn't work or didn't happen for whatever reason, and the phone was not thus able to configure DP-Alt mode in the dock.

It may be due to some connector issue. Pinephone Type-C connector is a bit too much recessed into the body of the phone, and some Type-C cables don't plug in completely.

2021–06–14: First pinephone keyboard typings with my FOSS firmware

https://youtu.be/Kx6B_OL4OJ4

:)

2021–06–14: Pinephone keyboard firmware

So I managed to write and test my USB flashing tool for Pinephone keyboard.

I've also managed to write FOSS firmware for the keyboard, that can provide 12×6 bitmap of all currently pressed keys. So any combination you'll manage to press will be properly detected, without limit.

Currently this interface is consumed using a userspace daemon that waits for H->L transition on pin PL12 (POGO interrupt pin) and reads out the current bitmap of all pressed keys.

In the future this daemon can provide a userspace implementation of Linux input device. This is a way to easily support any kind of keymap without having to touch the firmware. All the mapping could be done in userspace and can be easily modifiable by users, without any re-flashing of the firmware. Userspace input devices behave just like the ones provided by the kernel, and work everywhere incl. Linux console.

I've also investigated the charging controller inside the keyboard, and it can be controlled over I2C from the phone itself. I've demonstrated reading of battery voltage/current and status of the button on the side of the keyboard. This button is connected to the charging controller.

Here is a repository with some of the code: https://xff.cz/git/pinephone-keyboard/

And here you can see the code in action: https://youtu.be/hj8DIqD74IM

Hardware-wise, the prototype I have has some issues. Some keys don't react well to being pressed (all the keys on the top row), and enter also sucks. Control and C keys are merged (pressing one, presses the other one too), and Z key is stuck (+ one other row/column combination is shorted). Shorted rows/columns on my prototype are R4/C2 and R5/C4. It may be better if enter key had two membrane contacts, because it doesn't press/react well due to its size.

All in all, things look rather bright on the software/firmware front. :) I'll probably continue hacking on the keyboard's software next weekend.

2021–06–11: PinePhone keyboard – HW testing

Pinephone keyboard arrived a workday earlier than advertised by DHL, so I started looking at it today.

Looking inside you see two boards. Top one is for the battery charger, the bottom one is for the keyboard controller:

This prototype has I2C interface from the charger controller wired ad-hoc to the I2C interface on the POGO pins. This will be useful to test how and if it really works and if the charging chip is controllable. :) I haven't checked whether the components on the other side of the board are not populated, to make the I2C interface available, but hopefully they are.

This prototype also has POGO pins and USB interface nicely exposed as solderable pads on the USB controller board:

You'll need to be careful when soldering the USB cable to the keyboard controller board, because battery wires are directly under the through hole pads you'll need to solder the USB cable to. If you do it with the board mounted as is, you'll most likely burn the insulation on the battery wires.

Soldering is necessary if you'll want to flash your own keyboard firmware.

To avoid burning the insulation, I've first unscrewed the board and padded it with a paper tape.

I've used jumper wires instead of soldering the USB cable directly, because I plan to add a USB flashing connector to the chasis, once everything is verified to work, and no further HW mods are necessary.

All that remains is to solder the jumper wires to the USB cable temporarily…

… and plugging in the keyboard to my PC to verify it works:

input: HID 04f3:1812 as /devices/pci0000:00/0000:00:01.2/0000:02:00.0/usb1/1-2/1-2.1/1-2.1:1.0/0003:04F3:1812.0038/input/input58
hid-generic 0003:04F3:1812.0038: input,hidraw5: USB HID v1.11 Keyboard [HID 04f3:1812] on usb-0000:02:00.0-2.1/input0
input: HID 04f3:1812 Mouse as /devices/pci0000:00/0000:00:01.2/0000:02:00.0/usb1/1-2/1-2.1/1-2.1:1.1/0003:04F3:1812.0039/input/input59
input: HID 04f3:1812 System Control as /devices/pci0000:00/0000:00:01.2/0000:02:00.0/usb1/1-2/1-2.1/1-2.1:1.1/0003:04F3:1812.0039/input/input60
input: HID 04f3:1812 Consumer Control as /devices/pci0000:00/0000:00:01.2/0000:02:00.0/usb1/1-2/1-2.1/1-2.1:1.1/0003:04F3:1812.0039/input/input61
input: HID 04f3:1812 Wireless Radio Control as /devices/pci0000:00/0000:00:01.2/0000:02:00.0/usb1/1-2/1-2.1/1-2.1:1.1/0003:04F3:1812.0039/input/input62
hid-generic 0003:04F3:1812.0039: input,hiddev99,hidraw6: USB HID v1.11 Mouse [HID 04f3:1812] on usb-0000:02:00.0-2.1/input1

And it does. :) This is USB interface exposed by the main firmware. It's probably possible to press keys and control the PC using the keyboard over USB HID, as is.

Entering the flashing mode requires issuing some command over one of these HID interfaces, that will cause jump to the bootloader, which will expose altogether different USB device interface meant for flashing.

Next step is writing and testing a FOSS flashing tool. It should also be possible to test the charger I2C interface. These two tasks are independent.

I'll start with the flashing tool.

2021–05–30: PinePhone keyboard – more observations

After further eyeballing of the firmware in Ghidra, with the help of a linker symbol map, I've reverse engineered the USB flashing protocol completely.

I'll not publish complete details at this time, but here's a general idea of how it works:

All the details of the flashing protocol are known to me now, and I have a flashing tool in the works. It should be possible to finish most of it even prior to getting my hands on any hardware, but I guess there's no hurry at this point with all the details known. :)

Out of all this research I personally plan to do these HW modifications on the keyboard to help make it more useable/robust:

Overall, the situation around flashing arbitrary firmware to the keyboard controller reliably seems quite reasonable, without too many gotchas for the end-users. With a known good keyboard firmware users should be able to recover from flashing failures without the need to open the keyboard or a huge risk of bricking the keyboard.

The only thing that will be troublesome to end-users is the possible non-confirmance of the charging circuit to USB specification, as mentioned in the previous post. It will be probably challenging to insert the keyboard into the USB ports on the computer without the port shutting down due to overcurrent.

I'll be also looking at the actual keyboard firmware (main app). Most of the existing code deals with exposing the USB HID interface, which is not used by Pinephone. Actual keyboard interface used by Pinephone is HID over I2C. Code for that is much simpler. There's a lot of dead code in the existing firmware provided by the vendor. All that's needed is just reading out currently pressed keys from the key matrix, and providing updates over I2C on changes. That should be a few hundred lines of code tops, not the current ~6000 lines. USB interface is only used for initiating the switch to flashing mode of the bootlader from the main app, but we can instead use a fixed key combination to do that (Pine key+F+M, for example) and drop all that other dead weight from the firmware. This will make firmware also much smaller and faster to flash (currently it's 19 KiB, or 16 KiB when compiled with optimizations, but it can be ~2 KiB).

If you'd like to support the effort to make FOSS firmware flashing tool and customizable firmware for the Pinephone keyboard, donations are welcome.

2021–05–26: PinePhone keyboard

I did some analysis of Pinephone keyboard prototype's schematics, components and firmware. Here are some of my observations.

Parts

First let's look at major parts of the keyboard, and what features they have.

Charging controller – IP5209

The chip is designed for use in power banks with optional LED flashlight.

It takes input from a 5V power supply or the battery and provides 5V output using a boost step-up DC-DC converter.

It has some other minor features, like LED indication, and timed key input (long/short press of the key do different things).

It also has rather detailed control of its functionality over I2C bus:

Charger/output control:

GPIO:

ADC:

This is quite a lot of useful functionality for Pinephone keyboard's use case!

Resources:

USB microcontroller – EM85F684A

This is rather generic and simple microcontroller with USB bus support. There is nothing really remarkable about it.

It uses 8051 instruction set, and has 32768 bytes of flash memory for code with 10,000 write/erase cycles. 2048 bytes of XRAM and 256 bytes of RAM.

Pin change wakeup is available on ports 5, 6, 9. There's a I2C slave peripheral, that is connected via POGO pins to Pinephone.

It has low power mode, that allows disabling CPU clock. Remaining features are not useful for the keyboard use case.

Documentation for the basic features is available on the Elan microelectronics website.

Resources:

Pinephone keyboard schematic

Keyboard schematic is rather simple:

Resources:

Proprietary firmware written in C was sent to some developers, incl. me, and I was able to compile it using Keil C. Firmware is half C code, half „bootloader“ distributed in binary object form. So it's possible and rather easy to modify the firmware. The issue will be with flashing it.

At this time, only sure way to program this microcontroller is to have a HW programmer from Elan microelectronics, install a windows driver and software, and use some windows GUI tool to flash the firmware over pads exposed on the PCB inside the keyboard.

There's also a possibility of flashing the firmware over USB, but the protocol is not documented, and there's no non-proprietary program available to do it.

This flashing method currently depends on proprietary bootloader, and cooperation of the existing keyboard firmware with entering the flashing mode. So if a broken application code is flashed, it will not be possible to re-flash it again using this method.

At my request, TL Lim sent me a Windows flashing tool for the keyboard's firmware. I've started reverse engineering it, and at this time I have the basic protocol reverse engineered, and I should be able to communicate with the keyboard using /dev/hidraw# device. I don't have the keyboard prototype HW, so I can't do much more at this time.

Next step is to get a capture of USB communication of the original flashing tool for Windows using usbpcap and cross-check the reverse engineering data with the actual communication and then write a Linux port of the flashing tool. This will allow me to create a better firmware updating mechanism for the keyboard controller, and experiment with key scanning routines. Looks like there should be no limit to how many keys can be pressed at the same time. :)

It would be nice if people with the keyboard prototype did the capture, otherwise the development will stall until I receive the prototype, too.

Pinephone charging/battery circuit schematic

Charging circuit can not be controled over I2C bus, because I2C bus from Pinephone is not connected to it. This is rather limiting.

Resources:

Suggested modifications

  1. Connect IP5209 to the i2c bus, instead of using 4 LEDs for charge indication.
  2. Revise the charger schematic according to the suggested reference circuit from the manufacturer of the chip. Add the current sensing resistor.
  3. (keyboard v2?) Add support for a lid switch.
  4. (myabe with optional mod/soldering bridge) Allow to share interrupt lines with keyboard controller and charge controller (not share by default)

Open questions

2021–05–22: PinePhone kernel news

There are a bunch of new notable things in my kernel since the last update 2 months ago. I've added some patches from others:

And I also continue to tune up and improve my and other drivers:

I've also done more power draw measurements a few weeks ago (that's where the before/after pictures linked above are from), and pinephone's power consumption in suspend is now around 60mW with modem off. Modem usually adds another 10–15mW. So with 11Wh battery, that means around 6 day standby, if that's all you do with the phone. :)

Overall 5.12 and 5.13 in my tree should be working quite well at this time! Samuel's time travel fix („clocksource/arm_arch_timer: Improve Allwinner A64 timer workaround“) seem to have solved the last stability issues I had with some of the pinephones I have access to. Things look pretty bright!

There are still some HDMI issues some people keep reporting, that manifest themselves by anx7688 firmware not getting past „DP status = 0×03“ (it should go from 0 all the way to 6 normally). When this happens it's caused by HDMI controller inside the SoC using incorrect clock frequency, and anx7688 fails to acquire the HDMI signal from the SoC. It should be easy to fix, but I have trouble reproducing it with my kernel and userspace, so I can't identify the root of the problem, yet.

I've also worked on musb DMA support, and I got quite far (my current work is here https://megous.com/git/linux/log/?h=musb-sunxi-dma-wip). It seems that DMA controller works (I can see DMA completion interrupts getting triggered), but integration of DMA driver with mainline musb driver is a bit complicated, and the only documentation is the existing BSP code. Musb seemingly has nice abstraction for supporting various DMA drivers, until you look closer and realize that musb core has many code paths that change how things are done based on what DMA driver is used. So in the end the interface between musb core and DMA drivers is quite messy. :(

2021–03–17: PinePhone eMMC measurements

I wrote a simple io_uring based benchmarking tool for block devices.

It's very simple. It creates a fixed size queue of random reads in the io_uring and then just keeps the queue full.

I wanted to see what my devices can do. It maxes out at 640k random read 4KiB IOPS on my 500GB Samsung 980 PRO nvme SSD, with no special Linux block subsystem tuning. Which is about in the middle of the supposed 500k and 800k values in the datasheet. It's hard to interpret various small print notes, so it's not clear what to expect with reads scattered across the whole drive. I may try tuning the kernel configuration to see if I can get higher, but this may already be close to the HW limit. So the tool can produce quite a bit of IO activity. :)

I used this tool on various uSD cards and eMMCs I have in my Pinephones.

Here are my results:

Pinephone 1 (early dev sample)

eMMC: (date=07/2019 manfid=0×0000da oemid=0×0101)

uSD: (32GiB Sandisk Ultra A1 08/2019)

Pinephone 2 (UBports endition)

eMMC: (date=07/2019 manfid=0×0000e1 oemid=0×0116)

uSD: (32GiB Sandisk Ultra A1 08/2019)

Pinephone 3 (Fixed USB-C UBports endition)

eMMC: (date=07/2019 manfid=0×0000e1 oemid=0×0116)

uSD: (32GiB Sandisk Ultra A1 09/2018)

Pinephone 4 (3GiB UBports endition)

eMMC: (date=12/2019 manfid=0×000045 oemid=0×0100)

uSD: (32GiB Sandisk Ultra A1 10/2019)

The tool, in case you want to check out the code.

Queue depth doesn't seem to matter much with eMMC/uSD cards in PinePhones, while it does with high performance SSDs on my desktop.

2021–03–15: Some PinePhone updates

There were a bunch of smallish news/fixes in my kernel since my last post:

Kernel log during boot is now a bit less noisy with non-important errors. This is mostly due to use of dev_err_probe() function that hides the temporary probe errors in case probe of some driver fails due to missing dependencies. If you want to see drivers that failed to probe due to a missing dependency, those are listed in /sys/kernel/debug/devices_deferred.

I've fixed the ANX7688 flashing timeout, so that flashing the firmware to ANX7688 works again on my 5.10 and 5.11 kernels.

There was some interest in mainlining the device tree bindings for ANX7688 this month, so I've added my observations to the discussion. Hopefully this will lead to getting ANX7688 driver mainline, eventually. Though the discussion has stalled.

There's an interesting new feature in my Linux kernel. Samuel proposed a nvmem based reboot driver. So I've took the idea, and updated the p-boot implementation so that it works with a modified variant of the proposal. It's now possible to reboot into a different boot option, or into eMMC, or into a p-boot boot menu, or even into FEL just by calling reboot(LINUX_REBOOT_MAGIC1, LINUX_REBOOT_MAGIC2, LINUX_REBOOT_CMD_RESTART2, "something"), where something can be one of emmc-egon, fel, menu, sd1, sd2, emmc1, emmc2, … and more. This will make the kernel inform p-boot to perform boot in a particular mode after the reboot, regardless of the default boot menu selection. Neat, eh? You can pass this argument to systemctl like this: systemctl reboot --reboot-argument sd3, and the phone will reboot to p-boot's 3rd boot menu option for SD card.

I've enabled USB_QUIRK_RESET for the modem's USB device in my kernel tree. This is supposed to help with modem USB interface resets during resume from sleep. more information With this it is not necessary to change power/persist: 0 and avoid_reset_quirk: 1, because that's done by the kernel automatically when the modem's USB device is probed.

With the help from #linux-sunxi/Peetz0r, I've enabled remote wakeup via WiFi packets in the 8723cs driver.

You can now configure the wifi driver to receive and react to some packets and wake the phone up if it's sleeping.

On PinePhone:

iw dev # and write down mac address, eg. 02:ba:7c:9c:cc:78
echo clean > /proc/net/rtl8723cs/wlan0/wow_pattern_info
iw phy0 wowlan enable any

# ... suspend to ram ...

On other device:

wol -i pinephone_ip_address 02:ba:7c:9c:cc:78

# ... pinephone wakes up over wifi ... :)

And the last change of note is a new patch that enables PMIC to wake the SoC from sleep when the battery is running low (<10% capacity), so that the OS may shut down gracefully, instead of just running out of power while asleep. This needs some userspace support, which is probably already implemented by some desktop environments. In case it isn't, the phone will just wake up from sleep and die sooner. I don't think the PMIC will keep sending interrupts, so the OS will need to be careful to use this opportunity to power off. It may still be more reliable to just perform periodic RTC based wakeups for various book keeping tasks, incl. battery monitoring, instead of relying on this one shot low battery interrupt from PMIC.

You can follow these kinds of changes and news in my kernel by looking at the latest orange-pi-$VER-$DATE tags in my kernel tree.

If you click at the tag name, you'll see the NEWS log. It's just not as pretty as this blog. :)

The latest tag also gets a build here: https://xff.cz/kernels/ for all the devices that I maintain my kernel for.

I didn't do much new work on my previously announced modem userspace project, because while I was away on holidays biktorgj seems to have poured a lot of effort into the same general idea, so there's no real need.

I have a bunch of things I'd like to try to tackle next. One of those is imprvoing the performance of USB peripheral mode. Currently when connecting PinePhone to a PC directly via USB cable, the performance of the USB is limited to speeds around 10 MiB/s. This is due to missing DMA implementation in the driver, so CPU has to push every byte individually into the FIFO, which is not very optimal. I have a work in progress implementation of the DMA support that I'd like to finish. Hopefully this will improve the speeds to be closer to the theoretical limit of ~40MiB/s.

2020–12–19: Away from keyboard for the rest of the month

I'll not be able to respond to e-mails until January. No worries, everything is fine. :)

2020–12–15: Battery thermistor measurements

A Mobian wiki user nivea made some interesting measurements of the thermistor present in the Pinephone battery https://wiki.mobian-project.org/doku.php?id=hardware:battery and the results show that the current safety values configured in PMIC by the kernel are roughly correct. Great!

2020–12–05: Alternate EG25G userspace project

I've been reverse engineering EG25G userspace on and off since mid July 2020.

I really dislike the Quectel's code. They took relatively clean (if a bit overengineered) Qualcomm code and made a total mess, that somehow barely holds together.

They manage to do things like sprintf(buf, "echo %s > /dev/kmsg", barely_checked_user_input); system(buf); in a superuser process, or system("rm /something") instead of unlink("/something"), etc. None of their code inspires any confidence in the modem's software as a whole.

Biktor has a nice bootloader/kernel cleanup project, so I decided to start a fully FOSS userspace rewrite project to complement it.

How it will look/work

Alternate FOSS userspace code will be very simple and will not depend on anything in the original root filesystem of the modem's firmware.

You'll be able to install the FOSS userspace along with the existing rootfs content and choose which one to boot to via my modem power manager:

# to power up the modem and boot the original userspace code
echo 1 /sys/class/modem-power/modem-power/device/powered

or:

# to power up the modem and boot the FOSS userspace code
echo N /sys/class/modem-power/modem-power/device/powered

Modem power manager will configure DTR and #W_ENABLE gpios to one of the 4 possible configurations, and then it will power up the modem. These two gpios can be relatively easily read from the modem's userspace during boot, and can be (a)bused to make boot decisions very early on after the kernel starts the /sbin/init process.

The existing firmware will need just a single simple and fairly safe modification. The rest of the changes will be just addition of new files to the root filesystem to a /foss/ directory.

In order to make boot decisions, the FOSS userspace will include a small /sbin/init binary that will be executed by the kernel instead of the original one, and will check the status of the above mentioned gpios and then execute either /foss/init.2, /foss/init.3, /foss/init.4, or the original firmware's /sbin/init.sysvinit. One of these will become the PID 1 process.

This will be the foundation for easy experimentation with the FOSS userspace for the modem's OS running on it's ARM CPU.

Anyone will be able to modify and test the FOSS userspace code quite fearlessly. Recovery will be as simple as booting to the original userspace, or keeping one of the /foss/init.* binaries in an always known-good state, and only experimenting with the others.

Plan / progress

This should hopefully be enough to be able to perform voice calls without having to use any part of the original Quectel's code that is present on the root filesystem of their firmware.

I expect the FOSS code to shrink the firmware quite a bit. Most of the heavy lifting is done by the Qualcomm/Quectel Linux kernel and their modem's DSP code. Userspace code just has to perform some setup tasks by modifying a few files in sysfs and keeping track of a few special needs the modem's DSP code may have, like configuring the codec during the call, and some other minor stuff.

This can probaly be easily handled all by a single 200kB static binary instead of a huge 80MB busybox based system with 15 interacting daemons, hundreds of shell scripts and hard to capture behavior.

And there will be 80MiB of newly released space for some actually interesting additional code to be run on the modem, if you choose to delete the original proprieatry code. :)

2020–12–02: Fixing fallout from the new WiFi driver

While testing caching and latency of call indication messages, I've noticed that the updated WiFi driver broke suspend/resume cycle, and also that it increased the resume times significantly.

Basically the driver has a compile option to either do the whole of WiFi resume synchronously during resume, or to punt it to the separate thread and don't block the resume process. The latter option only works with Android wakelock API. Though the only thing that the driver needs to be able to do this is to block further suspend to RAM until the asynchronous resume job finishes. That was fairly easy to achieve with mainline API, so now WiFi driver will do its 850ms of resume tasks without blocking the resume process as a whole, which makes userspace responsive just 400ms after resume is initiated, instead of after 1.2s.

Also WoWLAN support code path made the driver return -ENOSYS (-38) from the suspend callback, which aborted the suspend to RAM. I disabled WoWLAN in this patch https://megous.com/git/linux/commit/?h=wifi-5.10&id=6959135b030f7b3bb4db4c42824ab1743436889a I don't think anyone uses WoWLAN support, so that seems like an acceptable solution for now.

One thing that I noticed about the WiFi on Pinephone with the updated driver and newly enabled power management is that it can take some time to ping through to the phone from my PC, after there was no network activity for some long period of time. This is the same behavior that the broadcom wifi on Pinebook Pro exhibits, and it's probably just some property of improved power management.

The fixes are in this branch: https://megous.com/git/linux/log/?h=wifi-5.10

2020–11–30: Improved ANX7688 driver and increased I2C speed

I've looked again at the mud stained hair ball that is ANX7688 firmware and extended my driver to better work with the USB-C docks when the docks are powered. The driver now works better also with other USB-PD enabled devices.

The changes mostly relate to making sure Pinephone will not overload a PD charger or a dock when powered from said dock. USB-C, USB-PD, and BC 1.2 specs specify various methods for determining the the current that can be used by the device, and it's quite complicated to put it all together reliably.

The approach the driver takes now is to wait for 3s after the cable is plugged in, and if no power source announces itself via a PD message, it will fallback to determining the maximum input current from the resistors connected to the CC pins. If the power source announces itself via PD message, the driver will wait 0.5s after the ANX7688 FW forwards the announcement and ramps up the input current limit to the one hopefully negotiated with the power source by the ANX7688 firmware.

After quite some testing, this approach seems to work for most of the USB-C devices and their combinations that I have at home, except for:

Most of the new patches are available here https://megous.com/git/linux/log/?h=anx-5.10

I've also bumped the I2C bus speed to 400kHz, which should make communication with I2C devices faster. This may be noticeable when using the touchscreen (~4ms shorter input latency).

One last thing I've changed was configuring PMIC via p-boot to limit power consumption, so that VBUS voltage never drops below 4.5V. This does away with USB-PD hard resets I've seen when using my USB-PD 5V/3A charger with Pinephone. USB-PD controllers probably do some VBUS voltage monitoring (ANX7688 does for sure), and try to recover when VBUS drops bellow certain threshold.

The patch is in p-boot for now: https://megous.com/git/p-boot/commit/?id=a80cac6bdea170ddeaee9475c976ba76558839ed

With this I could charge my Pinephone from USB-C PD charger without a single hard reset. Previously there were many.

2020–11–25: New multi-distro and p-boot released, WiFi power savings

I've put together a new version of the multi-distro image. More details and what's new can be found on https://xnux.eu/p-boot-demo/

p-boot also got a new release with some sought after UI improvements. Menu is more easily navigable thanks to wraparound and volume keys auto-repeat. :)

WiFi power savings on PinePhone

I've found a very recent rtl8723cs driver in some rockchip kernel tree on github. The driver is just slightly more than a month old, and I was able to enable power saving in it. This saves 350mW when the WiFi is idle, without any negative side effect that were present with the old driver.

The new driver is integrated into my 5.10 kernel.

2020–10–31: New p-boot released, new pre-built/updated kernels

New p-boot is released. It has the 60 FPS fix as mentioned previously.

It also has some usability improvements (it shows battery status in the footer and information about where the p-boot was booted from and from where it loaded the boot filesystem as these can mix and match quite a bit). It will also report if the battery is being charged or not.

When the phone is off and you insert the charger, this will normally lead to powering up the phone. It's not avoidable. p-boot will detect this situation and force the display of the menu followed by a regular 10s poweroff timeout, regardless of any default boot option setting. This makes it easy to connect a turned off phone to a charger without booting the OS.

If you run my multi-boot SD card image, you can update it with the new p-boot to get a reboot to eMMC feature, so that you don't have to remove the SD card to boot to your normal eMMC OS, that came with the phone.

New kernel releases

I've pulled in Martijn's front camera auto-focus support into my kernel branches – both 5.10 and 5.9. I've also added the 60 FPS fixes to both, too.

Release notes/NEWS are part of the new tags I'll be making from now on in my git repos each time there's a new kernel release/update from me:

Updating p-boot

You can find some notes on how to update multi-distro's p-boot on pine64 forum:

https://forum.pine64.org/showthread.php?tid=11347&page=8

2020–10–29: New p-boot features – reboot to FEL/eMMC, 60FPS fix

The most common request I get for p-boot is to make it possible to skip the boot from SD card and boot from eMMC.

I decided to implement this along with an option to reboot to FEL, and it works now. :)

I'll make a new p-boot release soon.

I've also finally found the fix for the PinePhone LCD panel issue where the panel was running at wrong refresh rate of 36.6Hz. With the fix, it run at expected 60Hz.

With panel running at 60Hz watching 30 or 60 FPS video should be possible without annoying stuttering, that was there previously.

I've made the fix for this available both in p-boot and in my kernel.

I've also implemented HW accelerated scaling in p-boot, that allows to do flashy animations, and use of smaller than full-res textures.

With all these new features available, it's probably time to look at making a new release of my multi-boot image. It will be interesting to see how smooth things are at 60 FPS. :)

2020–10–12: Backlight changes when swithcing between USB-C power modes

I noticed that backlight brightness changes significantly when connecting a VBUS powered device to Pinephone 1.2, like USB-C dock or USB-C hub.

I tracked down the issue to a VBUS_CTRL signal that ANX7688 provides to control VBUS power direction. This signal is connected to AXP803 to tell it to not try to charge from VBUS, when the VBUS is generated from the battery. (Perpetuum mobile is still not a thing, even in 2020)

Unfortunately this signal is also needlessly connected to PL9 pin of the SoC. VBUS_CTRL is 3.3V signal, and PL port is 1.8V port, so this is a rehash of WiFi chip IO port voltage issue, that was fixed some time ago, that was also causing backlight instability, previously.

In this case, there's no SW fix for this issue. But I've informed TL Lim, and was told that HW team will still manage to fix the issue for the Manjaro edition. The fix is to remove a 0 Ohm resistor that connects PL9 to VBUS_CTRL.

Compared to WiFi IO voltage overdriving the PL port, this issue is less annoying, because VBUS_CTRL is stable. So you'll have lower brightness when Pinephone sources power to USB-C device and higher brightness when it sinks power, but the brightness will not vary. I guess some power will also be saved, because overdriving the PL9 probably causes some small amount of power to be lost needlessly.

If you like DIY fixing your Pinephone, you can fix existing 1.2 devices by removing R1318 resistor. This should be the easiest of the HW fixes so far. ;-) Pinephones 1.0 and 1.1 don't have this issue.

2020–10–05: Reboot to FEL, skipping SD card boot, external display fixes

I've noticed that A64 BROM (Boot ROM) contains a special code path for booting via a specially formatted image that can be stored in SRAM, if some registers in CPUCFG range contain a magic value and an address to the image.

I prepared the image that would toggle Pinephone LEDs and hang, and a loader program that would load the image to SRAM A1, setup the magic registers, and reset the SoC.

Well it didn't work. :)

What probably happens is that the magic registers are reset when SoC is reset.

I did all this with hope that this could be used to reboot to FEL from Linux.

Alternative options to achieve reboot to FEL from Linux require cooperation from the bootloader. For example Linux can set some value to one of the RTC data registers, and bootloader will detect it early on and jump back to BROM FEL (allowing to use USB to boot the kernel/initramfs).

One other fun thing that can be tried would be to jump into the middle of the BROM code, right after the SD card boot code, in effect skipping the SD card boot and making BROM boot from eMMC. Let's say p-boot would boot into GUI menu, and if user would choose to boot from eMMC, p-boot would write a magic value to RTC data register and reboot. p-boot would get loaded again by BROM from SD card, and it would check the flag in RTC data registers, and based on it it would jump to eMMC boot code in BROM, forcing BROM to try booting from eMMC.

This would be a fairly cheap implementation of skipping boot from SD card, while having SD card inserted.

External display fixes

With help from ollieparanoid, I noticed that whether external display works in my kernel or not depends on whether the user uses p-boot display mode or not (p-boot without display support, or without splash screen, or u-boot).

I fixed this issue. It related to what parent the MIPI-DSI clock had on boot.

There's still one issue where external display only works if the phone is booted with dock connected but not when the dock is first connected after boot. That one still remains to be investigated.

The symptoms are that TCON1 clock is not enabled when the dock is connected only after boot, for some reason.

After this is fixed, the use of external display should become much more reliable.

Hopefully, I'll manage to figure out the fix prior to 5.9 release next week.

2020–09–29: New codec driver in my 5.9 kernel

Samuel updated his codec patches a few days ago, noting in the dev chat:

I have a branch ready for anyone wanting to test the new audio patches: https://github.com/smaeul/linux/commits/pine64-5.9 three notes:

  1. Set PulseAudio to use a 48 kHz sample rate and an 8 kHz alternate rate. Both must be multiples of 8 kHz or modem/BT audio won't work.
  2. You must use a QDAI config with the codec as master (modem as slave) and an 8 kHz sample rate. Use AT+QDAI=1,1,0,1,0,0,1,1 unless you have a very good reason not to.
  3. You must use a BT config with the codec as master (BT as slave) and 2 PCM slots. Use len=2 offset=00f6 { 85 10 }.

This does include jack detection and headset mic button detection. I don't have any three-button headsets, so I can't test the volume up/down functionality. it has the „80% solution“ to call audio quality – it will sound fine unless something tries to play/record audio from/to the CPU at a sample rate >8 kHz during a call

To clarify note 1 above: you can use both 44.1 kHz streams (music) and 8 kHz streams (modem), but not at the same time. so unless you have some way to kill all 44.1 kHz streams before starting a call (can UCM do that?), the sample rates have to be compatible

And some time later:

I pushed some more changes, including a fix for the high-pitched whine from the speaker it was a harmonic from the audio PLL, and I believe it's leaking from VCC-PLL → AVCC, since the noise is the same on the headphones and speaker.

So I made two changes:

  1. triple the frequency of PLL_AUDIO, which needed to be done anyway for using the sample rate converter. now the harmonic at 44100 (previously the whine) is inaudible, and the harmonic at 48000 is lower frequency, so less annoying
  2. decreased the PLL bias current, which drastically decreases the volume of both the harmonics and the white noise if you want to try out a different series of frequencies (1×, 4×, 6×), adjust the last two hunks in https://github.com/smaeul/linux/commit/7b51d5b09136

So I spent some time pulling the new codec patches to my kernel and rebasing my patches for X-Powers AC100 codec support on top of them, so that I can merge them into my kernel.

This went fine, and the results are as described by Samuel. Modem audio is nice and smooth, as long as you don't use alsa PCM interfaces during a call with any other sample rate than 8kHz. And the new jack detection works too.

One issue that people may hit with the new codec support is that some ALSA controls were renamed. I made the list of changes from the code diffs:

#!/bin/sh

sed -i 's/AIF1 AD0 Mixer AIF1 DA0 Capture Switch/AIF1 Slot 0 Digital ADC Capture Switch/' "$1"
sed -i 's/AIF1 AD0 Mixer AIF2 DAC Capture Switch/AIF2 Digital ADC Capture Switch/' "$1"
sed -i 's/AIF1 AD0 Mixer ADC Capture Switch/AIF1 Data Digital ADC Capture Switch/' "$1"
sed -i 's/AIF1 AD0 Mixer AIF2 DAC Rev Capture Switch/AIF2 Inv Digital ADC Capture Switch/' "$1"
sed -i 's/DAC Mixer AIF1 DA0 Playback Switch/AIF1 Slot 0 Digital DAC Playback Switch/' "$1"
sed -i 's/DAC Mixer AIF1 DA1 Playback Switch/AIF1 Slot 1 Digital DAC Playback Switch/' "$1"
sed -i 's/DAC Mixer AIF2 DAC Playback Switch/AIF2 Digital DAC Playback Switch/' "$1"
sed -i 's/DAC Mixer ADC Playback Switch/ADC Digital DAC Playback Switch/' "$1"
sed -i 's/AIF3 DAC Playback Route/AIF2 DAC Source Playback Route/' "$1"
sed -i 's/AIF3 ADC Capture Route/AIF3 ADC Source Capture Route/' "$1"

exit

# and then manually:

AIF2 DAC Source Playback Route:
        "None", "AIF2 Left", "AIF2 Right" 
        "AIF2", "AIF3+2", "AIF2+3"        

AIF3 ADC Output Mux:
        "None", "AIF2 Left", "AIF2 Right" 
        "None", "AIF2 ADCL", "AIF2 ADCR"  

Remove "AIF3 Loopback Switch"
       "AIF1 Loopback Switch"
       "AIF2 Loopback Switch"


##########################
# Direct mapping from old to new names (semantics are the same):
##########################

- "AIF1 AD0 Mixer AIF1 DA0 Capture Switch"
+ "AIF1 Slot 0 Digital ADC Capture Switch"

- "AIF1 AD0 Mixer AIF2 DAC Capture Switch"
+ "AIF2 Digital ADC Capture Switch"

- "AIF1 AD0 Mixer ADC Capture Switch"
+ "AIF1 Data Digital ADC Capture Switch"

- "AIF1 AD0 Mixer AIF2 DAC Rev Capture Switch"
+ "AIF2 Inv Digital ADC Capture Switch"

- "DAC Mixer AIF1 DA0 Playback Switch"
+ "AIF1 Slot 0 Digital DAC Playback Switch"

- "DAC Mixer AIF1 DA1 Playback Switch",
+ "AIF1 Slot 1 Digital DAC Playback Switch",

- "DAC Mixer AIF2 DAC Playback Switch"
+ "AIF2 Digital DAC Playback Switch"

- "DAC Mixer ADC Playback Switch"
+ "ADC Digital DAC Playback Switch"

- "AIF3 DAC Playback Route"            "None", "AIF2 Left", "AIF2 Right"
+ "AIF2 DAC Source Playback Route"     "AIF2", "AIF3+2", "AIF2+3"

- "AIF3 ADC Capture Route"             "None", "AIF2 Left", "AIF2 Right"
+ "AIF3 ADC Source Capture Route"      "None", "AIF2 ADCL", "AIF2 ADCR"

- "AIF3 Loopback Switch"
- "AIF1 Loopback Switch"
- "AIF2 Loopback Switch"

You can run the above script to patch up your UCM profile or whatever. I've updated my call audio setup script, for the new controls too.

If you like to test new audio patches you can grab my kernel at usual places: https://xff.cz/kernels or at https://megous.com/git/linux/

Modem reset

It turns out that Quectel managed to break modem reset via AT+CFUN=1,1, while adding support for fast poweroff. This manifests itself by modem powering off instead of rebooting when fast poweroff is enabled.

I had to patch my modem driver to disable fast poweroff during a reset after QDAI update, so that QDAI updates work smoothly with the new QDAI necessary for the above mentioned codec changes.

2020–09–20: Downsizing the multi-boot image

If the multi-boot image doesn't fit your 8GiB uSD card, because it's a tad too big, you can downsize it a bit by using this script:

#!/bin/bash

set -e -x

mkdir -p m
L=`losetup -P --show -f multi.img`
mount -o compress-force=zstd ${L}p2 m
btrfs filesystem resize 7000M m
echo ",7000M" | sfdisk -N 2 ${L}
umount m
losetup -d "$L"

truncate -s $((128+7000))M multi.img

If something fails in the middle, you may need to recover by calling umount and losetup -d yourself. losetup -l can tell you if the image is still exported as a loop device, and which one.

The image will have the size of 7128 MiB after resize and this should fit more „8“ giga something uSD cards, as there are obviously some other giga units than gigabyte and gigibyte used by some manufacturers.

2020–09–20: Some ways to improve Pinephone safety

This is a follow up on some issues from the previous article. On surface, solutions to some of the previously presented issues can seem simple. Toggle a few registers in PMIC, and we're mostly done. Trouble is that safety mechanisms are barely ever triggered, by definition. Safety events occur rarely. That means that the mechanisms are not regularly tested, and it is not known that they work.

Also it's not clear which code's responsibility fixing the issues should be. Bootloader, or kernel, or userspace? Finally, there are a bunch of devils in the details, that complicate the upstreamability of any solution. And having fixes upstream is necessary to make sure they reach the most users.

Necessary minimum

Nevertheless, at least enabling some pre-existing PMIC functionality blindly is better than nothing, so that's precisely what I decided to do in p-boot. It's the easiest place to start resolving these issues for me personally, and for other p-boot users.

I fixed two issues:

I didn't measure the 3kOhm NTC used in Pinephone battery and third party batteries I bought. I just used a table from some random 3kOhm NTC spec on The Internet, that seemed like it could match. Hopefully it's close enough.

Trouble with the second fix is that it's a hard power cut-off, so data loss may occur when PMIC overheats. There are three fixed temperature levels in AXP803. On level 1 the charging is limited, on level 2 the interrupt is sent to SoC, on level 3 the PMIC shuts down if configured to do so (by default it keeps running, and this is what my p-boot fix changes). Ideally, the crossing of level 2 would be handled by Linux to make it safely shut down the system, and level 3 forced power cut-off would never happen. Arguably, if charging was source of the heating, crossing level 1 will lead to resolving the issue, so the next level will not be reached.

Suggested fixes elsewhere

These fixes will reach a very limited audience. It would be nice to have these fixes in U-Boot too, but that's not possible at the moment, because U-Boot doesn't have access to PMIC.

Other places to put the fix is to ATF or Linux kernel. That can reach more people faster, but there would have to be some generic mechanism to make the fix upstreamable, otherwise it will not reach people using the mainline Linux kernel or mainline ATF.

There are some ways to approach this:

First, the most generic solution would be to have a description of the battery in DT describing the Pinephone. Sadly, the current bindings don't include battery temperature limits.

Also converting from temperature to NTC resistance (which is necessary to determine the code word from ADC for the limits used by the temperature monitor logic in PMIC) is not straightforward. It is usually defined in NTC datasheet as a table. Do I have NTC datasheet? No. I bought the batteries online from some mobile phone service shop.

There are also equations that approximate the temperature – resistance relationship for the NTC, which could be used instead of a fixed table, if one knows the relevant coefficients. These can be calculated after measuring the NTC's resistance at a few temperature points when we lack the datasheet.

So generic solution may look like this:

Kernel also has support for NTC devices, so maybe NTC can be described outside of the battery node (even though it's part of the battery).

This may all fail to be upstreamed on one thing: the battery is user swappable, so it's arguably not part of the Pinephone, and describing it inside the pinephone DT will not be appropriate.

I don't have any plans implementing any of the above, atm. Maybe with the exception of adding a 4th approach to the fix to my Linux kernel (the easiest one ;)). I'd like to work on my multi-boot image. So these are mostly pointers for somebody else who'd like to tackle this.

Other issues

Fast charging is not necesary in many situations, so having it as a default is not great. User should be able to decide if he wants to trade off slower, safer charging and battery longetivity over speed. This tradeoff can be realized in many ways.

All this is already controllable from userspace via sysfs. Ideally there would be some charging monitoring daemon that would take into account users's wishes and select proper strategy for charging, based on preference for battery longetivity or speed.

There are several trade offs the deamon would be able to handle:

All this is decision making that doesn't belong to the kernel.

Similar daemon could monitor power usage of the phone and try to limit it to safer levels, or warn the user if that's not possible.

2020–09–18: Let's talk about safety of Pinephone

My gf read me some articles about exploding phones today. :) I think there needs to be some serious conversation about Pinephone safety. Safety needs to become an important concern now, when more and more people are getting their Pinephones every month. It's just a matter of time before the first major safety incident hits this community, and it may be more than just a hacked store. It's just a numbers game.

Pinephone is an interesting device in one way. You can run whatever software you like on it (and you do!), and this software comes almost universally with zero guarantees. Read the license to any of the program you run on your Pinephone and it will almost certainly tell you:

THIS SOFTWARE IS PROVIDED ``AS IS'' AND WITHOUT ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.

or

THIS SOFTWARE IS PROVIDED BY <COPYRIGHT HOLDER> AS IS AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL <COPYRIGHT HOLDER> BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES

or

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

etc.

In case of Pinephone you have to take these warnings very seriously, because this software is not provided by the manufacturer (Pine64), and as far as I know, there's no software related safety testing going on at all.

Some skeletons, hiding at the lower levels…

I'll give you a few reasons why things may not be so rosy, when it comes to safety.

There's no unchangeable well tested guardian angel management engine that safely manages battery, power supplies, thermal behavior, that is provided by the manufacturer, and that is independent of the operating system.

Pinephone's SoC is quite bare when it comes to software/firmware (that's why FOSS enthusiasts like it, no blobs, you know!). This has a dark side, too. All the safety critical parts are written (or rather were not written, yet) by some random people on The Internet.

You can already choose among more than 10 Linux distributions to run on your Pinephone. How do you know any one of these is safe to run on your Pinephone?

How well safety critical parts of Pinephone function (or if at all), depends on how well Linux distribution vendors understand the platform, and how well they test its safety features. Pinephone safety depends on the software they do put together, afterall.

Is device safety even a thing distribution vendors test for or plan for at the moment? I don't know, but I doubt it.

For many months pretty much all distributions used misconfigured „official“ kernel (it's a meaningless moniker, btw, pine64 doesn't do software), that didn't regulate CPU temperature at all. It could go up as high as the thermals of the surrounding environment allowed. There were indications something is wrong. People reporting their phones felt too hot, displays showing burn artifacts, temperature indicator in sysfs returning error. Regardless, it persisted for many months, until I discovered the root cause while helping one user with thermal issues on her Pinephone.

Maybe I'm a bit biased, because I was one of the people working on the sunxi thermal driver over the years, and thus understand this element of the SoC quite well, but I was surprised nobody figured out such a serious issue, or realized what it meant. So let's go through some other issues that I'm aware of, of which other people may not be:

1. Pinephone battery uses a 3 kOhm NTC to monitor the temperature. Power management chip in Pinephone expects 10 kOhm variant by default. So early on, when the times were adventurous, someone decided to patch the kernel to disable battery thermal monitoring completely. Quick and dirty fix for Pinephone not charging due to false under-temperature alarm.

Now guess what… up to now, all distributions run with battery temperature sensing and regulation disabled. If you're unlucky and use a dud battery that will heat up more easilly during fast charging, you can burn down your house.

There will be nothing to stop it going past 50, 60, 70, 600°C. Phone will happilly provide current to the battery until it explodes and burns. Unlikely scenario for any single user, but this safety mechanism that's present on regular customer phones is missing here, just because nobody cared to configure it, yet.

2. And you know what, I also suspect that all distros fast charge all the way through the constant-current phase of charging, by default. Other than contributing to making the above mentioned house burning scenario more likely to happen, this also contributes to overheating issues on Pinephone.

3. Another thing. PMIC has an emergency thermal shutdown feature, for a situation when the chip itself overheats. It's disabled by default. It's also not well documented. ¯\_(ツ)_/¯

4. And, another one! Battery is rated for some sustained continuous discharge current (0.5C, 1500mA, ~6W). I guess it overheats if it is dicharged at a significanlty faster rate for prolonged periods of time, and potentially becomes a safety hazard again. Maybe again only if it's a suboptimal piece that passed QA.

Now, Pine64 sells convergence edition Pinephone meant for use with a dock and a monitor. When you connect the phone to the dock with monitor connected over HDMI, power consumption jumps by 4W. That alone is close to this specified limit of the battery. Baseline power consumption is 2.8W (with phone display on). Now if you actually use the phone the CPU alone can add further 2.5W during full load (likely, when trying to use the pinephone as a desktop machine). That is >9W total. That is well over the specified safe limit. Of course, the rest of the phone heating up a lot from this huge power draw don't help the at all, either.

So it's quite possible for the user to load the battery well out of spec, with just normal use. This needs to be managed somehow.

And those are just things that I'm aware of.

So what…

I'm trying to be a bit inflamatory here, to start the conversation. Nevertheless, the above issues are real. It's really just a numbers game. When the distros will not take safety seriously, by planing for it, testing for it, and verifying the mechanisms they are supposed to ensure are in place for the safety of their users, the odds somehting will happen will stay needlessly high.

Also, don't be a person whose house burns down for FOSS. Ask your favorite distribution's authors what they're doing to make their OS safe. Pine64 itself can only go so far to ensure safety of Pinephone, software you put on your Pinephone matters a lot, too!

2020–09–17: Video acceleration experiments with PinePhone

Recently, gstreamer gained support for utilizing v4l2-requests API, for h264 acceleration, so I compiled it for Arch Linux, patched the kernel with the help of ndufresne from #cedrus to fix kernel panics, figured out how to make gstreamer play to kms, and made a bunch of experiments with these results:

https://www.youtube.com/watch?v=dHOgVmxH_dA

https://www.youtube.com/watch?v=6P76lQX70iI

https://www.youtube.com/watch?v=yyRm8kccyG4

External display video playback, even to a fairly large display works very nicely. Quite a feat for an entry-level phone. :)

CPU utilization during playback is miniscule. Comparing power consumption, between idle and video playback, the video decoding takes just additional 0.2W of power, compared to CPU based decoding which could take up to 2.4W.

There's an issue with internal display being locked to 36 FPS, which makes playback of 30FPS video a bit choppy.

I also hacked my kernel to workaround gstreamer kmssink bugs, to test playback to an external 1440p@60fps monitor. 1440p@30 video decoding works fine. 1440p@60 video is too much, and drops frames. 1080p@60 video decodes and scales to 1440p fine, too.

Another issue is a heavy power consumption (4W) from the dock when HDMI output is enabled. Also dock's display output doesn't seem to work when powering the dock from the USB-C power supply.

Overall, it's nice. :)

2020–09–16: PineBook Pro and Levinboot again

Crystalgamma released Levinboot 0.7.2 yesterday so I decided to update, and try to make my Pinbook Pro boot times even faster. With his help I found that I accidentally disabled optimizations in my build and that was the reason for 4s decompression times for my kernel/initrd payload, and some stack overflows in levinboot's decompression code.

Unoptimized code turns out to be really slow. What a surprise.

I switched to lz4, enabled -O3, and now levinboot decompression and load times changed from 4s to 600ms. Significantly better! :)

With the latest levinboot my Pinebook Pro Arch Linux setup lets me to enter LUKS password in 4s since I start pressing the power button:

Perfect!

2020–09–14: Putting 13 PinePhone distributions on a 8GiB uSD card

This is a summary of how I was able to make a 5 GiB multi-boot 13-distro image, that easily fits on a 8GiB SD card, and can be used to test-drive most of the OSes that are available for Pinephone today.

Starting with a 32GiB image and an old-school approach

At first I did the obvious, and started with traditional partitioning. I created 10 partitions on a 32GiB SD card, and was barely able to fit 9 Linux distributions into that image. Each paritition needed its own filesystem and some free space to allow for proper operation of the OS. It was also hard to mange. Each time I needed to change anything, I had to figure out which partition held which OS, mount it, fix things and unmount again.

It was not good.

Swithching filesystems and sharing free space

The breakthroug came with a realization that I can mount subvolume in btrfs as if it was a filesystem in its own right. This allowed me to have just a single partition with a btrfs filesystem on the SD card, and have each distribution contained in its own subvolume. This way, all distributions could share from the same pool of free space for the data. There was no longer any need to plan partition sizes for each distro. This was the first major win.

Subvolumes are also much easier to manage than partitions. Need a new subvolume? Just create one and copy some files to it. Don't need the subvolume anymore? Just delete it.

Subvolumes appear as normal directories in the filesystem when the root subvolume is mounted, so one mount operation is all that's needed to have access to files of all included Linux distributions.

I also used another feature of subvolumes to debug startup issues in some distributions. Snapshotting. It's possible to create a snapshot of a current state of any of the distributions indivudally, and restore it in the future. Ubuntu Touch gave me a headscratch, when it didn't boot during the first boot, but it did on the second one. So I made a snapshot of the initial state of the filesystem, and kept comming back to it and tweaking it until I found the reason. (One of the boot scripts checked for the presence of a file named userdata/.writable_image and if it was missing it tried to reboot the system and failed, crashing the lightdm process.)

In fact, the latest multi-boot image keeps the initial state of all included distributions in a snapshot, so it's possible to selectively restore state of any of the distributions to the original.

Compression

Looking at btrfs, I found that it supports transparent compression for the stored files. Enabling zstd compression and decompressing rootfs tarballs of 9 distributions resulted in a filesystem that used 5.8 GiB of space with a compression ratio of 50%. This was exciting, because it looked like having a 8 GiB SD card image will be possible.

Using compression is also good for antoher reason. SD card access in Pinephone is limited to 24MiB/s max. At this speed, 4 core Cortex-A53 CPU can easily keep up, and this makes the loading of data from the SD card faster.

Some interesting features of COW filesystems

So far the optimization steps were fairly trivial. Just enabling features of existing filesystem and using them in a smart way.

Getting from 5.8GiB image with 9 Linux distributions to a 5 GiB image with 13 Linux distributions was harder. It was likely that those distributions have a lot of files in common. I made a simple test, and found that there's a space for further savings in the range of 12–13% if I could make the filesystem share data for duplicate files among the distributions.

COW filesystems, like btrfs, can sometimes allow to share file data among multiple files without resorting to hardlinking. In fact, there's now a generic VFS-level Linux API to make a filesystem share file data between files if it supports this feature.

Side note: Hardlinking would not be a great solution for reducing file data duplication in the image anyway, because it would have meant that if user changed a file in Ubuntu Touch, the change might be visible in other distros that may share that file.

The next hurdle was figuring out how to effectively use this API in my specific scenario. There are tools that allow scanning the existing filesystem, search for duplicities and use this API to remove them. Then it would be possible to scrub the filesystem, discard empty space, and compress the resulting block device image for easy re-distribution. I checked a bunch of those tools, and they seemed exceedingly complicated, or buggy when used on btrfs subvolumes.

Instead I decided to write a extraction tool using libarchive that takes multiple tarballs on the input (one per Linux distribution) and decompresses them while keeping track of content of already extracted files and using the above mentioned FICLONE API to share data whenver it finds file that was already extracted before. It does deduplication on the fly, so there's no need to apply any further cleanup steps to the filesystem afterwards. This approach to deduplication is also very fast and efficient, because there's no extra IO necessary.

At this point I tried to add 4 more Linux distributions to the image. I did this just to stress test the new extraction tool. In particular, I added several variants of postmarket OS, which were bound to have a lot of the files in common.

The resulting image had the same size as the previously made 9 distribution variant.

Final space savings

The final opportunity for size optimization was a cheap one. Remove files that are not used/needed. I didn't want to tweak the distributions too much, so that they are as close to their official state as possible. I had to make one change, though. I was not able to use distributions' own official kernels, because neither had a driver for btrfs built in. I also was not very fond of supporting outdated EOLed kernels many of the distributions use, or re-building them. So I used my own kernel and made all the distributions share it.

Due to this I was able to remove the modules, firmware, and kernels that distributions package themselves. With 13 distributions this led to another 0.8GiB of saved space.

The result is a 5 GiB 13-distro image, that easily fits on a 8GiB SD card, and can be used to test-drive most of the OSes that are available for Pinephone today.

The image also serves as a demo of the GUI bootloader I wrote for Pinephone.

2020–09–11: Adding postmarket OS to multi-distro image

So I finally tried the pmbootstrap tool to create a bunch of rootfs tarballs for several postmarket OS UI variants: GNOME, Phosh, Plasma Mobile, fbkeyboard.

I've added them all to my multi-distro image without any increase in size since the previous public build of the image. So now it's a 13 distro multi-boot image with pretty much every major and a bunch of minor PinePhone supporting distributions, and it still fits on a 8GiB SD card!

2020–09–11: Ways to help improve Pinephone kernel

There are a bunch of open questions about Pinephone's HW. When answered, these could help improve the kernel behavior. Here's one of them.

Minimal display brightness is too bright

This gets asked quite a bit, but just maybe 2 people helped do anything about it, so here's a more detailed guide on how to help.

Sadly fixing this issue requires gathering a lot more samples than 2, regardless of whether your Pinephone has the problem or doesn't. Trouble is that if only people who have this issue do the following test, the results will be useless. Even people who's backlight seems to work correctly need to take part. So it's a hard problem to solve. :)

If enough people report at what brightness level the backlight turns on on their Pinephone, we could improve the default minimum brighness level, so that most users can reduce their Pinephone screen brightness to lower levels.

There's an open question on how to set the backlight brightness values on post 1.0 revision phones, since lower PWM duty cycles lead to backlight being basically off. It would be nice if more people can test the various backlight levels on 1.1 and 1.2 revision with this change in dts in the backlight node:

/backlight {
        brightness-levels = <0 1000>;
        num-interpolated-steps = <1000>;
};

The above change can be made by fdtput, which is part of dtc package:

fdtput -t u your.dtb /backlight brightness-levels 0 1000
fdtput -t u your.dtb /backlight num-interpolated-steps 1000
fdtput -t u your.dtb /backlight default-brightness-level 500

You don't need to recompile the kernel or anything. Just identify what dtb file is your OS using (it will be in /boot directory) backup it up and apply the above commands to the original (just replace yout.dtb with the name of your dtb file).

This change will make it so that PWM backlight driver sets up 1000 linear backlight steps, so you can enter values 1–1000 to /sys/class/backlight/backlight/brightness and these are directly mapped to PWM duty cycle. So level 1 is 0.1% level 100 is 10%, and so on.

What's improtant is to report:

2020–09–11: PinePhone multi-boot image deduplication tool complete

Since the last post, I was able to create the deduplicating tarball extractor tool exactly as I described it, and everything seems to work as expected.

The image size after compression shrank from 5.8 GiB to 5.2 GiB. :)

Now I have to figure out how to build a few different postmarket OS rootfs variants, to give this shiny new tool a real stress test.

I'm pretty sure I can add maybe 3 postmarket OS mobile UI variants without increasing the image size significantly. After all, the image already contains sxmo, which is based on pmOS, so most of the base data is already in the image.

Then, I'll implement p-boot DTB auto-selection, and the image will be ready to enter beta state.

New kernels

I've released new 5.9 kernels with a few more new tweaks:

2020–09–10: PinePhone multi-boot image deduplication

Btrfs has a nice feature where you can tell it when creating a file to initially use content of some already existing file. This saves space on the filesystem, because the duplicit files can share the same data space without needless data copies. This is almost like hard links, except that data stop being shared, when the file is written.

This can be used in the multi-boot image to save even more space. The multi-distribution image offers many opportunitites for this kind of space saving, due to distributions sharing quite a lot of identical files.

I've written a tool that takes all the rootfs tarballs on input, and goes through all of them, finds duplicities, and calculates space used in total by all files, and space used by unique files. For the current multi-distro image, the numbers look like this:

So the needless overhead of duplicate files accross all included distributions currently is 12.6%, btrfs compression ratio is 50% and the wasted space on the compressed filesystem by the duplicate files is 762 MiB.

That's fairly significant. I can fit another distribution in the wasted space alone. :)

If the included distributions would be much less diverse. For example when including multiple UI variants of postmarket OS, the savings could be even greater. I can potentially add 3 to 4 variants of pmOS if I was able to share duplicate files content, without the current image size changing at all.

I'll try to add FICLONE support to my extraction tool. The tool will basically put content hashes of the files and their paths into a hash table as they are being extracted, and if there's a duplicity, instead of extracting the content of a file, it will use FICLONE to share the data of previously extracted copy of the file.

Resizing the partition after flashing the image

To get more free space when using the multi-image, you can resize it to the full size of your SD card. You can do this from a PC using a SD card reader.

Warning! You'll need to run all the commands as root.

Multi-distro main data partition block device can be found with:

blkid -lt PARTUUID="12345678-02"

For me it prints:

/dev/mmcblk0p2: UUID="0cb50b0b-77a3-45bd-a605-857472b88281" \
   UUID_SUB="a1d75445-1ffe-4e6f-860a-1acdbb4aad2a" \
   BLOCK_SIZE="4096" TYPE="btrfs" PARTUUID="12345678-02"

So for me, btrfs partition is /dev/mmcblk0p2 and the device containing the partition table is /dev/mmcblk0. For you it may be different. Make absolutely sure you're using the correct block device!

Resize the main partition

First, dry run:

echo ", +" | sfdisk -n -N 2 /dev/mmcblk0

Outputs:

Quote:
Disk /dev/mmcblk0: 29.74 GiB, 31914983424 bytes, 62333952 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x12345678

Old situation:

Device        Boot  Start      End  Sectors  Size Id Type
/dev/mmcblk0p1 *      8192  262143  253952  124M 83 Linux
/dev/mmcblk0p2      262144 15359999 15097856  7.2G 83 Linux

/dev/mmcblk0p2:
New situation:
Disklabel type: dos
Disk identifier: 0x12345678

Device        Boot  Start      End  Sectors  Size Id Type
/dev/mmcblk0p1 *      8192  262143  253952  124M 83 Linux
/dev/mmcblk0p2      262144 62333951 62071808 29.6G 83 Linux
The partition table is unchanged (--no-act).

Now, verify changes look sane, and do a real resize:

echo ", +" | sfdisk -N 2 /dev/mmcblk0

Resize the main filesystem

Now that the partition is stretched to the end of the SD card, let's resize the contained filesystem too. Btrfs needs to be mounted during resize. The steps thus are:

mount /dev/mmcblk0p2 /mnt
btrfs filesystem resize max /mnt
df -h
umount /mnt

(Make sure when substituting your device for /dev/mmcblk0p2 that you keep the partition number at the end – p2. Thanks to Gazza for pointing out this pitfall.)

You should see:

Resize '/mnt' of 'max'
/dev/mmcblk0p2  30G  6.6G  24G  23% /mnt

And that's all. :)

Forum post

I've posted about my image on the pine64.org forum. That post contains some additional information:

https://forum.pine64.org/showthread.php?tid=11347

2020–09–08: PinePhone multi-boot finishing touches / modem improvements

So I dealt with a few of the „annoying details“ from the last post, and created a repository for my build scripts and overrides for each distro.

The last thing to do prior to final testing is to add support for selecting dtb for PP 1.1 or 1.2 automatically.

Modem power manager

I've also improved my modem power manager a bit, and added support for fast poweroff mode that's available in the newer modem firmwares. (I believe it's those firmware that are in Braveheart and onwards.) Modem can now be powered off in ~2 seconds.

I've also fixed issues with RDY not being received. Looks like one of the persistent configuration options makes the modem not send RDY. I just poll for the successfull empty AT command, as a way to detect when the modem is ready, and that seems to be more reliable.

All these changes are avialable in my 5.8 and 5.9 kernel branches.

2020–09–08: PinePhone multi-boot image optimizations

Since I'll be using my kernel for all the distros anyway, I thought I might spice it up a bit, and use something a bit more modern than ext4.

Instead of complicated partitioning scheme that wastes space and is not easy to optimize, I decided to use a single btrfs filesystem for everything with each distribution having its own subvolume, and all files being compressed by zstd. This made it possible to create a 8GiB SD card image, with 9 distributions on it.

The image creation process is now much simpler. First I create a distros/ directory with a subdirectory for each distribution. Then I download and prepare rootfs.tar.zst tarballs, for each distribution I want to have in the image, as previously described in this blog.

And then I create the image:

#!/bin/sh

if [ "$(whoami)" != "root" ] ; then
        exec sudo sh "$0" "$@"
fi

set -e -x

. ./config

rm -f $IMG
truncate -s $IMGSIZE $IMG

sfdisk -W always $IMG <<EOF
label: dos
label-id: 0x12345678
unit: sectors
sector-size: 512

4M,124M,L,*
128M,,L
EOF

L=`losetup -P --show -f $IMG`

mkfs.btrfs ${L}p2

mkdir -p m
mount -o compress-force=zstd:15 ${L}p2 m

for ddir in distros/*
do
        test -f $ddir/config || continue
        name=${ddir#distros/}
        btrfs subvolume create m/$name
        bsdtar -xp --numeric-owner -C m/$name -f $ddir/rootfs.tar.zst
done

./mkimage-apply-fixes.sh

umount m
losetup -d "$L"

./mkimage-boot.sh

This does away with the need for having to deal with partition numbers. Subvolumes have the name of the distro, and are easily referenced in the boot arguments.

During boot I just mount the appropriate subvolume instead of the main volume of the btrfs filesystem. All that's needed is to pass rootfsopts=subvol=distro-name.

Modifying and accessing files in all distributions at once also becomes trivial. All I need to do is to mount the main volume, and the subvolumes for each partition appear nicely available there as subdirectories in the main volume. Previously I had to juggle with mounting 10 arbitrarily named partitions.

It's also very easy to add/remove distributions without the need for re-partitioning, and all distributions share the remaining free space on the SD card.

I find this setup excellent! Btrfs partition with 9 simultaneously installed distributions takes just 6.5GiB of space. Previously just mobian alone took 3GiB of used space, and a 4 GiB partition.

Making the image bootable

To make the image bootable, I just build the boot.conf file for p-boot, based on a per-distribution config file, and use it to format the boot partition.

Config file looks like:

version=2022-01-20
name="Ubuntu Touch"
bootargs="logo.nologo vt.global_cursor_default=0"

And the script to format make the image bootable:

#!/bin/sh

if [ "$(whoami)" != "root" ] ; then
        exec sudo sh "$0" "$@"
fi

. ./config

# loglevel=15
serial="console=ttyS0,115200 earlycon=ns16550a,mmio32,0x01c28000"
silent="quiet loglevel=0 systemd.show_status=false"
bootargs_base="$serial $silent cma=256M console=tty1 consoleblank=0 panic=3 rw rootwait root=PARTUUID=12345678-02 rootfstype=btrfs rootflags=compress-force=zstd,nodatacow,subvol"
kbuilds=../builds

echo "device_id = Distro Demo Image 2022-01-26" > boot.conf

no=0
for ddir in distros/*
do
        test -f $ddir/config || continue
        dist=${ddir#distros/}

        (
                . ./$ddir/config
                        
                echo "no = $no"
                echo "  name = $name $version"
                echo "  atf = ../p-boot/dist/fw.bin"
                echo "  dtb = $kbuilds/ppd-5.16/board-1.1.dtb"
                echo "  dtb2 = $kbuilds/ppd-5.16/board-1.2.dtb"
                echo "  linux = $kbuilds/ppd-5.16/Image"
                echo "  bootargs = $bootargs_base=$dist $bootargs"
                if test -f files/$dist.argb ; then
                        echo "  splash = files/$dist.argb"
                else
                        echo "  splash = files/xnux.argb"
                fi
        ) >> boot.conf

        no=$(($no+1))
done

# JumpDrive is special
if test -d distros/jumpdrive
then
        (
                echo "no = $no"
                echo "  name = Jumpdrive 0.8"
                echo "  atf = ../p-boot/dist/fw.bin"
#               echo "  dtb = distros/jumpdrive/sun50i-a64-pinephone-1.1.dtb"
#               echo "  dtb2 = distros/jumpdrive/sun50i-a64-pinephone-1.2.dtb"
#               echo "  linux = distros/jumpdrive/Image"
                echo "  dtb = $kbuilds/ppd-5.16/board-1.1.dtb"
                echo "  dtb2 = $kbuilds/ppd-5.16/board-1.2.dtb"
                echo "  linux = $kbuilds/ppd-5.16/Image"
                echo "  initramfs = distros/jumpdrive/initramfs"
                echo "  bootargs = loglevel=0 silent console=tty0 vt.global_cursor_default=0"
                echo "  splash = files/jumpdrive.argb"
        ) >> boot.conf

        no=$(($no+1))
fi

set -e -x

../p-boot/.build/p-boot-conf-native . boot-part.img
#zstd boot-part.img
#exit

L=`losetup -P --show -f $IMG`
../p-boot/.build/p-boot-conf-native . ${L}p1
losetup -d $L

dd if=../p-boot/.build/p-boot.bin of=$IMG bs=1024 seek=8 conv=notrunc

And that's really all there's to it, as far as making a bootable multi-distro image is concerned.

Some annoying details are still remaining

Support

If you'd like to support this effort, you can contribute at https://xnux.eu/contribute.html

2020–09–07: PinePhone multi-boot image boot testing

So let's say that distros are really finicky about where they're booted from if their initramfs image is used to boot them. Initramfs images mostly get in the way and don't offer much for the multi-boot image, so the simple solution is to not use them at all. All distros should be bootable fine without them.

The process should be quite simple: Linux mounts root partition provided via root= boot parameter at / and runs /bin/init. The distro itself should not care about what the real partition is, because it's already running from it. It should just not do anything stupid to the current rootfs mount, and just run.

Actually most modern distros should run fine with an empty /etc/fstab, in this situation.

Using my kernel

To have some unity/sanity, I decided to use my own kernel with all necessary drivers built in. This way I don't need to copy my kernel's modules to rootfs of each distro. I did also built firmware binaries into the kernel image, so it's all self-contained.

This approach has some other benefits:

Problems encountered so far

Other than this, each distro seems to boot fine with my 5.9 kernel.

The last step is figuring out how to make modem initialize properly on each distro. This will involve finding the script that powers up the modem and replacing it with echo 1 > /sys/class/modem-power/modem-power/device/powered.

See https://megous.com/dl/tmp/multi2.mp4 for another preview. :)

2020–09–05: PinePhone multi-boot image

I made a further progress on the multi-boot image:

The result is:

I have yet to boot test all the distributions. Prior to that, I have to check whether the distros are not doing something destructive on boot, like resizing or deleting partitions, etc.

So far my boot.conf for p-boot looks like this:

#!/bin/sh

#serial="console=ttyS0,115200 earlycon=ns16550a,mmio32,0x01c28000 loglevel=15"
bootargs="cma=256M console=tty1 consoleblank=0 quiet loglevel=1 panic=3 rw rootwait root=PARTUUID=12345678"

dists=/mnt/dists

#/dev/mmcblk0p2: PARTUUID="12345678-02"  mobian (f2fs, 5.7 kernel, PP 1.0-1.2)
#/dev/mmcblk0p3: PARTUUID="12345678-03"  KDE neon (ext4 only, 5.7 kernel, PP 1.1 only)
#/dev/mmcblk0p5: PARTUUID="12345678-05"  arch (f2fs, my 5.9 kernel, , PP 1.0-1.2)
#/dev/mmcblk0p6: PARTUUID="12345678-06"  sxmo (ext4 only, 5.7 kernel, PP 1.1, 1.2)
#/dev/mmcblk0p7: PARTUUID="12345678-07"  lune (ext4 only, 5.5 kernel, PP 1.1 only)
#/dev/mmcblk0p8: PARTUUID="12345678-08"  maemo (ext4 only, 5.6 kernel, PP 1.1 only)
#/dev/mmcblk0p9: PARTUUID="12345678-09"  ut  (ext4 only, 5.6 kernel, PP 1.1 only)
#/dev/mmcblk0p10: PARTUUID="12345678-0a" sailfish (ext4 only, 5.6 kernel, some custom DTB with partial support for up to PP 1.2)
#/dev/mmcblk0p11: PARTUUID="12345678-0b" pureos (f2fs, 5.8 kernel, PP 1.0-1.2)

zcat $dists/part2/boot/Image.gz > $dists/part2/boot/Image
#zcat $dists/part3/boot/vmlinuz-5.7.0-pine64-g3823929aa > $dists/part3/boot/Image
#zcat $dists/part8/boot/Image.gz > $dists/part8/boot/Image
#zcat $dists/part9/boot/vmlinuz > $dists/part9/boot/Image

cat << EOF
device_id = Multi-Distro Demo Image

no          = 0
  name      = Mobian
  atf       = fw.bin
  dtb       = $dists/part2/boot/dtb/allwinner/sun50i-a64-pinephone-1.1.dtb
  linux     = $dists/part2/boot/Image
  initramfs = $dists/part2/boot/initrd.img
  bootargs  = $bootargs-02 splash plymouth.ignore-serial-consoles vt.global_cursor_default=0
  splash    = files/mobian.argb

no          = 1
  name      = KDE Neon
  atf       = fw.bin
  dtb       = $dists/part3/boot/dtb
  linux     = $dists/part3/boot/Image
  initramfs = $dists/part3/boot/initrd.img-5.7.0-pine64-g3823929aa
  bootargs  = $bootargs-03 splash
  splash    = files/neon.argb

no          = 2
  name      = Arch Linux ARM
  atf       = fw.bin
  dtb       = $dists/part5/boot/dtbs/allwinner/sun50i-a64-pinephone-1.1.dtb
  linux     = $dists/part5/boot/Image
  initramfs = $dists/part5/boot/initramfs-linux.img
  bootargs  = $bootargs-05
  splash    = files/arch.argb

no          = 3
  name      = Sxmo
  atf       = fw.bin
  dtb       = $dists/part6/boot/sun50i-a64-pinephone-1.1.dtb
  linux     = $dists/part6/boot/vmlinuz-postmarketos-allwinner
  initramfs = $dists/part6/boot/initramfs-postmarketos-allwinner
  bootargs  = $bootargs-06 init=/init.sh PMOS_NO_OUTPUT_REDIRECT PMOS_FORCE_PARTITION_RESIZE pmos_root=/dev/mmcblk0p6
  splash    = files/sxmo.argb

no          = 4
  name      = Lune OS
  atf       = fw.bin
  dtb       = $dists/part7/boot/sun50i-a64-pinephone.dtb
  linux     = $dists/part7/boot/Image
  initramfs = $dists/part7/boot/initramfs-uboot-image-pinephone.uboot
  bootargs  = $bootargs-07 bootmode=normal LUNEOS_NO_OUTPUT_REDIRECT
  splash    = files/lune.argb

no          = 5
  name      = Maemo Leste
  atf       = fw.bin
  dtb       = $dists/part8/boot/allwinner/sun50i-a64-pinephone-1.1.dtb
  linux     = $dists/part8/boot/Image
  bootargs  = $bootargs-08 fbcon=rotate:1
  splash    = files/maemo.argb

no          = 6
  name      = Ubuntu Touch
  atf       = fw.bin
  dtb       = $dists/part9/boot/dtb
  linux     = $dists/part9/boot/Image
  initramfs = $dists/part9/boot/initrd.img
  bootargs  = $bootargs-09 systempart=/dev/mmcblk0p9 devnum=0 logo.nologo vt.global_cursor_default=0
  splash    = files/ut.argb

no          = 7
  name      = Sailfish
  atf       = fw.bin
  dtb       = $dists/part10/boot/sun50i-a64-pinephone-1.1.dtb
  linux     = $dists/part10/boot/Image
  bootargs  = $bootargs-0a
  splash    = files/sailfish.argb

no          = 8
  name      = Pure OS
  atf       = fw.bin
  dtb       = $dists/part11/boot/dtb/allwinner/sun50i-a64-pinephone-1.1.dtb
  linux     = $dists/part11/boot/Image
  initramfs = $dists/part11/boot/initrd.img
  bootargs  = $bootargs-0b init=/sbin/init splash plymouth.ignore-serial-consoles
  splash    = files/pureos.argb


EOF

2020–09–04: Pinebook Pro and Levinboot

Pinebook Pro is a very nice laptop. Software support just has a few annoying quirks:

Thankfully, a discord user CrystalGamma took to creating Levinboot, which is a specialized bootloader for Pinebook Pro and other similar RK3399 based computers. Its goals are almost identical to my p-boot project: boot the kernel as fast as possible from local storage and get out of the way. Its limitations are similar too. No or very limited support for partition schemes, no support for traditional filesystems, etc.

Version 0.7.1 was released a few days ago, so I decided to give it a try on my Pinebook Pro, and I'm happy to report that it works quite nicely, and it looks like it's going to have a bright future. :)

Boot process now shows a feedback. At first, red LED turns on for 1 sec, then green LED turns on for another sec, then red LED turns off and in another 2–3 seconds the tty shows up, and the system is interactive. That's with a small 7MiB gzip compressed kernel payload. I also tried a large 20MiB zstd compressed payload from eMMC, but that doesn't yet work as expected. I get load speeds of about 5MiB/s with that.

With lack of support for system suspend, fast boot times help make the laptop more useable.

My Levinboot setup

I use eMMC to load the payload, and SD card to load Levinboot itself. This way I can easily upgrade and test new Levinboot versions, and not worry about it breaking, because I just pop out a SD card, and update the bootloader from my PC, in case something fails.

To get Levinboot, I compile it from source code:

CROSS=aarch64-linux-gnu
export CC=$CROSS-gcc
export OBJCOPY=$CROSS-objcopy
export LD=$CROSS-gcc

git clone https://gitlab.com/DeltaGem/levinboot.git
cd levinboot
# currently contains fixes for reading GPT partition table
git checkout dev
./configure.py --payload-emmc --payload-zstd --payload-initcpio --with-tf-a-headers $ATF_DIR/include/export
ninja

# Copy levinboot-sd.img to my Pinebook Pro
scp levinboot-sd.img root@pbp:/boot/levinboot-emmc.img

Then I install it to SD card from Pinebook Pro itself, along with flashing payload to eMMC:

#!/bin/bash
set -e -x

ROOT_UUID=6795bcbb-fba8-4d22-81ef-dc628324b021

BOOTOPTS=(
  console=tty1
  video=eDP-1:1920x1080@60
  rd.luks.uuid=$ROOT_UUID
  rd.luks.name=$ROOT_UUID=root
  rd.luks.options=timeout=0,discard
  root=/dev/mapper/root
  rootfstype=f2fs
  rootflags=x-systemd.device-timeout=0
  rootwait
  rw
  mitigations=off
  quiet
  loglevel=2
)
BOOTOPTS="${BOOTOPTS[@]}"

cp -f board.dtb board-lv.dtb
fdtput -pt s board-lv.dtb /chosen bootargs "$BOOTOPTS"

truncate -s 0 payload-lv.img
zstd -zc bl31.elf >> payload-lv.img
zstd -zc board-lv.dtb >> payload-lv.img
zstd -zc Image >> payload-lv.img
zstd -zc initramfs.img >> payload-lv.img

dd if=payload-lv.img of=/dev/mmcblk2p1
dd if=levinboot-emmc.img of=/dev/mmcblk1 seek=64
#dd if=levinboot-sd.img of=/dev/mmcblk1 seek=64

sync

When booting from eMMC, Levinboot expects to find payload in a special GPT partition. I just use a single partition with GUID e5ab07a0-8e5e-46f6-9ce8-41a518929b7c.

This is all documented in detail in the Levinboot README file.

2020–09–02: Progress on the multi-boot image

Basic process of creating a multi-boot image is:

At this point each distro will be bootable, but may fail if it has some hardcoded expectations for the partition table structure, filesystem labels, etc. This will need to be fixed manually.

Getting the rootfs tarballs

Many distros just publish a block device image. Getting the rootfs files from the image requires mounting the filesystems contained in the image and backing up all files with bsdtar. Something like this can be used in most cases:

L=`losetup -P --show -f "$1"`
BOOT_PART=1
ROOT_PART=2

mkdir -p m
mount ${L}p${ROOT_PART} m
test -n "$BOOT_PART" && mount ${L}p${BOOT_PART} m/boot
bsdtar -cvf - -C m --numeric-owner . | zstd -z -4 - > rootfs.tar.zst
test -n "$BOOT_PART" && umount m/boot
umount m

losetup -d "$L"

I excluded the distros that don't publish rootfs tarballs or block device images, because they're too much hassle, at the moment. That is pmOS and nemo mobile.

Images/tarballs can be downloaded at:

Excluded:

Partitioning

p-boot requires MBR partitions, so we need to use that with an extended partitioning scheme, to be able to have more than 4 partitions. Creating the partition table is quite simple. We just need to decide how much space each distro will need. zstdcat rootfs.tar.zstd | wc -c can give a clue. There are some larger than 3GiB, but most fit within 2GiB, so let's create a table like this:

At least 32GiB SD card is needed.

Warning: The following script is broken, and leads to first two partitions overlapping. Fixes welcome. :)

sfdisk -W always /dev/mmcblk0 <<EOF
label: dos
label-id: 0x12345678
unit: sectors
sector-size: 512

4M,1020M,L,*
,4G,L
,4G,L
,21G,Ex
,3G,L
,3G,L
,3G,L
,3G,L
,3G,L
,3G,L
,,L
EOF

Filesystems

Now let's create filesystems. My kernel supports f2fs, so if I boot any of the above distros with my kernel, I can just use f2fs everywhere. Unfortunately, I don't know which of the above distros are able to boot from f2fs with their own kernel.

They all probably support ext4. F2FS is much better for SD cards, though. A conundrum :-)

After an investigation, I've decided on this layout (PARTUUIDs will be useful later on).

The table also has the current status of the distros and their assignment to the partitions.

/dev/mmcblk0p1: PARTUUID="12345678-01"  p-boot
/dev/mmcblk0p2: PARTUUID="12345678-02"  mobian (f2fs, 5.7 kernel, PP 1.0-1.2)
/dev/mmcblk0p3: PARTUUID="12345678-03"  KDE neon (ext4 only, 5.7 kernel, PP 1.1 only)
/dev/mmcblk0p5: PARTUUID="12345678-05"  arch (f2fs, my 5.9 kernel, , PP 1.0-1.2)
/dev/mmcblk0p6: PARTUUID="12345678-06"  sxmo (ext4 only, 5.7 kernel, PP 1.1, 1.2)
/dev/mmcblk0p7: PARTUUID="12345678-07"  lune (ext4 only, 5.5 kernel, PP 1.1 only)
/dev/mmcblk0p8: PARTUUID="12345678-08"  maemo (ext4 only, 5.7 kernel, PP 1.0-1.2)
/dev/mmcblk0p9: PARTUUID="12345678-09"  ut  (ext4 only, 5.6 kernel, PP 1.1 only)
/dev/mmcblk0p10: PARTUUID="12345678-0a" sailfish (ext4 only, 5.6 kernel, some custom DTB with partial support for up to PP 1.2)
/dev/mmcblk0p11: PARTUUID="12345678-0b" pureos (f2fs, 5.8 kernel, PP 1.0-1.2)

The status is not great, only maybe 3 distros really support PinePhone 1.2, and only one uses a kernel that's not EOLed upstream. Triste!

Anyway, this leads to:

for bd in /dev/mmcblk0p{3,6,7,8,9,10}
do
  mkfs.ext4 $bd
done

for bd in /dev/mmcblk0p{2,5,11}
do
  mkfs.f2fs $bd
done

After this we can just mount all the partitions one by one and extract the prepared rootfs contents there, with:

mount /dev/mmcblk0p# /mnt
bsdtar -xp --numeric-owner -C /mnt -f rootfs.tar.zst
umount /mnt

That's about it for today. I'll do first boot tests in the following days, and you can look forward to a bootloader setup guide. :)

2020–09–01: More p-boot cleanups and an example configuration

With some suggestions from Yoda, I added a few more sanity checks to the boot partition configurator, and more importantly, I prepared an example boot configuration directory with some ready-made p-boot themes, splashscreens and scripts.

This should make it much easier to get people started using p-boot, even if they will not read the (currenlty somewhat outdated) README file.

The sample config is in the example/ directory in p-boot git repo.

2020–08–31: Releasing p-boot GUI bootloader

Today I finally decided to release display support for p-boot. Along with it, p-boot's code also got quite a bit of cleanup. At this point main.c is fairly readable. There's also a new support for 3GiB variant of PinePhone, that TL Lim sent me some time ago.

I've copied my megatools.megous.com website template, and made it work better on mobile phones, to make a nice landing page for it, too: xnux.eu/p-boot

Since previous publicly released version, p-boot can now be configured to show the name of the phone on the boot screen, which is quite useful if you're like me and TL Lim keeps sending you PinePhones for development every other full moon. :)

I also use the name to describe HW mods I did to the particular phone.

There are some things I'd still like the p-boot to do. Since p-boot has direct access to PMIC, I thought of using it for following things:

And more:

In the other news, TODO

I tried to cleanup and organize my kernel upstreaming TODO list, and it's still too long. So instead I decided to give p-boot a little boost in exposure by preparing a multi-boot image that will feature a range of diverse distributions. With help of #pinephone channel users, I came up with this list:

Looks like making some of these share space and play nice with the others will be quite a challenge. If you're a distro maintainer, publishing a ready-made rootfs tarball is a great way to simplify your inclusion into a multi-boot image.

The image will be great for people new to PinePhone to try various distros with very little hassle. Just flash one image to SD card, and select the distro on boot.

I'll try documenting the image creation steps, but it will most probably be a one shot experiment for me. So if anyone will want to maintain this image further into the future, they'll be very wellcome.

I'm still undecided whether I'll force my kernel on all the above distros, or let them use their own. Probably the latter.

2020–08–31: Getting started

I already maintain xnux.eu which is a regular website that provides summarized information about my projects organized by device/topic. I decided to start this log to provide more of a in-progress information to people interested in my PinePhone related work. That is mostly kernel drivers/bootloader development, and performance optimizations.

Send any feedback you like to x@xnux.eu.