2021–08–10: Wrapping up Pinephone keyboard firmware development

The keyboard production seems to start soon.

I'm ending the firmware development for now. The final code is here:

Samuel Holland helped with the final testing, and this code will be flashed in factory. You can read about the design of the code here:

https://xff.cz/git/pinephone-keyboard/tree/README

and in other readme files in that repository.

What I've achieved since June

Initially I received the vendor's flashing tool, vendor's proprietary firmware code for the keyboard and some schematics for the keyboard.

I crawled the web for any and all the information I could find about the obscure MCU and the charger chip used in the keyboard.
Before receiving the hardware prototype I reverse engineered the bootloader and parts of the windows based flashing tool and I started writing the much simplified USB flashing tool for Linux.
After receiving the keyboard, I finished the flashing tool and used it to download the bootloader from the actual HW, only to realize that it's different from the one included in the vendor's firmware code. Though the protocol was mostly the same.
Keyboard matrix on the prototype was broken (some conductive traces on the plastic backplane were shorted), so I tried to figure out if I can fix it, but there was nothing obvious that I could do.
The keyboard MCU (some obscure 8051 based USB controller with little to no code available online) has no documented debugging interface, so I had to create one. I could not make it accessible over I2C, because what I needed it for was to debug I2C interface.
So I wrote a simple USB device stack for the MCU and used it to have an easy way to switch to flashing mode, read out status of the keyboard matrix, and to be able to read out text logged to a circular buffer (used for debugging via printf()).
To access the debugging information I wrote some helper tools to read out the log buffer over USB from the MCU and display it on my PC.
I used this debugging access to figure out the quirks of the interrupt driven I2C B slave interface inside the MCU and to write and debug the I2C slave code, and iron out some nasty quirks (timing related issues are the worst, because your added debug code changes the timing!).
After some basic I2C interface code was working, and I could communicate with the MCU from the phone, I spent some time designing the I2C interface by documenting it prior to trying to implement something (https://xff.cz/git/pinephone-keyboard/tree/README.i2c-intf).
So that users don't need to solder on the USB cable to the PCB in order to flash the new firmware, I decided to figure out some way to flash the firmware over the I2C interface directly from the phone. This is not just about writing some code over I2C to flash. You can't overwrite the code that's currently running on the MCU, so there needed to be some kind of split between code doing the flashing and the code being flashed. So I designed a stock firmware/user firmware split.
Stock/user firmware split introduces its own issues that needed to be solved. First, interrupt vectors are at fixed addresses and need to be forwarded to user firmware for the user firmware to be able to use them. This forwarding can't be fixed, because stock firmware also uses interrupts, so I had to design some mechanism to turn the forwarding on/off dynamically based on some variable stored in RAM. Switching between stock and user firmware is also fraught with issues, because the MCU is not in a clean post-reset state with all resources in a known state. So this known state has to be achieved manually (tedious!).
Next I had to reverse engineer the undocumented flashing interface, because that's omited from the datasheet, so that I could implement my own flash writing routines for the flashing over I2C.
At this point the firmware was getting quite complete, so I turned to power usage optimizations. The MCU is consuming quite a bit of power unless agressively powered down each time it doesn't need to run. There's only limited ways to wake up from power down, so I needed to figure out what jobs inside the MCU need to finish running prior to power down and can't wake up the MCU. As long as these jobs are not finished, powerdown has to be prevented. I've modified my testing circuit so that I could measure the power usage of the keyboard, and managed to get the power use down from 20mW to around 2mW.

And this was just the keyboard MCU part. The second part of the problem is the charging chip.

It only comes with a chinese datasheet for its I2C interface. It could be mostly translated by throwing it in google translate, but some parts required OCRing low-res Chinese text to be able to copy paste it to google translator. I was quite surprised that it worked well, given the low resolution of the images.
The chip's I2C interface is just fucked up. When the chip wakes up it checks if I2C pins have high level and if not it will fuck up the interface by switching it to LED mode and driving the I2C pins as if charge indication LEDs were connected to it. This pure stupidity makes it so that the charger can't be put on a shared bus. I2C is a shared bus by design. I wasted ~2 days of my life figuring out how to get around this idiocy.
I've tried a bunch of things with the charger, and wrote a simple tool to control the charger over I2C, and to read the status of the battery from the SoC for testing.
In the end there was some pressure to wrap things up quickly, so I bailed on getting the I2C communication with the charger work in a fully tested manner. Not for the lack of trying though, but due to pressure to wrap things up. I had plans and HW modifications ready to test three different approaches to fixing the issue.
So as a last ditch effort to have at least a chance to make battery status monitoring work eventually with the final hardware, I suggested a few most likely to work modifications that should allow it, based on all that was learned so far and a few last minute tests.

All in all this project required a lot of bootstrapping, reverse engineering, and custom tooling support. That's why you'll find many tools in my code repository, each can still be used for one of the purposes I used them for originally.

I don't measure time spent on my hobbies, but this easily took ~100h of time over the last ~2.5 months and I quite like the result and all the new stuff that I've learned. That's also why I haven't done much on Pinephone kernel last few months. ;)

Lessons learned

I didn't know much about using USB from Linux and writing USB code for microcontrollers. Nor did I use USB in any of my hobby HW projects in the past, and that changed after this project. I thought USB is complicated. I found that using USB as a dumb transport is quite comparable to using a serial port. It needs more code, but USB solves issues that you have to solve by hand in some upper protocol layer you layer on top of serial port communication, so the complexity is comparable.

On Linux side you can access USB devices with about 4–5 ioctl calls, and in most microcontrollers that I have access to, you can write basic USB device code in about 200 lines (no need for large USB stacks supplied by the vendor when you're just using USB as a dumb transport instead of trying to implement some of the standard USB device classes).

In fact I wrote 2 USB device implementations during the development of the keyboard. One for the controller inside the keyboard, and one for FX2 board I used to simplify testing. And it was not that hard. FX2 is especially easy and well documented, with hardware helping the firmware writer quite a bit at each step.

It was a fun ride, mostly, except for some stressful frustrations towards the end. I don't mind frustration with trying to fix a broken HW, reverse engineer stuff, etc. There was a lot of that in this project after all. Interrupt driven MCU code is especially tedious to write, since you have to re-read the code several times, searching for potential cirical sections, etc. Compiler does not help at all here.

What I do mind is being put under time pressure on a project where I'm just volunteering my free time. That's very easy way to get bitter feelings, which I'd like to avoid.

I've damaged the last prototype trying to figure out some way to connect I2C of the keyboard MCU to the charger I2C so that phone can monitor the keyboard battery charge and control the charger.

At the moment I have no way to do anything with the keyboard, until I fix my prototype to revert the various HW modifications I attempted, or get the final keyboard, so the feeling of accomplishment is a bit bittersweet. Oh well. :)

Some random photos from my efforts

First effort to give access to charger I2C to the phone (using TXS0108 level converter):

Second attempt to give access to charger I2C to the phone (via direct connection of I2C A MCU interface to the charger chip). A plan:

Execution:

And giving up after weird voltages on I2C pins and no more time to try things: :(

Final untested suggested changes for having I2C access to the charger proxied by the keyboard, that will be present in the final keyboard design, to have a chance to figure out the charger I2C communication later on (as a firmware update):

My new favorite USB dev board (Cypress FX2), that I used to test I2C interface of the keyboard:

FX2 controlling the keyboard over I2C from the PC: