2021–08–10:
Wrapping up Pinephone keyboard firmware development
The keyboard production seems to start soon.
I'm ending the firmware development for now. The final code is here:
https://xff.cz/git/pinephone-keyboard/
Samuel Holland helped with the final testing, and this code will be flashed
in factory. You can read about the design of the code here:
https://xff.cz/git/pinephone-keyboard/tree/README
and in other readme files in that repository.
What I've achieved since June
Initially I received the vendor's flashing tool, vendor's proprietary
firmware code for the keyboard and some schematics for the keyboard.
- I crawled the web for any and all the information I could find about the
obscure MCU and the charger chip used in the keyboard.
- Before receiving the hardware prototype I reverse engineered the bootloader
and parts of the windows based flashing tool and I started writing the much
simplified USB flashing tool for Linux.
- After receiving the keyboard, I finished the flashing tool and used it to
download the bootloader from the actual HW, only to realize that it's different
from the one included in the vendor's firmware code. Though the protocol was
mostly the same.
- Keyboard matrix on the prototype was broken (some conductive traces on the
plastic backplane were shorted), so I tried to figure out if I can fix it, but
there was nothing obvious that I could do.
- The keyboard MCU (some obscure 8051 based USB controller with little to no
code available online) has no documented debugging interface, so I had to
create one. I could not make it accessible over I2C, because what I needed it
for was to debug I2C interface.
- So I wrote a simple USB device stack for the MCU and used it to have an
easy way to switch to flashing mode, read out status of the keyboard matrix, and
to be able to read out text logged to a circular buffer (used for debugging via
printf()
).
- To access the debugging information I wrote some helper tools to read out
the log buffer over USB from the MCU and display it on my PC.
- I used this debugging access to figure out the quirks of the interrupt
driven I2C B slave interface inside the MCU and to write and debug the I2C slave
code, and iron out some nasty quirks (timing related issues are the worst,
because your added debug code changes the timing!).
- After some basic I2C interface code was working, and I could communicate
with the MCU from the phone, I spent some time designing the I2C interface by
documenting it prior to trying to implement something (https://xff.cz/git/pinephone-keyboard/tree/README.i2c-intf).
- So that users don't need to solder on the USB cable to the PCB in order to
flash the new firmware, I decided to figure out some way to flash the firmware
over the I2C interface directly from the phone. This is not just about writing
some code over I2C to flash. You can't overwrite the code that's currently
running on the MCU, so there needed to be some kind of split between code doing
the flashing and the code being flashed. So I designed a stock firmware/user
firmware split.
- Stock/user firmware split introduces its own issues that needed to be
solved. First, interrupt vectors are at fixed addresses and need to be forwarded
to user firmware for the user firmware to be able to use them. This forwarding
can't be fixed, because stock firmware also uses interrupts, so I had to design
some mechanism to turn the forwarding on/off dynamically based on some variable
stored in RAM. Switching between stock and user firmware is also fraught with
issues, because the MCU is not in a clean post-reset state with all resources in
a known state. So this known state has to be achieved manually (tedious!).
- Next I had to reverse engineer the undocumented flashing interface, because
that's omited from the datasheet, so that I could implement my own flash
writing routines for the flashing over I2C.
- At this point the firmware was getting quite complete, so I turned to power
usage optimizations. The MCU is consuming quite a bit of power unless
agressively powered down each time it doesn't need to run. There's only limited
ways to wake up from power down, so I needed to figure out what jobs inside the
MCU need to finish running prior to power down and can't wake up the MCU. As
long as these jobs are not finished, powerdown has to be prevented. I've
modified my testing circuit so that I could measure the power usage of the
keyboard, and managed to get the power use down from 20mW to around 2mW.
And this was just the keyboard MCU part. The second part of the problem is
the charging chip.
- It only comes with a chinese datasheet for its I2C interface. It could be
mostly translated by throwing it in google translate, but some parts required
OCRing low-res Chinese text to be able to copy paste it to google translator.
I was quite surprised that it worked well, given the low resolution of the
images.
- The chip's I2C interface is just fucked up. When the chip wakes up it
checks if I2C pins have high level and if not it will fuck up the interface by
switching it to LED mode and driving the I2C pins as if charge indication LEDs
were connected to it. This pure stupidity makes it so that the charger can't be
put on a shared bus. I2C is a shared bus by design. I wasted ~2 days of my life
figuring out how to get around this idiocy.
- I've tried a bunch of things with the charger, and wrote a simple tool to
control the charger over I2C, and to read the status of the battery from the SoC
for testing.
- In the end there was some pressure to wrap things up quickly, so I bailed
on getting the I2C communication with the charger work in a fully tested manner.
Not for the lack of trying though, but due to pressure to wrap things up. I had
plans and HW modifications ready to test three different approaches to fixing
the issue.
- So as a last ditch effort to have at least a chance to make battery status
monitoring work eventually with the final hardware, I suggested a few most
likely to work modifications that should allow it, based on all that was learned
so far and a few last minute tests.
All in all this project required a lot of bootstrapping, reverse engineering,
and custom tooling support. That's why you'll find many tools in my code
repository, each can still be used for one of the purposes I used them for
originally.
I don't measure time spent on my hobbies, but this easily took ~100h of time
over the last ~2.5 months and I quite like the result and all the new stuff
that I've learned. That's also why I haven't done much on Pinephone kernel
last few months. ;)
Lessons learned
I didn't know much about using USB from Linux and writing USB code for
microcontrollers. Nor did I use USB in any of my hobby HW projects in the past,
and that changed after this project. I thought USB is complicated. I found
that using USB as a dumb transport is quite comparable to using a serial port.
It needs more code, but USB solves issues that you have to solve by hand in some
upper protocol layer you layer on top of serial port communication, so the
complexity is comparable.
On Linux side you can access USB devices with about 4–5 ioctl calls, and in
most microcontrollers that I have access to, you can write basic USB device
code in about 200 lines (no need for large USB stacks supplied by the vendor
when you're just using USB as a dumb transport instead of trying to implement
some of the standard USB device classes).
In fact I wrote 2 USB device implementations during the development of the
keyboard. One for the controller inside the keyboard, and one for FX2 board
I used to simplify testing. And it was not that hard. FX2 is especially easy
and well documented, with hardware helping the firmware writer quite a bit at
each step.
It was a fun ride, mostly, except for some stressful frustrations towards the
end. I don't mind frustration with trying to fix a broken HW, reverse engineer
stuff, etc. There was a lot of that in this project after all. Interrupt driven
MCU code is especially tedious to write, since you have to re-read the code
several times, searching for potential cirical sections, etc. Compiler does not
help at all here.
What I do mind is being put under time pressure on a project where I'm just
volunteering my free time. That's very easy way to get bitter feelings, which
I'd like to avoid.
I've damaged the last prototype trying to figure out some way to connect I2C
of the keyboard MCU to the charger I2C so that phone can monitor the keyboard
battery charge and control the charger.
At the moment I have no way to do anything with the keyboard, until I fix
my prototype to revert the various HW modifications I attempted, or get the
final keyboard, so the feeling of accomplishment is a bit bittersweet. Oh
well. :)
Some random photos from my
efforts
First effort to give access to charger I2C to the phone (using TXS0108 level
converter):
Second attempt to give access to charger I2C to the phone (via direct
connection of I2C A MCU interface to the charger chip). A plan:
Execution:
And giving up after weird voltages on I2C pins and no more time to try
things: :(
Final untested suggested changes for having I2C access to the charger proxied
by the keyboard, that will be present in the final keyboard design, to have a
chance to figure out the charger I2C communication later on (as a firmware
update):
My new favorite USB dev board (Cypress FX2), that I used to test I2C
interface of the keyboard:
FX2 controlling the keyboard over I2C from the PC: