Note: all the source code referenced in this blog post is for educational use only, and is provided with no support, endorsement, warranty, guarantee or implied fitness for a particular purpose. This does not constitute sample driver code. Do not reach out to VMware to support this code. Do not copy and reuse in any commercial or product setting.
How do you write a driver for ESXi, on Arm, for the Raspberry Pi, to interface with the GPIO pins without any knowledge of how GPIO works?
Having now written a working GPIO driver, I can tell you. This is how to do it:
- Conceive of writing a driver
- Gain access to the SDK
- Hack it until it works
- Realize you have no idea how the device you’re driving works
- Read the wrong datasheet
- Get distracted by a datasheet for an LED element because you couldn’t admit to yourself that your code is buggy
- Read the right datasheet
- Realize your driver doesn’t work because your Raspberry Pi is broken
- Use your flatmate’s Raspberry Pi instead
- Code, compile, reboot
- Repeat step 14 until you start question whether the Raspberry Pi’s boot logo looks like a raspberry at all
- Wake up your flatmate at 2am because you’ve finally gotten it to work
That’s it! Easy!
The good news is, since I’ve gone through the above, perhaps you won’t have to.
Beginnings
Getting in touch with the right people was easy thanks the public Slack channel for the ESXi on Arm Fling (#esxi-arm-fling). I asked the channel whether there were plans for releasing the Native Driver Development Kit (NDDK) to the public, since there are so many different peripherals for the Raspberry Pi, not to mention Arm devices in general.
Turns out, while the NDDK is not currently available to the public, a few folks on the ESXi on Arm team had been working on a port of the NDDK to the Arm platform. After a bit of work, I was able to use it to build a small “Hello, World!” module.
A Device on the ACPI Bus
Printing “Hello, World!” into the vmkernel.log is one, but I had no idea how GPIO worked and, what’s worse, how to access the device via the VMKAPI.
After digging around in VSISH for a bit, I found the device tree located at /system/devicetree/devices/. A device tree is a blob of binary data that an OS can use to discover devices that don’t have auto-discovery capabilities. For example, the Raspberry Pi’s GPIO pins or it’s VideoCore multimedia processor (which, incidentally, provides a lot more than just access to the framebuffer).
Now, if you have a look at the device tree on your own system, you’ll notice it’s not very easy to navigate.
To remedy this, I wrote a Python script to output the device tree in a hierarchy.
I noticed a device called “GPI0” (written with a zero), which, I assumed, was probably the GPIO device. The device was located on the ACPI bus.
Now, how do we access the GPIO device? Cyprien was kind enough to send me a copy of the NDDK 7.0 programming guide shared with partners. Fortunately, it has a section on ACPI device drivers. Unfortunately, calling it terse would make this blog post look like the Encyclopedia Britannica. Another snag.
Eventually, I found the source code for an ACPI test driver, which allowed me to map the GPIO device’s memory-mapped I/O (MMIO) region into virtual memory.
At this point, I had a function VMKernel module and the first step towards a working driver.
Working with the Raspberry Pi Hardware
The following few days were spent delving into the enticing prose of the BCM2711 ARM Peripherals datasheet. Not without detours, of course.
The Raspberry Pi seemingly misreports the SoC as being a BCM2835, and the VMKernel accordingly reports the MMIO region being 180 bytes in size. The BCM2835 has an MMIO region of that size, but not the BCM2711, which goes up to 244. Which made me very confused about which chip I was working with. That is, until I inspected the etching on the chip itself. It clearly said BCM2711. (I’ve since created an issue for this behavior)
Many failed attempts later, I managed to read and write to registers, select pin functions, and set/clear individual pins. In other words, I had a semblance of a device driver.
Setting and clearing pins is, however, not very interesting.
Enter the Pimoroni Fan SHIM.
The Fan SHIM
The folks over at Pimoroni have created a lot of cool accessories (called HATs) for the Raspberry Pi, including the Fan SHIM. The Fan SHIM is a piece of kit that you push on top of the GPIO pins and includes a fan, an LED, and a button. All programmable, of course.
This was also when I realized my Raspberry Pi was broken. It took me a while to figure it out, too. It was only after installing the Fan SHIM on my flatmate’s Raspberry Pi and it immediately working (on Raspberry Pi OS), that I realized mine was toast. Thank goodness my flatmate had one or I’d have probably pulled all my hair out before I figured out the problem. Graciously, Patrik, my flatmate, lent me his Raspberry Pi for further development.
Toggling the fan on and off was the easiest part. Getting the LED working, was a bit more difficult. To find out how to program the LED, I read through the source code for the official Fan SHIM Python library, an unofficial C library written by a guy called Florian Bernd, and even an obscure library written in Rust. After a grueling few hours, I started doubting everything and ended up reading through the datasheet for the LED module (the APA102) as well, thinking it would hold the answers to why nothing worked. Riveting literature as you can imagine. Turns out, it was a bug in my code. Figuring that out felt like I was a butler reading an Agatha Christie novel — quelle surprise, I was the perp all along. My code was overwriting the value of a GPIO register because I messed up the bitwise operations on it.
So, now I had a clunky driver wrapped into a VMKernel module that communicated with a Raspberry Pi HAT.
I was delighted with myself.
Régis, yet another member of the ESXi on Arm team, and Andrei urged me to make a generic version of the driver.
So I glued a character device driver onto it. That way, one could communicate with the GPIO device using “commands” echoed into a file.
One step closer to a generic driver.
A Generic GPIO Driver
The final step was making the GPIO driver generic, and not dependent on a particular piece of kit attached to the GPIO pins.
Based on Régis’ feedback, I slimmed down the driver, so that, currently, it “only” presents the MMIO region as a file which can be read and written to like any other.
The offsets from the start of the file correspond to the offsets of the various GPIO registers.
A Python library now does the heavy lifting, so that programming the GPIO pins can be done easily from the user world.
Since the GPIO pins can be programmed individually, the driver supports all sorts of HATs, not just the Fan SHIM.
You can find the code and the VIB on my GitHub page: https://github.com/thebel1/thgpio
What’s Next?
The next challenge is to figure out how to present the GPIO device to a VM.
If we can figure that out, you’ll be able to use all of your Raspberry Pi HATs directly inside a VM!
In the meantime, I’ll be hacking away at whatever half-baked idea pops into my head.
