One of the great mantras of developing software is that one should also build test cases and documentation for that software. For most software projects this is incredibly useful, but there are areas where this is fundamentally hard, and one of the hardest (though possibly not THE hardest) is at the operating system level. To actually test an operating system, you need to be fully running it to actually get the complex interactions working and interacting, and this presents some unique challenges from a testing perspective.
A number of years ago, this very problem came up to Steven Rostedt (now a VMware employee) and he wrote and published ktest for the Linux kernel. Now, this isn’t a test suite in and of itself — it’s a framework to help standardize how to run tests, and allow for bisecting them against the Linux kernel. One of the ways it does this is by making use of one of the most maligned pieces of technology: the humble serial port. Why, the humble and simple serial port is so maligned it’s a mystery to this author, as it is probably one of the single most useful things to have on a computing device. This doesn’t mean that serial ports are not often poorly implemented, but that doesn’t negate their fundamental usefulness. What ktest fundamentally does is compile a kernel, get it installed on a target machine, reboot into that kernel and then run a series of tests defined in a configuration file, and can report back based on its findings. It monitors the target system through — you guessed it — the serial port, as that tends to be the most basic level of communication a system can provide and also tends to be the most reliable way to get error or panic messages out of a system (which, when you are dealing with the core kernel of an operating system where anything can, and does, go wrong, is exceptionally useful).
The target system doesn’t require a lot for ktest to be able to use it:
- A means of reading from the serial console
- A way to move the boot image onto it
- A means of, possibly forcibly, rebooting it
Physical systems have some advantages in certain types of hardware testing, but for the most part, virtual machines may be dramatically more convenient. Let’s take a quick look at what it takes to make use of a virtual machine running in either VMware Workstation, Player or Fusion.
Ktest needs a configuration file to run. It’s a framework, not a set of tests in and of itself, and for this we are going to go over the sample VMware config that’s available with the Linux Kernel (latest version can be found here).
The guest machine that is the target needs a little bit of prep work, but it’s not complicated:
- Edit the Virtual Machine (“Edit virtual machine settings”)
- Add a Serial Port
- You almost certainly want it set “Connect at power on”
- Select “Use socket (named pipe)”
- Select a name that you’ll recognize, like ‘ktestserialpipe’
- From: Server
- To: A Virtual Machine
- Save
- Make sure you note the name, it will be in the base directory of the virtual machine (where the “disks” are stored. The default is /var/lib/vmware/<virtual machine name>/<the name you entered above>)
Really that’s just the instructions for adding a serial port to a VMware virtual machine. After that, the setup is nearly identical to how you would handle any other system you would want to run ktest against, with one specific note: CONSOLE, for ktest, needs to be something that can read a Unix Pipe, as that’s how VMware exposes the serial console. Ncat is a great tool for this, and is fairly ubiquitous so it’s what is suggested. If you follow the rest of the sample configuration file from the upstream Linux kernel you’ll note a number of other variables prefixed with VMWARE_. These are mostly there as helpers for finding where things live and help keep the configuration a bit cleaner. We make use of vmrun on the local system to issue any forcible reboot/reset commands (this can be seen in VMWARE_POWER_CYCLE) and you’ll note that vmrun does need to know if it’s running on Workstation (ws), Fusion (fusion) or Player (player) respectively, so that it knows how to issue the command successfully.
Once you’ve got the basics setup here, running “../ktest.pl vmware.conf” from the examples directory should get you a run that will not only compile the Linux kernel, but will be able to get it installed and reboot a target that’s running as a VMware Guest.
If you are curious to learn more about ktest, one of the best places to look is from a presentation that Steven Rostedt did about it a few years ago.
In his setup he uses a physical machine, but it’s fundamentally no different than running the target as a guest, just with fewer physical wires!
Ktest starts giving you the pieces you need to begin testing something as complex as the kernel of an operating system. Tracking down issues (particularly at this level) can be challenging. You may know that something worked at one point, and you know that it doesn’t anymore and this is where git bisect can be invaluable in tracking down where the breakage happens. Ktest can actually find the breakage for you. If you set up your tests correctly, give ktest a place to start bisecting. It can binary search the commits looking for where the breakage occurred. If you’ve ever had to repeatedly do these kinds of tests, having a framework available to help automate it is amazing. Testing at this level may still be difficult, but ktest helps to make it consistent and doable for most kernel developers, whether they run on real hardware, or are looking for various bugs by using a virtual machine.
