You can make debugging a lot less onerous by using tracing software. To implement tracing, though, you first need to determine what ran when and the relevant context, and then do it with as little overhead as possible.
The Linux kernel uses a tool called recordmcount at build time as part of the kernel’s tracing solution. It parses the intermediate ELF-formatted object files produced by the compiler from the kernel C source code and creates a table that you can use to enable/disable tracing at runtime.
But a second kernel build-time tool, objtool, also parses the intermediate ELF-formatted object files and plays an important role in producing reliable stack dumps, checking stack usage and more.
Both objtool and recordmcount parse the kernel pieces produced by the compiler and both can assemble tables of useful information about the code. These redundancies suggest that sharing code is possible. In this post, I want to explain why that might be worth doing.First, though, it’s useful to delve a little deeper into how these Linux kernel tools work.
Objtool is a newer set of code that is capable of scanning the kernel for multiple purposes. It derives its name from the fact that the compiler outputs pieces of the kernel called object files (files ending with .o). It takes these files (which are not yet linked into a complete kernel) and analyzes the produced code. This lets objtool add additional data to the kernel that can then be used to modify the kernel as it runs.
Recordmcount, in contrast, has been part of the kernel build process for a long time but never evolved to do more than one very specific job. It finds where function calls are instrumented by the compiler — historically called “monitor count” or mcount and now called fentry — and collects these locations in a table that it adds to the ELF object file. When the object files are linked into a complete kernel, the information is aggregated and becomes available to the kernel code at runtime, after which the kernel can implement tracing. By changing the code slightly at runtime, the kernel can avoid the overhead of always tracing everything—it enables tracing only when requested and lets you limit the scope of data collected.
The kernel build tools that support collecting the information you need to implement tracing are complementary to other purposes within the kernel. That includes live patching, which can fix bugs and mitigate security issues. Here, when the kernel code checks and reveals an important inconsistency, it can use the called function information to produce stack traces that augment information like register contents, as well as memory addresses representing the lowest levels of insight into the computer’s operation at the time that the inconsistency is discovered. This information can then be written to simple devices like small persistent (aka non-volatile) memory (similar to this), crude communication links like a serial port, or the simplest forms of ethernet packets (e.g. UDP).
Recent changes in the kernel have allowed these tools to provide reliable stack dumps, confirm that the kernel code complies with certain rules which ensure safe operation, and more. This is true for both objtool and recordmcount.
These dynamic kernel modifications ensure that the performance costs of collecting this information are only incurred when absolutely necessary. So objtool, for example, can be used to scan for function calls or check speculative call-return paths related to mitigating the infamous Spectre and Meltdown security problems. It also produces reliable stack metadata in a format called ORC, which can be used to discover what functions are currently executing in the kernel.
Recordmcount also scans the code produced by the compiler. It collects a list of locations that call and return from functions, then adds them into the kernel in the form of tables. These function calls can then be patched at runtime to efficiently implement the tracing functionality the kernel offers.
This is why it’s worth considering merging objtool and recordmcount. It would make collecting the function call information for tracing just another part of objtool’s suite of capabilities. The two would share the code that parses the kernel pieces as it’s being built, reducing the burden of maintaining the code.
Stay tuned to the Open Source Blog and follow us on Twitter (@vmwopensource) for all the latest open source projects and updates.
