VAssert is a new API, debuting in Workstation 6.5, that uses the Record and Replay functionality that we've been talking about for some time now. As you can tell by its name, VAssert is a relative of your standing programming ASSERT debugging tool, but by delaying assert-checking until later when the exact machine instructions are replayed, it can be very fast. That's some virtualization Deep Magic.
VMware engineers Weiming Zeng and Min Xu give us this guest post on demonstrating VAssert within Apache, and include the Apache patches they used so you can give this a try at home.
A Virtual Buffer-overflow Checker for Apache
by Weiming Zeng & Min Xu
The Record and Replay feature in workstation 6.5 introduces a new guest programming API – VAssert (Virtual Assertions). It is intended that software developers can use it to move expensive program error checking, such as buffer-overflow, to the deterministic replay phase. But does VAssert live up to its promise? As an experiment, we applied VAssert to Apache httpd and wrote a simple buffer-overflow checker by modifying the memory manager in Apache Portable Runtime (APR). Comparing with the same buffer-overflow checker implemented using traditional assertions, the virtual assertions incur 78.77% less runtime overhead.
2 The idea
Our idea to detect buffer-overflow is simple. When allocating memory, append a byte of magic number (the guard) to the end of the memory block; during execution, we frequently check whether the guard is changed. If so, a buffer-overflow is detected.
One of the benefits of this detector is that it is simple to implement. There is no need to intercept all (or most) memory accesses, as other detectors require. But this detector can cause a huge program slowdown if the guard bytes are checked frequently. The slowdown might alter a program's behavior so that bugs disappear when the detector is activated. With a “virtual” detector, however, the slowdown happens mostly during replay time. Since the replay is deterministic, the detector can find bugs without altering a program's behavior.