VMTN Blog

3 Background

3.1 Record and Replay

Our Record and Replay is at an instruction-exact level. During replay, the exact same internal program state is recreated, not just the program screen output.

This is useful for application debugging and testing, especially for multi-threaded programs. Often the behavior of these many threads differ, and will never be the same in different runs of the program. With Record/Replay, all the things that ever happened during the recording will be reproduced exactly when replayed. Problems with irreproducible bugs will no longer trouble the developers. Still confused by Record and Replay? See this blog post for more information.

3.2 VAssert

While replaying, we provide several methods for developers to probe the internal state of a program. You can use a debugger or insert replay-time-only code, such as virtual assertions.

VAssert provides an API like ordinary ASSERT(), which is named Vassert_Assert(). It is intended to be used as an ordinary ASSERT(), but it takes effect only in replay mode.

Internally, the code called from VAssert_Assert() is skipped unless a program is replayed. The place where a programmer calls VAssert_Assert() is really a place holder for inspecting the internal state of the program during replay time. During replay, once a VAssert is evaluated successfully, the virtual machine goes back to the state before the VAssert and continues normal replaying. Otherwise, if a VAssert fails, the virtual machine stops replaying and allows users to interact with the program, e.g., through a debugger.

Here is the the programming guide to VAssert.

3.3 The APR pools

Apache-httpd uses APR for memory management. All memory blocks allocated by APR are contained in memory pools. There can be multiple pools at any given time, and these pools are organized as a tree. The tree structure ensures all the pools are freed when APR terminates, to avoid memory leaks.

Memory management related functions in APR are apr_pool_initialize(), apr_pool_terminate(), apr_pool_create(), apr_pool_destroy(), apr_palloc(), apr_pcalloc() and apr_pool_clear().

4 Implementation

Setting the guards. A good place is to set the buffer-overflow guard is when apr_palloc() is called and the memory block is going to be returned to the caller.

We modify apr_palloc() to do these things, which are transparent to the caller:

1. increase the block size by 1 to make room for the guard
2. allocate a block of memory from a pool
3. append a guard (a magic number) at the end of the block.
4. keep a pointer to the guard in a buffer for checking purpose
5. return the block to the caller

The data structure looks like this:

When to check the guards. One time that a guard should be checked is before the memory block to which it's appended is freed. So we can check buffer-overflow in apr_pool_clear() and apr_pool_destroy(), like this:

1. ASSERT(guardOf(mem_block) == 0xFA);
2. the rest of work in the original code

However, it can be too late to detect a buffer-overflow only when memory blocks are freed. What if the block never gets freed? Therefore, we implemented a function named check_all_pools() to check all guards in all pools in more function call sites, such as apr_palloc(), apr_pool_create(), apr_pool_create_ex(), apr_pool_clear(), apr_pool_destroy() and many more places.

Making it virtual. To make the detector “virtual,” we simple replaced the ASSERT() call to check_all_pools with a call to Vassert_Assert(). During the recording of normal Apache execution, the call to check_all_pools() is skipped and the actual checking of those guards are deferred until replay-time.

5 Performance

We studied the performance of the buffer-overflow detector using ab (Apache benchmark). On a Fedora 8 as Guest OS inside VMware Workstation, we did the following:

1. Build and deploy the Apache-httpd into the guest OS, both the original version and the version with the buffer-overflow detector. Start httpd.
2. Use Apache Benchmark (the ab command) to drive httpd:
   ab -n10000 http://hostname/sample_file
   This command should generate 10000 requests to the “sample_file” on the server and output performance statistics.
3. Run the benchmark in both non-recording mode and recording mode.
4. The data reported by ab of ‘Time taken for tests’ are shown below:

   	Without Detector	Normal Detector	Virtual Detector
   without R/R	6.32 (secs)	33.56 (secs)	7.13 (secs)
   with R/R	19.31 (secs)	59.33 (secs)	21.73 (secs)

The first row is the time taken to run without using VMware Record and Replay and the second row is using Record and Replay. The difference in time between the two rows is influenced by overhead, because we used an unoptimized internal build of VMware Workstation. The actual overhead of Record and Replay is much lower.

What is interesting is that the virtual buffer-overflow detector (21.73 secs) beats the normal detector in both non-recording (33.56 secs) and recording modes (59.33 secs).

We didn’t measure the speed of replaying the virtual detector. It is much slower than recording, yet still deterministic. We imagine the replay runs can be down overnight, since all replaying can be done unattended. The only cost is machine time.

6 Conclusion

We found VAssert is effective in removing program-checking overhead at runtime. It is relatively easy to add the VAssert detector to existing programs, which mostly use assertions already. Since VAssert is not performance sensitive, programmers may concentrate on correctness, not performance, when implementing their program-checkers.

7 Patch files to Apache APR mentioned in this blog

Download files.zip