VMTN Blog

VAssert is a new API, debuting in Workstation 6.5, that uses the Record and Replay functionality that we've been talking about for some time now. As you can tell by its name, VAssert is a relative of your standing programming ASSERT debugging tool, but by delaying assert-checking until later when the exact machine instructions are replayed, it can be very fast. That's some virtualization Deep Magic.

VMware engineers Weiming Zeng and Min Xu give us this guest post on demonstrating VAssert within Apache, and include the Apache patches they used so you can give this a try at home.

A Virtual Buffer-overflow Checker for Apache

by Weiming Zeng & Min Xu

1 Overview

The Record and Replay feature in workstation 6.5 introduces a new guest programming API – VAssert (Virtual Assertions). It is intended that software developers can use it to move expensive program error checking, such as buffer-overflow, to the deterministic replay phase. But does VAssert live up to its promise? As an experiment, we applied VAssert to Apache httpd and wrote a simple buffer-overflow checker by modifying the memory manager in Apache Portable Runtime (APR). Comparing with the same buffer-overflow checker implemented using traditional assertions, the virtual assertions incur 78.77% less runtime overhead.

2 The idea

Our idea to detect buffer-overflow is simple. When allocating memory, append a byte of magic number (the guard) to the end of the memory block; during execution, we frequently check whether the guard is changed. If so, a buffer-overflow is detected.

One of the benefits of this detector is that it is simple to implement. There is no need to intercept all (or most) memory accesses, as other detectors require. But this detector can cause a huge program slowdown if the guard bytes are checked frequently. The slowdown might alter a program's behavior so that bugs disappear when the detector is activated. With a “virtual” detector, however, the slowdown happens mostly during replay time. Since the replay is deterministic, the detector can find bugs without altering a program's behavior.

Very good intro to some of the history and challenges of x86 virtualization, the difference in various approaches, what's being used by the various players today, what the future is bringing with the coming generations of hardware assist, and how VMware is supporting open standards to get us all to an interoperable, high-performance virtual future.

Link: Understanding Full Virtualization, Paravirtualization, and Hardware Assist

VMware offers a flexible “multi-mode” VMM architecture depicted in Figure 12 that enables a separate VMM to host each virtual machine. VMware allows you to select the mode that achieves the best workload-specific performance based on the CPU support available. The same VMM architecture is used for ESX Server, Player, Server, Workstation and ACE. While today’s workloads can employ a 32-bit BT VMM or a 64-bit VMM with BT or VT-x, tomorrow’s workloads will be hosted on VMMs that support 32 and 64-bit versions of AMD-V + NPT and VT-x + EPT. VMware provides a flexible architecture to support emerging virtualization technologies. Multi-mode VMM utilizes binary translation, hardware assist and paravirtualization to select the best operating mode for each workload and processor combination. Hardware assist will continue to mature and broaden the workloads that can be readily virtualized.

This is also an interesting point to keep in mind: Moore's Law trumps everything else we're doing with respect to performance:

Compute-intensive workloads already run well with binary translation of privileged instructions and direct execution of non-privileged instructions, but NPT/EPT will provide noticeable performance improvements for memory-remapping intensive workloads by removing the need for shadow page tables that consume system memory. Increased performance and reduced overhead expected in future CPUs will provide motivation to use hardware assist features much more broadly, but don’t expect revolutionary improvements. As processors get significantly faster each year, each year’s processor performance increases will likely have a greater impact on virtualization capacity and performance than future hardware assist optimizations.

And why paravirtualization won't solve your problems today (but with open standards, may in the future):

To be clear, VMware does find processor paravirtualization to increase performance significantly on some workloads today, but the longer term performance delta when second generation hardware assist features are available is unclear. The performance difference may be reduced, eliminated, or expanded as enhancements to the paravirtualization interface may create new opportunities. It’s an open question.

As VMware sees it, the major problem with processor paravirtualization is the need for guest OS modification that makes it dependent on a specific hypervisor to run. The Xen interface, for example, implements deep paravirtualization with strong hypervisor dependency. The OS kernel is closely tied to structures in the hypervisor implementation. This creates an incompatibility as the XenLinux kernel can’t run on native hardware or other hypervisors, doubling the number of kernel distributions that have to be maintained. Additionally, it’s limited to newer, open source operating systems as the intrusive changes to the guest OS kernel require OS vendor support. Finally, the strong hypervisor dependency impedes the independent evolution of the kernel.

Notes and video from this morning's keynote will arrive soon. VMware's Andy Tucker has already blogged about an exciting new technology that Mendel Rosenblum demoed this morning, and goes into a bit of detail about what's going on behind the scenes:

What was that again?.

This morning during his keynote address at VMworld Mendel Rosenblum talked about (and demoed) a new virtual machine monitor capability we've been playing with in VMware R&D, called record/replay.  The basic idea is to be able to record the instruction stream that a virtual machine executes and be able to reproduce it exactly at a later time.  This isn't just the instructions associated with a single application process or thread; it includes all code executed within a VM, including multiple processes, kernel code, and interrupt handlers.  ...

Personally as a kernel developer I've spent many hours staring at object code and remnants of register and kernel memory state and trying to deduce why a problem occurred, wishing I had a time machine that would allow me to back up and see the state of a given register before it was clobbered by unrelated code, or figure out what thread scribbled garbage onto a critical data structure. ...

You can also probably think of other uses for this technology - one that comes to mind is keeping a log of execution for analyzing security attacks.