Author Archives: Adam Gross

Getting smart cards to work with replay debugging

After moving from working on Workstation to working on the Windows View Client, I try to use Workstation’s Visual Studio plugin as much as I can when debugging the View Client.  Being able to use the remote debugging integration to debug in my VMs is really helpful, but replay debugging takes the cake as the single most helpful feature that Workstation offers me.  I seriously cannot say enough about its power in changing a bug from nearly impossible to fixable in the course of an hour or two.

The most recent example was in the upcoming smart card removal policy feature for the Windows View Client.  Due to the nature of this feature, if a smart card removal policy was also configured on the Windows machine, we were both triggering at exactly the same time.  Most of the time this would be fine; Windows would lock and the View Client would disconnect from the View Connection Server and show the Connect dialog.  But occasionally (anywhere from 10-30% of the time) I would see weird behavior where the dialog was arbitrarily moved to the upper-left corner of the screen and resized to be 100 pixels taller!

As I figured out quickly, debugging using the traditional methods was not an option.  Either I would not see the issue or if I spent too much time in breakpoints I would lose my connection to the View Connection Server.  I could feel replay debugging calling me because it fit so perfectly; I could run the program in the VM and make sure I would reproduce the issue, then attach a debugger and spend however much time I needed to figure it out.  Features like Reverse Run to Cursor and Reverse Continue would make debugging faster as well.

I was all ready to start up on it when I remembered that VMware’s record/replay technology doesn’t support USB devices.  Whether you connect your smart card exclusively to the virtual machine or connect it in shared mode, it is still exposed to the virtual machine as a USB device.  If you start a recording in Workstation, this USB device is immediately disconnected.

The workaround that I figured out was simply to start a recording in a VM then RDP into the VM and use RDP’s built-in smart card redirection to connect the smart card to the VM while recording.  Then in the RDP session, I would run the View Client and reproduce the bug, close the View Client, then stop the recording.  This works because RDP doesn’t actually connect the smart card to the virtual machine as a USB device (seriously, check Device Manager!).  Instead it performs API redirection from the server (in this case the VM) to the client (in this case the host machine) and this certainly still works when recording execution in Workstation.

The only downside of this approach is that when you are replaying execution, you won’t be able to see the contents of the console, as seen below.  So this approach is best for uncomplicated debugging sessions where you generally remember the procession of commands executed in the application.  The reason why you can’t see the console is that when you RDP into a machine, the console session is locked and RDP creates a new session to protect your privacy.  When you replay this recording, there is no actual network traffic (it is just simulated) so there is no way to reattach an RDP client to it.  You are only left viewing the locked console session, but the program still is executing in the virtual machine.

Locked
This is the only view you get into the VM when replaying, but the app is running and the smart card activity really is being replayed!

Once I figured out how to get smart cards working with replay debugging, isolating and fixing the bug was extremely easy.  As a parting shot, I want to commend the improvements made in replay debugging for Workstation 7.0.  As an internal dogfooder, I am often one of VMware’s harshest critics.  But when testing replay debugging this time, I was taken aback at the speed improvements and optimizations when doing normal replay sessions and especially when doing reverse operations.  These folks have worked so hard on the product and it really shows.

As a summary, here are the steps to create a recording of your app with smart cards:

  1. Start your VM and find the IP address.
  2. Start recording execution in Workstation by selecting VM > Replay > Record.
  3. RDP into your VM, making sure to enable smart card redirection in the Local Resources tab by clicking More…
  4. Run your app in the RDP session, reproducing your bug.
  5. Close your app, disconnect your RDP session, and stop your recording.

Using the VMware Visual Studio plugin

Last year for Workstation 6.0, one of the features we added was a plugin for Visual Studio 2005 that allows developers to debug a project (native and managed C/C++, C#, or Visual Basic) inside of a Windows virtual machine (specifically Windows 98, Windows 2000, and later).  All of you Windows developers know how much of a pain it is to support multiple versions of Windows, but this tool is designed to make that process much, much easier.  The Visual Studio Integrated Debugger (VSID in short) plugin utilizes Visual Studio remote debugging technology to allow you to debug a project inside of a VM as if you were debugging on your host computer with the click of one button.

The plugin’s toolbar and menu, added to Visual Studio 2005:

Toolbar_3
Menu_2

The most difficult part of getting all of this to work is the initial setup.  Once you can get your configuration working,
it should work well for a long time.  But it is certainly a chore getting your host and guest set up to comply with all of the security regulations that Microsoft requires to successfully remotely debug on another machine.  This walkthrough is intended to get you through the process of setting up remote debugging and provide helpful troubleshooting tips in case everything does not go smoothly.

For a full list of all setup steps, Chapter 2 in the VSID manual is a must-read (open up Visual Studio 2005 and go to the VMware menu and click Help Topics), but I will try and summarize the main points.  The first thing you will need to do is to make sure that you have the same username and password on your host and guest.  This is a built-in security requirement so not just anyone can remotely debug in your machine; Visual Studio automatically uses your host’s login and password as authentication and if they are different than the guest’s, it won’t work.  It is very strongly recommended that both your host and guest be on the same domain (or if not on a domain, that they both be in the same workgroup), although my own testing has shown me that it is not always necessary.  Anecdotally, I have found that Vista hosts are a little more strict about this than XP hosts, but it is certainly suggested if at all possible.

The next task is to configure the firewall on both your host and guest.  For the guest, we recommend that you disable the firewall.  For the host, I would recommend that at the least, you set Visual Studio as an exception so that the firewall leaves it alone.  If possible, I have found it helpful to disable the firewall on your host as well, because it can sometimes be a bit disruptive.  If you have any other security products running on your host or guest, you should make sure that they are also configured correctly.  For more information about setting up remote debugging, here are some instructions from Microsoft.  This is such a crucial step and I have found that most remote debugging failures come from a host firewall blocking remote debugging.

The final step in setting up remote debugging is changing a few settings in your guest.  Here is a short list:

  1. Disable the security prompts that come from running programs off a network share (which is what happens when the plugin runs your host program in the guest through Shared Folders).  To do this, open up Internet Explorer, choose Tools > Internet Options > Security > Local Intranet, click Sites, click Advanced and add a new Web site "file://*..host" (without the quotes).
  2. Make sure the name of your virtual machine is unique on the domain.  Conflicts like this often happen if virtual machines are cloned and handed around between multiple people.  To change it, go to Start > Control Panel > System > Computer Name and select Change.  On Windows 2000, the tab is named "Network Identification" instead of "Computer Name".
  3. For Windows XP virtual machines, change the guest user authentication scheme to Classic.  To do this, go to the Control Panel > Administrative Tools > Local Security Policy > Local Policies > Security Options page and set "Network access: Sharing and security model for local accounts" to "Classic – local users authenticated as themselves".  Windows XP Home does not have this setting and does not respect it even if you set the corresponding registry key, so that’s why we don’t support debugging in XP Home guests.
  4. If you are running a Windows 98 VM, you will need to perform a couple of extra steps.  Take a look at the VSID user manual for the list.

Now that you have your virtual machine set up for remote debugging, it’s time to set up your Visual Studio project.  Thankfully, this is much easier.  If the VSID plugin has been installed in Visual Studio 2005, you should see an extra toolbar and a VMware menu.  To configure your project, open up your solution in Visual Studio and go to the VMware menu and click Options.  To set up your project for remote debugging enter the following pieces of information:

  1. In the Command space, enter the full path to the executable that you want to debug in the guest.  If it is a path on your host machine, make sure "Run Command as" is set to "a host path through a shared folder".  If it is a path in your virtual machine, make sure that "Run Command as" is set to "a guest path".
  2. Verify that the Remote Debug Monitor path is set correctly.  If you are debugging a 32-bit application, it should be the path to the 32-bit remote debug monitor.  If you are debugging a 64-bit application, it should be the path to the 64-bit remote debug monitor.  Typically, the 32-bit remote debug monitor is installed in %Program Files%\Microsoft Visual Studio 8\Common7\IDE\Remote Debugger\x86\msvsmon.exe.
  3. Click "Virtual Machine" in the left column and enter the path to the virtual machine in the "Virtual Machine" space of that page.
  4. Look through any other options that you would like to specify.  However, all other settings are optional or already have a default specified.

One of the configuration pages:

Configpage_2

If all goes well, you should be able to start debugging in your virtual machine by hitting VMware > Start.  If you get an error message, the unfortunate situation is that the Visual Studio extensibility framework doesn’t give us any information about why debugging failed, so you are left to go through more official routes to get that information.

The first reason why debugging often fails is that your virtual machine is not configured to run your application properly.  If your application requires the .NET Framework to be installed, make sure that it is installed.  You should also make sure that all the dll’s you need are registered in the guest and any other configurations are all correct.  To verify that your application is configured properly, simply go to the VMware menu in Visual Studio and select Start Without Debugging.  If your application runs properly, then you don’t have any of these problems.  But often times you will see an error message complaining that a required component is missing.  This is probably the reason why remote debugging is failing, because Visual Studio is trying to debug the program in your guest but the program can’t even run successfully.

If you are still running into problems, the next step is to manually remotely debug in your virtual machine using Visual Studio’s built-in framework.  The reason why this is important is that Visual Studio will give you much more descriptive error messages than the extensibility framework gives to the plugin.  Here are the steps to set it up using a C/C++ project:

  1. With your virtual machine running, go to Workstation, open up the VM Settings editor, select the Options tab, and select Shared Folders.  Enable Shared Folders and add two shared folders: 1) Add the folder your application is being compiled in on your host machine; name it "Project". 2) Add the folder where the remote debug monitor is on your host machine; name it "RDM".
  2. Go into your guest and open up ..host\Shared Folders\RDM\msvsmon.exe.  This is the equivalent of running your host’s remote debug monitor in your guest.
  3. Open up the Project menu and select Properties.  Open up the Configuration Properties node on the left and select Debugging.  Now write down any existing debugging settings you have, because we may be overwriting these.
  4. Under "Debugger to launch:" select Remote Windows Debugger.  For "Remote Command", enter the path to your host executable using the Shared Folder you just created.  This will be \\.host\Shared Folders\Project\<executable name>.exe
  5. Change "Remote Server Name" to the name of the server that the remote debug monitor created in your guest.  To find this, look in your guest at the remote debug monitor; it should say something like "msvsmon started a new server named ‘grossag@foo’. Waiting for new connections."  In this case, the remote server name is grossag@foo
  6. Now go to Start > Debug.  If remotely debugging through the VMware plugin wasn’t working, then this should fail as well and give you a better error message.

If debugging fails with the following error, you should follow the troubleshooting tips above.

Error

Well that’s about it for me.  Good luck with remote debugging and feel free to post any questions that you have!  And I almost forgot that if you are looking for Visual Studio 2008 support, feel free to download our new Workstation 6.5 beta, although I will warn you that the current beta (build 84113) doesn’t work well with debugging in Windows 2000 guests.