Last year for Workstation 6.0, one of the features we added was a plugin for Visual Studio 2005 that allows developers to debug a project (native and managed C/C++, C#, or Visual Basic) inside of a Windows virtual machine (specifically Windows 98, Windows 2000, and later). All of you Windows developers know how much of a pain it is to support multiple versions of Windows, but this tool is designed to make that process much, much easier. The Visual Studio Integrated Debugger (VSID in short) plugin utilizes Visual Studio remote debugging technology to allow you to debug a project inside of a VM as if you were debugging on your host computer with the click of one button.
The plugin’s toolbar and menu, added to Visual Studio 2005:
The most difficult part of getting all of this to work is the initial setup. Once you can get your configuration working,
it should work well for a long time. But it is certainly a chore getting your host and guest set up to comply with all of the security regulations that Microsoft requires to successfully remotely debug on another machine. This walkthrough is intended to get you through the process of setting up remote debugging and provide helpful troubleshooting tips in case everything does not go smoothly.
For a full list of all setup steps, Chapter 2 in the VSID manual is a must-read (open up Visual Studio 2005 and go to the VMware menu and click Help Topics), but I will try and summarize the main points. The first thing you will need to do is to make sure that you have the same username and password on your host and guest. This is a built-in security requirement so not just anyone can remotely debug in your machine; Visual Studio automatically uses your host’s login and password as authentication and if they are different than the guest’s, it won’t work. It is very strongly recommended that both your host and guest be on the same domain (or if not on a domain, that they both be in the same workgroup), although my own testing has shown me that it is not always necessary. Anecdotally, I have found that Vista hosts are a little more strict about this than XP hosts, but it is certainly suggested if at all possible.
The next task is to configure the firewall on both your host and guest. For the guest, we recommend that you disable the firewall. For the host, I would recommend that at the least, you set Visual Studio as an exception so that the firewall leaves it alone. If possible, I have found it helpful to disable the firewall on your host as well, because it can sometimes be a bit disruptive. If you have any other security products running on your host or guest, you should make sure that they are also configured correctly. For more information about setting up remote debugging, here are some instructions from Microsoft. This is such a crucial step and I have found that most remote debugging failures come from a host firewall blocking remote debugging.
The final step in setting up remote debugging is changing a few settings in your guest. Here is a short list:
- Disable the security prompts that come from running programs off a network share (which is what happens when the plugin runs your host program in the guest through Shared Folders). To do this, open up Internet Explorer, choose Tools > Internet Options > Security > Local Intranet, click Sites, click Advanced and add a new Web site "file://*..host" (without the quotes).
- Make sure the name of your virtual machine is unique on the domain. Conflicts like this often happen if virtual machines are cloned and handed around between multiple people. To change it, go to Start > Control Panel > System > Computer Name and select Change. On Windows 2000, the tab is named "Network Identification" instead of "Computer Name".
- For Windows XP virtual machines, change the guest user authentication scheme to Classic. To do this, go to the Control Panel > Administrative Tools > Local Security Policy > Local Policies > Security Options page and set "Network access: Sharing and security model for local accounts" to "Classic – local users authenticated as themselves". Windows XP Home does not have this setting and does not respect it even if you set the corresponding registry key, so that’s why we don’t support debugging in XP Home guests.
- If you are running a Windows 98 VM, you will need to perform a couple of extra steps. Take a look at the VSID user manual for the list.
Now that you have your virtual machine set up for remote debugging, it’s time to set up your Visual Studio project. Thankfully, this is much easier. If the VSID plugin has been installed in Visual Studio 2005, you should see an extra toolbar and a VMware menu. To configure your project, open up your solution in Visual Studio and go to the VMware menu and click Options. To set up your project for remote debugging enter the following pieces of information:
- In the Command space, enter the full path to the executable that you want to debug in the guest. If it is a path on your host machine, make sure "Run Command as" is set to "a host path through a shared folder". If it is a path in your virtual machine, make sure that "Run Command as" is set to "a guest path".
- Verify that the Remote Debug Monitor path is set correctly. If you are debugging a 32-bit application, it should be the path to the 32-bit remote debug monitor. If you are debugging a 64-bit application, it should be the path to the 64-bit remote debug monitor. Typically, the 32-bit remote debug monitor is installed in %Program Files%\Microsoft Visual Studio 8\Common7\IDE\Remote Debugger\x86\msvsmon.exe.
- Click "Virtual Machine" in the left column and enter the path to the virtual machine in the "Virtual Machine" space of that page.
- Look through any other options that you would like to specify. However, all other settings are optional or already have a default specified.
One of the configuration pages:
If all goes well, you should be able to start debugging in your virtual machine by hitting VMware > Start. If you get an error message, the unfortunate situation is that the Visual Studio extensibility framework doesn’t give us any information about why debugging failed, so you are left to go through more official routes to get that information.
The first reason why debugging often fails is that your virtual machine is not configured to run your application properly. If your application requires the .NET Framework to be installed, make sure that it is installed. You should also make sure that all the dll’s you need are registered in the guest and any other configurations are all correct. To verify that your application is configured properly, simply go to the VMware menu in Visual Studio and select Start Without Debugging. If your application runs properly, then you don’t have any of these problems. But often times you will see an error message complaining that a required component is missing. This is probably the reason why remote debugging is failing, because Visual Studio is trying to debug the program in your guest but the program can’t even run successfully.
If you are still running into problems, the next step is to manually remotely debug in your virtual machine using Visual Studio’s built-in framework. The reason why this is important is that Visual Studio will give you much more descriptive error messages than the extensibility framework gives to the plugin. Here are the steps to set it up using a C/C++ project:
- With your virtual machine running, go to Workstation, open up the VM Settings editor, select the Options tab, and select Shared Folders. Enable Shared Folders and add two shared folders: 1) Add the folder your application is being compiled in on your host machine; name it "Project". 2) Add the folder where the remote debug monitor is on your host machine; name it "RDM".
- Go into your guest and open up ..host\Shared Folders\RDM\msvsmon.exe. This is the equivalent of running your host’s remote debug monitor in your guest.
- Open up the Project menu and select Properties. Open up the Configuration Properties node on the left and select Debugging. Now write down any existing debugging settings you have, because we may be overwriting these.
- Under "Debugger to launch:" select Remote Windows Debugger. For "Remote Command", enter the path to your host executable using the Shared Folder you just created. This will be \\.host\Shared Folders\Project\<executable name>.exe
- Change "Remote Server Name" to the name of the server that the remote debug monitor created in your guest. To find this, look in your guest at the remote debug monitor; it should say something like "msvsmon started a new server named ‘grossag@foo’. Waiting for new connections." In this case, the remote server name is grossag@foo
- Now go to Start > Debug. If remotely debugging through the VMware plugin wasn’t working, then this should fail as well and give you a better error message.
If debugging fails with the following error, you should follow the troubleshooting tips above.
Well that’s about it for me. Good luck with remote debugging and feel free to post any questions that you have! And I almost forgot that if you are looking for Visual Studio 2008 support, feel free to download our new Workstation 6.5 beta, although I will warn you that the current beta (build 84113) doesn’t work well with debugging in Windows 2000 guests.
