Real-Time Audio-Video (RTAV) for Horizon View

By Peter Brown, Senior R&D Manager, and Tarique Chowdhury, Software Engineer

For information about what’s new with Horizon Real-Time Audio-Video, read this 2023 blog.

Webcam support in VMware Horizon View is a feature that has frequently been requested.

It is therefore with great pleasure that we are able to bring you Real-Time Audio-Video functionality. This functionality enables the support of webcams and audio-in and audio-out  in Horizon View, without the need to use USB redirection. Analog audio-in devices are also supported!

This blog gives a quick overview of the Real-Time Audio-Video functionality, along with some quick tips and guidance on its use and configuration. There is also a short YouTube video covering similar topics available here: http://youtu.be/mG0ke377FOw.

Why Were Webcams Not Supported?

It has not been feasible to support webcam functionality via USB redirection – mainly due to the bandwidth that webcams consume. For example, most webcams send uncompressed images over USB at a high frame rate. Tests in our lab show that many cameras consume around 60 to 80Mbps! This is bandwidth consumption which obviously does not scale in a VDI environment.

Audio-in functionality that has been supported in Horizon View was always done via USB redirection. The quality of audio using USB redirection (in or out) depends greatly on network conditions and guest load. Some improvements were made recently in this area which will help some devices – see the Knowledge Base article How to improve audio quality when using USB headsets or speakers with a View desktop.  Audio-in using analog-style devices (e.g., with 3.5mm audio jacks) were not supported at all.

Audio-out (for example, playing music in the guest desktop, but hearing the audio from the local client) via PCoIP audio redirection has been supported in Horizon View for some time, and this gives a much higher audio quality than using USB redirection.

However you can’t split a USB audio device such that the audio-out functionality remains local to the client and the audio-in is redirected. So using a USB headset in a VoIP type application required the entire headset to be forwarded to the guest. Again, the audio quality then greatly depends on network/load. More information about USB forwarding can be seen in USB Device Redirection in VMware Horizon View 5.1 and 5.2.

Overview of Real-Time Audio-Video

Real-Time Audio-Video (RTAV) does not forward audio and webcam devices using USB. Instead the devices are left local to the client, and audio/images are pulled from the local devices. The audio/images are then encoded, delivered to the guest virtual machine, and decoded. A virtual webcam and a virtual microphone are installed in the guest virtual machine, which then “play” the received audio/video, and 3^rd-party apps (e.g., Skype, WebEx, Google Hangout) can use these virtual devices.

Audio-out is performed from the standard View Agent audio-out functionality which provides better audio quality than using USB redirection.

Due to the fact the images and audio are being encoded, this results in much better bandwidth usage (more on this later), allowing this feature to scale across many desktops.

RTAV can support:

• Webcams and audio-in at the same time; e.g., for VoIP video-conference-type applications
• Audio-in only (without video); e.g., VoIP-type applications
• Webcam on its own; e.g., for webcam-monitoring-type applications, or user photo-taking, etc.[We would love to hear about any interesting use cases you find for this functionality, as we are sure there will be many interesting things you can do with this!]

Note that RTAV only works when using the PCoIP protocol. It does not work with RDP.

How Do I Enable This Feature?

This feature currently works only on Windows clients, although other clients will add support for RTAV in due course. You will need the Horizon View Windows Client 5.4 or the new Windows Client v2.2 or later. [note, other Client OS may be supported in the future so check back for other blog updates]

In addition on your guest virtual machine, you need a Horizon View 5.2 (or later) View Agent along with the VMware Horizon View 5.2 Feature Pack 2. These can be downloaded from the Horizon View Downloads page. The optional RTAV component can be installed via the Remote Experience Agent, so make sure this is installed!

Making Your First RTAV Call

RTAV functionality is now enabled and activated by default. It is expected that it should just work!

You should not forward the audio or video devices using USB – if the devices are left local to the client, then RTAV can use them. Any devices forwarded via USB to the guest are not available for use by RTAV.

At your guest, you should see a “VMware Virtual Microphone” and a “VMware Virtual Webcam” available as options in your 3^rd-party applications. Make sure these are selected for use.

Simply start a call in your 3^rd-party application, and RTAV should just work.

Ending a call stops the capturing from the audio/video device.

Note that if your camera is in use (e.g., you are in a Skype call) in the client machine, then it will not be available for use in the guest and vice versa. The camera can be shared between applications in the client and guest, but not at the same time.

Which Devices Are Selected from My Client?

You may have multiple audio and video devices available on your client machine. If the video or audio does not come from your expected source device, then you may need to select a specific source device for use.

We believe that all webcams should be supported – however, we have of course not been able to test with every single webcam ever produced, so it is possible that we don’t support some. Do let us know if you find a device that does not work.

We also suspect (but have not tested) that other video source devices may work too. For example, TV tuners?! [We would love to hear about any fun use cases you identify with non-webcam devices!]

In terms of audio devices, any type of audio input device supported by Windows should work. Whether these be analog-in using the older 3.5mm jack plugs, or digital devices such as USB headsets, they should all be usable.

Audio

It is expected that you will have multiple audio input options (e.g., built-in microphone, external USB headset, possibly analog audio-in, and even microphones built into a webcam!). Usually things should just work, but you may want to configure the system to select audio from a specific audio input device.

By default, RTAV will use the “default” audio-in device as specified in Windows Recording Sound Properties.

To select a different source audio device for RTAV:

1. End your current RTAV session if one is active (by stopping a call in the 3^rd-party app).
2. On the client machine, right-click on the speaker icon in Notification Area Icons and select “Recording Devices” (or via Control Panel –> Sound).
3. Right-click on the device you want to use, and select “Set as Default Device”:
4. Now start a new call in the guest, and the new audio-in device should be used. (No need for any Windows restarts!).

If you do not want to change your default client-side settings, it is also possible to configure the View Client to select a specific audio device if it is available. This is termed a “preferred” audio device. If the preferred audio device is not available, then the Windows default device would be used instead.

To configure a preferred device (remember this isn’t necessary and Windows default device selection is probably sufficient):

1. Start and stop a call in your guest 3^rd-party app where your preferred device is connected to your client machine (this generates the necessary log messages).

2. On your client machine, find the log file:

Windows XP
C:\Documents and Settings\username\Local Settings\Application Data\VMware\VDM\Logs\

Windows 7 or Windows 8:
C:\Users\%username%\AppData\Local\VMware\VDM\Logs\

3. Open the latest debug log in a text editor (titled “debug-20XX-XX-XX-XXXXXX.txt”), and search for the following text:

[ViewMMDevRedir] AudioCaptureBase::LogDevEnum

In a log on our system here, we can see something like:

[ViewMMDevRedir] AudioCaptureBase::LogDevEnum – 3 Device(s) found

[ViewMMDevRedir] AudioCaptureBase::LogDevEnum – Index=0   Name=Headset Microphone (Plantronics C620-M)   UserId=vid_047f&pid_aa01&mi_00#7&3361ff59&0&0000   …

[ViewMMDevRedir] AudioCaptureBase::LogDevEnum – Index=1   Name=Rear Input (SoundMAX Integrated Audio)   UserId=func_01&ven_11d4&dev_194a&subsys_10280293&rev_1004#4&cebec42&0&0001 …

[ViewMMDevRedir] AudioCaptureBase::LogDevEnum – Index=2   Name=Microphone (Webcam 200)   UserId=vid_046d&pid_0802&mi_02#7&2b5ee315&0&0002   …

4. Decide which of the devices you want to make the preferred device, and copy the UserId text.As an example, let’s say you wanted to make the microphone in your webcam the default, you would copy the values “vid_046d&pid_0802&mi_02#7&2b5ee315&0&0002”

5. Start the Registry Editor (regedit.exe). Navigate to: HKEY_LOCAL_MACHINE\SOFTWARE\VMware, Inc.\VMware VDM\RTAV

6. Paste the value copied in step #4 into the REG_SZ  value srcAudioInId.

7. Save your changes and exit the registry.

8. Start a new call, and, providing the preferred device is available, then it will be used (regardless of the default device selected in Windows).

Video

Windows does not have a concept of a default webcam. So, if you want to override the one that is selected for you by RTAV, then you need to edit the preferred video device. Again, like audio, if the device is available on your system, then it will be used, but if not, then an alternative device would be selected automatically.

If you want to set a preferred video device, then follow similar steps to the audio:

1. This time, look in the logs for device enumeration with logs looking like
[ViewMMDevRedir] VideoInputBase::LogDevEnum

2. Again, select the UserId for the device you want to use, but this time copy/paste that value into the registry in a field REG_SZ  value srcWCamId.

Configuring the Video Resolution and Frame Rate
(and Therefore Bandwidth!)

The video resolution and frame rate affect the bandwidth of the encoded video stream to the guest desktop. Precise numbers cannot be provided because the actual video image itself has a bearing on the encoded bit rate, so you will need to do some testing in your own environment.

By default, RTAV runs at 320×240 resolution @ 15 frames per second (fps).

If you want, then these can be adjusted up to a maximum value set by the administrator.

As an example of typical bandwidth consumed, we ran some tests here with a typical “office” webcam setting. As mentioned, actual values recorded on your system may vary from this a bit.

All tests were run @ 15fps.

If you increase the frame rate, then the bandwidth will be higher than this, and reducing frame rate will lower it.

Depending on your webcam, you may not be able to encode and send higher resolutions than this at a full 15fps. If you request big resolutions (that your webcam does not support), then client-side scaling is done, and this takes time. In addition, encoding and decoding requires more CPU cycles, and hence the resultant frames per second may be lower than requested. You will see from logs at the end of a call what the resulting frame rate was, so you can run tests to adjust resolution/bandwidth to suit.

During development, we wondered about restricting the maximum frame size to a value where we could be certain that you could achieve the maximum frame rates, however we decided that some customers may have interesting use cases where they would prefer a larger image, but were okay with a much lower frame rate, and therefore we decided to leave these settings adjustable.

Setting Administrator Configuration Limits

Administrators can configure / restrict the use of RTAV via agent-side GPOs. To do this, load the RTAV GPO template vdm_agent_rtav.adm (available for download in the GPO bundle) into the group policy editor.

The GPO provides the following options:

Remember that the above webcam settings are a “maximum” limit, and so the client can select any value up to the maximum. By default, the clients operate at 320×240 (width x height in pixels) at 15fps.

Setting Client-Side Resolution / Frame Rate

It is possible to configure image frame rates and resolution up to the maximum value specified by the administrator. If the value specified in the client exceeds the value set by the admin, then the value will be capped at the admin-set maximum.

To configure these values, start the Registry Editor (regedit.exe). Navigate to:
HKEY_LOCAL_MACHINE\SOFTWARE\VMware, Inc.\VMware VDM\RTAV

Note that if you request an image size that your webcam does not natively support, then client-side scaling is performed to scale to the requested size. This will potentially reduce the actual throughput of frames possible in RTAV, so it is suggested that you select a standard resolution (4:3 or 16:9 ratio) and one that your webcam supports.

Audio quality is not configurable at this time.

Using RTAV with 3^rd-Party Applications

RTAV webcam support requires that the 3^rd-party application supports DirectX-compatible 3^rd-party applications.

Flash-based applications or websites are not currently supported.

We have been testing internally with Skype, Google Hangout, WebEx, and QQ. We are sure there are many other apps too that will work.

[We would love to hear about any other apps you use successfully, or let us know if you come across any that don’t work!]

General

• When we tested RTAV at scale, we assumed that RTAV would not be a primary use case for VDI. We thought that RTAV within VDI would be for occasional use or on some (but not all) desktops concurrently. We ran tests with 10% of virtual machines on a given host running RTAV and 90% running standard VDI workloads. We did not observe any degradation in performance for either set of desktops. However, depending on your use case, 3^rd-party application, and spec of virtual machines, you may need to run your own scaling tests to be confident that the solution will work in a large-scale environment for you.
• If you physically unplug a webcam during a call and then plug it in again, video will stop (but audio should continue). Start a new call to make the webcam ready for use.

Skype

We did make a few observations when testing with Skype that are worth considering:

• Skype seems much happier if you keep the standard 320×240 image resolution.

If you do change resolution on the client side, then ensure you completely restart Skype on the guest –- or strange images may appear!

Skype does not like some resolutions, and if you set those, then no video will be available.

• We also observed that Skype can be quite CPU-intensive. It is recommended that you configure virtual machines with 2 x vCPU if you plan to use Skype. There are also a number of articles on the web that talk to the high CPU usage, and some make suggestions for ways to reduce the usage. Obviously a high CPU load may affect your consolidation ratio of virtual machines on a given host.

Conclusion

Real-Time Audio-Video brings webcam and audio-in support to your Horizon View desktop for the first time. We are very excited by this feature, and we are sure that many of you will be, too. We would love to hear feedback about it, and if you find any interesting use cases, then please do drop us a line!

Check out Part #2 of this blog talking about Linux Client support for RTAV here: http://blogs.vmware.com/euc/2013/11/vmware-horizon-view-real-time-audio-video-rtav-part-2.html

You can follow Pete @pete_brown_wimb