By Joe Carvalho

This is a repost from Joe’s personal blog, Thoughts and Trends.

I want to cover something important: how can a VMware vCloud Hybrid Service customer deploy intrusion detection and prevention technology? There are many ways, but in this post I’ll go into detail on deploying Trend Micro’s Deep Security 9.0 solution in a self-service manner.

Trend Micro’s Deep Security 9.0 product provides many options for protecting cloud servers, usually referred to as “protection modules.” Some of these include “anti-malware,” “web-reputation,” “firewall,” “intrusion prevention,” “integrity monitoring,” and “log inspection” protection modules. We are going to focus on the “intrusion prevention” module in this post.

There are two ways to deploy Trend Micro’s Deep Security 9.0 product in VMware vSphere environments: one is an agent-based protection model, while the other is agentless. This post assumes the cloud tenant wants to deploy their own Trend Micro Deep Security solution, so we are going to cover the agent-based protection model. One of the benefits of the agent-based approach is that Deep Security (agent-based) can be deployed within your vCD Organization Virtual Data Center like any other software. Here is a high level diagram:


The agentless protection model is much different, and is typically delivered as a service from the cloud provider. I highly recommend you read through Trend Micro’s Deep Security documentation.

Before we begin, I want to highlight this post assumes the following:

  • You are familiar with Puppet (Puppet Labs)
  • You are familiar with Microsoft SQL Server 2008 (including installation/configuration)
  • You are comfortable with the vCD UI (Organization Administrator view)
  • You are comfortable with Linux
  • DNS and NTP is running in your environment

To start, I have a single vCHS cloud instance running. I created two vApps inside my Org vDC: a “Trend Micro Deep Security” vApp and a “Test” vApp.  The Trend Micro Deep Security software packages I used are also listed here.

The “Trend Micro” vApp has four VM’s in it:

  1. Deep Security Database (Windows 2008 Server Standard R2 64-bit, MS SQL 2008)
  2. Deep Security Manager
  • Red Hat Enterprise Linux 6.2 64-bit
  • Deep security Manager 9.0 (
  • Deep Security Relay 9.0 (Relay-RedHat_EL6-9.0.0-2008.x86_64.rpm)
  • Deep Security Windows agent (Agent-Windows-9.0.0-2014.x86_64.msi)
  • Deep Security Red Hat agent (Agent-RedHat_EL6-9.0.0-2008.x86_64.rpm)
  1. DNS/NTP server (CentOS 6.4, BIND 9.8, NTP server)
  2. Puppet Master (CentOS 6.4, Puppet Server 3.4)

The “Test Systems” vApp has three VM’s in it:

1. Safe Linux VM (CentOS 6.4, Deep Security Agent)

2. Safe Windows VM (Windows Server 2008 Standard R2 64-bit)

3. Test VM (CentOS 6.4, nmap, Metasploit)

The “Trend Micro” vApp has all the necessary management VM’s for the Deep Security solution, including DNS and NTP.  I also added a Puppet Master, which I consider an absolute necessity in any critical environment. The “Test Systems” vApp includes a Windows and Linux VM, both protected by Trend Micro Deep Security, and a test VM, loaded with some security tools to generate reconnaissance scans and vulnerability exploits against the protected VM’s. Here are a few screenshots of my vApps:

vapps vapp-vm-deepsecurity

vapp-vm-testsystems For the Deep Security Database server, install Microsoft SQL Server 2008 R2. I prefer to install it using Puppet. Take a look at the MSSQL module out on Puppet Forge and adjust based on your needs. To install it, simply run the following on your Puppet node:

[jcarvalho@puppetm]# puppet module install puppetlabs/mssql –version 0.2.0

Here are the MSSQL settings I used (see “class mssql” below). Of course, the passwords below are made up. Adjust your Puppet MSSQL class based on your needs. If you prefer to install manually, that works too.

class mssql (

# See

$media = ‘D:\\’,

$instancename = ‘MSSQLSERVER’,

$features = ‘SQLEngine,RS,Tools’,

$agtsvcaccount = ‘SQLAGTSVC’,

$agtsvcpassword = ‘Sql!@gt#2008′,

$rssvcaccount = ‘SQLRSSVC’,

$rssvcpassword = ‘Sql!Rs#2008′,

$sqlsvcaccount = ‘SQLSVC’,

$sqlsvcpassword = ‘Sql!#2008′,

$instancedir = “D:\\Program Files\\Microsoft SQL Server”,

$ascollation = ‘Latin1_General_CI_AS’,

$sqlcollation = ‘SQL_Latin1_General_CP1_CI_AS’,

$admin = ‘Administrator’

) {

User {

ensure => present,

before => Exec[‘install_mssql2008’],


user { ‘SQLAGTSVC’:

comment => ‘SQL 2008 Agent Service.’,

password => $agtsvcpassword,


user { ‘SQLRSSVC’:

comment => ‘SQL 2008 Report Service.’,

password => $rssvcpassword,


user { ‘SQLSVC’:

comment => ‘SQL 2008 Service.’,

groups => ‘Administrators’,

password => $sqlsvcpassword,


file { ‘C:\sql2008install.ini’:

content => template(‘mssql/config.ini.erb’),


dism { ‘NetFx3′:

ensure => present,


exec { ‘install_mssql2008′:

command => “${media}\\setup.exe /Action=Install /IACCEPTSQLSERVERLICENSETERMS /QS /CONFIGURATIONFILE=C:\\sql2008install.ini /SQLSVCPASSWORD=\”${sqlsvcpassword}\” /AGTSVCPASSWORD=\”${agtsvcpassword}\” /RSSVCPASSWORD=\”${rssvcpassword}\””,

cwd => $media,

path => $media,

logoutput => true,

creates => $instancedir,

timeout => 1200,

require => [ File[‘C:\sql2008install.ini’],

Dism[‘NetFx3’] ],



Once the Deep Security Database server is installed, launch Microsoft’s “SQL Server Management Studio” utility and connect to the SQL server. We need to create a database and database user account for Deep Security manager. I created a database named “dsm” (Deep Security Manager) and a user named “dsmuser,” which uses SQL server authentication. In addition, I modified the server-wide security privileges of this user and assigned the DB owner role for the “dsm” database. Finally, I refined the rights even further, giving the database user “dsmuser” the ability to modify the schema and access the data. Here are the screen shots:

dsmuser dsmuser-second-level dsmuser-third-level


Now that our database is ready, let’s install and configure Trend Micro’s Deep Security Manager. To make things easy, copy all of your installation packages to the same directory on the Deep Security Manager virtual machine. The installer checks for the “Deep Security Relay” package and agent packages during the Deep Security Manager installation. If a “Deep Security Relay” is found, it will give you the option of installing the Relay along with the Deep Security Manager. I copied the following packages to “/root/dsm” on the Deep Security Manager virtual machine:

  2. Relay-RedHat_EL6-9.0.0-2008.x86_64.rpm
  3. Agent-RedHat_EL6-9.0.0-2008.x86_64.rpm
  4. Agent-Windows-9.0.0-2014.x86_64.msi

The command line installation of the Deep Security Manager is simplified through the use of a Java properties file. This is known as a silent install. I have the following settings in my properties file:




DatabaseScreen.DatabaseType=Microsoft SQL Server






Here is the install command:

[root@dsmgr]# ./ -q -console -varfile propertiesfile

If you want to compare the console output details of my install to yours, here it is:

Unpacking JRE …

Starting Installer …

Stopping Trend Micro Deep Security Manager Service…

Jan 2, 2014 7:13:34 AM java.util.prefs.FileSystemPreferences$2 run

java.util.prefs.FileSystemPreferences$2 run

INFO: Created system preferences directory in java.home

Detecting previous versions of Trend Micro Deep Security Manager…

Upgrade Verification Screen settings accepted…

Database Screen settings accepted…

License Screen settings accepted…

Address And Ports Screen settings accepted…

Credentials Screen settings accepted…

Security Update Screen settings accepted…

Relay Screen settings accepted…

Smart Protection Network Screen settings accepted…

All settings accepted, ready to execute…

Extracting files…

Downloading …

Extracting files…

Setting Up…

Connecting to the Database…

Creating the Database Schema…

Creating admin Account…

Recording Settings…

Creating Temporary Directory…

Installing Reports…

Installing Modules and Plug-ins…

Creating Help System…

Validating and Applying Activation Codes…

Configure Localizable Settings…

Setting Default Password Policy…

Creating Scheduled Tasks…

Creating Asset Importance Entries…

Creating Auditor Role…


Importing Software Packages…

Importing Software Package: Agent-RedHat_EL6-9.0.0-2008.x86_64.rpm

Importing Software Package: Agent-Windows-9.0.0-2014.x86_64.msi

Importing Software Package: Relay-RedHat_EL6-9.0.0-2008.x86_64.rpm

Configuring Relay For Install…

Importing Performance Profiles…

Recording Installation…

Clearing Sessions…

Creating Properties File…

Creating Shortcut…

Configuring SSL…

Configuring Service…

Configuring Java Security…

Configuring Java Logging…

Cleaning Up…

Starting Deep Security Manager…

Finishing installation…

By default, Trend Micro’s Deep Security web management console listens on HTTPS port 4119. Verify Deep Security Manager is running and that it’s listening on the correct port:

[root@dsmgr ~]# netstat -tulpn | grep 4119

tcp        0      0 :::4119                     :::*                        LISTEN      1722/java

[root@dsmgr ~]# ls -l /proc/1722/exe</pre>

lrwxrwxrwx. 1 root root 0 Jan 30 20:08 /proc/1722/exe -> /opt/dsm/jre/bin/java

The only thing left is to deploy the agents on the protected VMs, register them and enable protection. I highly recommend using Puppet to simplify your agent deployment, but for now we will go through the manual process.  For Windows, I did the following:

  1. Copied the installation file “Agent-Windows-9.0.0-2014.x86_64.msi” onto the protected Windows VM (safe-windows-vm)
  2. Double clicked the installation file and ran the installer
  3. Clicked “next” to begin the installation
  4. Read the license agreement, accepted the terms and clicked “next”
  5. Under “custom setup”, selected all the features and clicked “next”
  6. Clicked “install” to continue the installation
  7. Clicked “finish” to complete the installation

Here are the screenshots:

step-1 step-2 step-3 step-4 step-5

For Linux, I did the following:

  1. Copied the installation file “Agent-RedHat_EL6-9.0.0-2008.x86_64.rpm” onto the protected Linux VM (safe-linux-vm)
  2. Used “rpm -i Agent-RedHat_EL6-9.0.0-2008.x86_64.rpm” to install the agent
  3. Verified the agent was running by issuing the command “/etc/init.d/ds_agent status”

Here is the command output:

[root@safe-linux-vm ~]# rpm -i Agent-RedHat_EL6-9.0.0-2008.x86_64.rpm

Loaded dsa_filter module version 2.6.32-71.el6.x86_64 [ OK ]

Starting ds_agent: [ OK ]

[root@safe-linux-vm ~]# /etc/init.d/ds_agent status

ds_agent (pid 1805) is running…

Now that the agent installation is complete, we will use the Deep Security Manager to configure protection on both VMs. You can access the Deep Security Manager web console using the following URL: https://managerhostname:4119. Of course, change “managerhostname” to match your Deep Security Manager hostname. Login using the username and password you supplied in your properties file. For the Windows VM, in the main dashboard, I did the following:

  1. Clicked on the “Computers” tab, clicked “New” in the toolbar and selected “New Computer”
  2. Provided the hostname of my “safe-windows-vm”
  3. For “Policy,” I picked “Windows Server 2008” (Base Policy, Windows, Windows Server 2008)
  4. Click “Next,” then “Finish” to automatically activate the agent

Here are the screenshots:

add-computer-1 add-computer-2 add-computer-3 add-computer-4

For the Linux VM, in the main dashboard, I did the following:

  1. Clicked on the “Computers” tab, clicked “New” in the toolbar and selected “New Computer”
  2. Provided the hostname of my “safe-linux-vm”
  3. For “Policy,” I picked “Linux Server” (Base Policy, Linux Server)
  4. Click “Next,” then “Finish” to automatically activate the agent

Here are the screenshots:

add-computer-1-linux add-computer-2-linux add-computer-3-linux

Once activation is complete, the Windows and Linux VM’s are protected and have a default policy of “Prevent,” meaning “Intrusion Prevention” is enabled.  If you want detection only (Intrusion Detection), you can change the IPS module  behavior. Trend Micro’s Deep Security Intrusion Prevention Module allows granular customization of policies, which can be applied at the base policy, OS specific base policy (example:  Base Policy > Linux Server) or per virtual machine. As an example, to change the IPS module behavior for my Linux VM (safe-linux-vm), I did the following:

  1. In the main dashboard, I clicked on the “Computers” tab
  2. I double clicked on the “safe-linux-vm” object
  3. In the new pop up window, I clicked on “Intrusion Prevention” in the left hand column
  4. In the “General” tab, I clicked on the drop down menu for “Intrusion Prevention State” and selected “On”
  5. In the same “General tab”, I selected the “Detect” radio button
  6. I clicked on “Save” in the bottom right hand corner to enable the policy changes

Here are the screenshots:

ips-behavior-change-linux ips-behavior-change-linux-2 ips-behavior-change-linux-3 ips-behavior-change-linux-4 ips-behavior-change-linux-5

As a final test, we are going to generate some attack traffic to see the Trend Micro Deep Security solution in action. I’m going to use Metasploit running on my test VM to attempt to exploit a MySQL authentication bypass vulnerability (Mitre CVE-2012-2122) against the safe-linux-vm. In addition, I’m going to activate the “Web Client Restrict Executable File Downloads” signature on the safe-windows-vm and try to download an executable file. Finally, I’m going to generate some reconnaissance scans against the safe-windows-vm using NMAP.

Before doing all of this, I made sure to disable the Trend Micro Deep Security Firewall running on both VMs; we don’t want the firewall to block the attack traffic. I also made sure the “Reconnaissance Scan Detection” option was enabled on the safe-windows-vm, which is the default.

Here are the screenshots:firewall-off recon

Starting with the MySQL vulnerability test against the safe-linux-vm, I ran the following using Metasploit:

msf > use auxiliary/scanner/mysql/mysql_authbypass_hashdump

msf auxiliary(mysql_authbypass_hashdump) > set RHOSTS


msf auxiliary(mysql_authbypass_hashdump) > set USERNAME root

USERNAME => root

msf auxiliary(mysql_authbypass_hashdump) > run

[+] The server allows logins, proceeding with bypass test

[*] Authentication bypass is 10% complete

[*] Authentication bypass is 20% complete

[*] Authentication bypass is 30% complete

[*] Authentication bypass is 40% complete

[*] Authentication bypass is 50% complete

[*] Authentication bypass is 60% complete

[*] Authentication bypass is 70% complete

[*] Authentication bypass is 80% complete

[*] Authentication bypass is 90% complete

[*] Authentication bypass is 100% complete

[-] Unable to bypass authentication, this target may not be vulnerable

[*] Scanned 1 of 1 hosts (100% complete)

[*] Auxiliary module execution completed

msf auxiliary(mysql_authbypass_hashdump) >

Here are the screenshots from the safe-linux-vm showing the detected attack:mysql-attack-1 mysql-attack-2

For the Windows test, I attempted to download the Foxit Reader EXE package. Of course, Trend Micro Deep Security blocked the download and generated an IPS event entry. Remember, earlier in the post we set the safe-linux-vm IPS behavior to detect only. For the safe-windows-vm, we left it at the default behavior setting of “Prevent,” which not only blocks the EXE download, but generates an event entry as well.

Here are the screenshots from the safe-windows-vm showing the failed EXE download and the Trend Micro IPS event:

windows-failed-download block-exe-download-1 block-exe-download-2

For our very last test, let’s run a reconnaissance scan against the safe-windows-vm. Using NMAP running on my test VM, I issued the following command:

[root@test-vm ~]# nmap -O -v

recon-scan1 recon-scan2 recon-scan3

This post is an example of how I deployed and tested Trend Micro’s Deep Security product. Take the time to learn the product and leverage its capabilities to meet your needs. It’s a powerful solution that can provide exceptional protection for your vCloud Hybrid Service virtual data center.

Joe Carvalho is a Senior Cloud Architect of vCloud Hybrid Services at VMware where he works on the creation, design, development and testing of VMware’s vCHS Cloud platform and services. In his spare time, Carvalho focuses on emerging technologies, security, scalability, cloud architecture and services, among other topics. Carvalho wrote this post on his blog, “Thoughts and Trends,” to help vCloud customers enhance intrusion detection and prevention. What follows is a guide to deploying Trend Micro’s Deep Security 9.0 solution in your vCloud architecture.

For future updates, follow vCloud on Twitter and Facebook at @vCloud and

For more information about the VMware vCloud Hybrid Service, visit