We are excited to announce a first step in our journey of bringing SaltStack Enterprise into our family of vRealize products via rebranding to vRealize Automation SaltStack Config! I published a blog recently on how to use vRealize Automation to deploy an environment with both a Salt Master and a Salt Minion, you can read that here. However in this blog I would like to highlight some other features of vRealize Automation SaltStack Config, especially around these topics:
- Reactors & Beacons
- State Files
- Orchestration
- Application Deployment
- vRealize Automation Cloud Config
This blog may be particularly helpful for our existing vRealize Automation customers who may not be very familiar with SaltStack and should also help the SaltStack customers understand vRealize Automation a bit more.
In this blog I will take you through a journey by which we will use vRealize Automation and vRealize Automation SaltStack Config together to deploy an application automatically on a deployed virtual machine, then simulate a hack to that application and then show how vRealize Automation SaltStack Config detects the change and restores our application to the proper state.
Reactors & Beacons?
When I mentioned reactors and beacons above I am referring to a feature of vRealize Automation SaltStack Config that provides some unique and powerful features to help with automation and self-healing.
Reactors are basically a mechanism that tell the system what to do when a certain event occurs. For instance, a Salt Minion wants to authenticate with the Salt Master, then auto-accept the salt key if certain criteria are met for example.
Beacons get deployed to minions and look for events, like file changes or operating system changes and report back to the Salt Master and then a reactor tell the system what to do about that change.
Orchestration can be triggered by all of this, as we will see in this blog. Basically a reactor detects some event, then the system can just install an application for instance.
Explanation of State Files
In vRealize Automation SaltStack Config a state file is a declarative file to instruct the system to do something.
State Files help with many various tasks, e.g., what packages(software) needs to be installed, what needs to happen across all Salt Minions (top files), and other tasks that help with orchestration. We will use a variety of state(.sls) files throughout the blog.
So now that we have gotten a few topics covered, let’s get started looking at this, should be fun!
vRealize Automation SaltStack Config Main Event
The main thing I want to show in this blog is how I can deploy a Shopping Cart application so I can run my business and my customers can purchase my products. However what if my website gets hacked! How can my business recover quickly without intervention from an operator.
So first thing I need to do is go into vRealize Automation and create a Cloud Template to declare the config of my virtual machine and then run a script to install the Salt Minion. This can be done by going into vRealize Automation and go to Cloud Assembly and click Design–>Cloud Templates–>New From –> Blank Canvas. Then drag and drop a virtual machine onto the canvas and edit the YAML code so a linux image gets installed and input some cloud-config code to install the minion.
The highlighted code above refers to how to bootstrap the Salt Minion on linux, however there are two lines of code above it that stop the Ubuntu upgrade services. I did this because those update services can interfere with orchestration that occurs later in regards to the application installation. The key will auto accept with Salt-Master without that code, but the rest of the orchestration could be affected.
Now that we have our cloud-template done, we can deploy the virtual machine and use a reactor to auto-accept the key. If you want to know more about auto-accepting keys then refer to my blog here.
So once our Salt-Minion has been registered via the acceptance of the key within vRealize Automation SaltStack Config then we have configured our reactor to look for machines with certain minion-id and then install our Shopping Cart application. Here is an example of our reactor:
The second command ‘salt/minion/*/start’ basically is stating that when a minion starts look at opencart.sls. Lets look at opencart.sls (our opencart orchestration file).
So looking at the opencart.sls file we can see a line at the top that is using “jinja” to filter our minions. Keep in mind the reactor does not do the filtering in our example, the .sls file will filter out minions who have a minion_id that start with “oc-cool”. Then the file runs a function to start the state.orchestrate and look to our directory called opencart (which is located in our base directory).
Ultimately will run another .sls file to perform the next instruction. However in our orchestration file I am just specifying the directory, that is because my next file is a “init.sls” file which gets automatically called when the directory is called, basically “init.sls” is a special file in vRealize Automation SaltStack Config. The init.sls file then call our installation .sls (ocinstall.sls) which has instructions on what needs to be done on the Salt-Minion. Here is an snippet of the ocinstall.sls file:
Update:
When you create the last two states to deploy the beacon file and restart it, you may want to use this code:
# Create Beacon to Protect Index.php
deploy_beacon_file:
file.managed:
- name: /etc/salt/minion.d/beacons.conf
- source: salt://opencart/files/beacons.conf
- makedirs: True
restart_minion:
cmd.run:
- name: service salt-minion restart
- bg: True
- onchanges:
- file: deploy_beacon_file
This file will install some packages, run a script to install our application, then deploy the beacon file onto the Salt-Minion. The packages are prerequisites for the Shopping Cart application. The setup.sh that is being called under the cmd: section contains the script needed to install the application.
The beacon file ultimately is looking for changes made to the index.php file, if that file is modified then the beacon will report that event back to the Salt-Master and then the reactor will know what to do next. If you look at the list of pkgs in the file you will see python-pynotify being one of them, that is a requirement on a Salt-Minion for the beacon to work, but the beacon will not be triggered until we simulate a hack on the index.php file.
Once the application is installed via vRealize Automation SaltStack Config then the website is up and running:
Now I am going to use vRealize Automation Code Stream to ssh into the minion and replace the index.php file with a “hacked” index.php. After my hack the website looks like this!
Then the beacon file should pick up on that change. Here is an example of the beacon file, notice it is look at files – /var/www/opencart/index.php and we are looking for modifications.
The reactor file has been modified to tell vRealize Automation SaltStack Config what to do after the beacon notifies about the change. An example of the reactor code is:
The fix_index_php.sls will instruct the system what to do next. Basically we are going to call another .sls file that is located in a fix_hack directory that has instructions to replace the hacked index.php with our original index.php file. Here is a snippet example of the init.sls that will help us do that.
This file replacement happens automatically without operator intervention and gets our website back to what it should look like. Here is an example of what the activity log looks like when the hack fix is performed within vRealize Automation SaltStack Config.
So we have seen how vRealize Automation SaltStack Config can help organizations continue operations and use powerful features of this technology to help with changes in the environment as well as installing applications automatically based on filter criteria of the salt-minion. We look forward to seeing our customers expand their vRealize Automation use cases with vRealize Automation SaltStack Config and stay tuned for more content and blogs on this topic!
