In the previous part of this series, we introduced SaltStack Beacons. In this article we will look at – a feature of SaltStack that allows us to react to the events generated by Beacons.
What are Reactors?
“Reactors” are a feature of SaltStack that allow you to define actions that will be taken when an event is triggered. For example, you can use a Reactor to automatically restart a service when a Beacon detects that it has stopped working, or send an alert to an administrator when an event occurs that a certain memory threshold has been reached. The range of events and the actions that can be triggered by them are numerous, and it is up to your creativity how to react to them.
Together, Beacons and Reactors provide a way to monitor your infrastructure and perform automated actions based on defined conditions. This can help you maintain a stable and reliable environment, so you may quickly respond to potential problems.
How to use a Reactor?
When you decide to use a Reactor to react on events to trigger an action, the first step is to define the desired State. This is not a step you perform in SaltStack, rather it is a planning step, that will help you to implement the self-healing magic of your machines. This means specifying the configuration, software, and other elements that you want to be present or configured in a certain way on the machine.
For example, you might specify that the machine should have a particular service running, a specific set of packages or software installed, or a certain configuration file in place. You can use SaltStack’s declarative configuration language, YAML and Jinja, to define the desired State of the machine in a clear and concise way.
Creating the Reactor State
The declaration of what action to trigger is done in Reactor States. In SaltStack, a “Reactor State” is an action that is triggered when a specific event occurs. A Reactor State can be used to perform a specific action in response to the event created by a Beacon.
Reactor States must be associated with a specific event. When an event is created, the associated Reactor State is automatically executed. In this way, you can automate the process of responding to events or conditions in your infrastructure and maintain a stable and reliable environment.
Linking a Reactor to an Event
The definition of which Reactor should react to which event is done in a Reactor configuration. This configuration can be in a separate file called Reactor.conf or a section within the master configuration wich is located in /etc/salt/ . If you create a separate Reactor.conf file, place it into /etc/salt/master.d
salt-run state.event pretty=true
Once the configuration is updated, restart your salt-master service.
Run “systemctl restart salt-master” on your master server to enable the reactor.
Depending on your Linux distribution, the command to restart the service can be different.
First half of the declaration is done, second is to define which Reactor State must be executed when the event is created. In our example above we trigger the Reactor State “salt://reactor/service.sls”
You are done, we have linked an event to a reactor State. Always make sure to use correct YAML syntax, when creating this configuration. There are free online tools like yamllint, which will help to validate and correct your YAML syntax.
Creating the Reactor State
Our Reactor configuration from above is executing a Reactor State called “service.sls”. Let’s have a look into the file:
{% if data[data['service_name']]['running'] == False and data['service_name'] == 'httpd' %}
start_service:
local.service.start:
- tgt: {{data['id']}}
- arg:
- httpd
{% endif %}
The State file is written in YAML in combination with Jinja syntax.
The State consists of two parts, a wrapping condition and the actual action.
The condition is to ensure that the action is only executed when the httpd service is not running.
Inside the condition is the action that starts the httpd service on the machine that caused the event.
“data” contains the entire event content. With [‘id’] you reference to the attribute ‘id’ inside the event data, which contains the minion name.
In our example the id would be vm-028, which is then used as the target for our action execution.
The Reactor State action is the equivalent to the following command which can be run on the master directly:
salt ‘vm-028’ service.start httpd
How to verify the Reactor is working
This is the final step in making your Reactor work, you need to test if the Beacon you have created is creating the correct event, which is linked to the Reactor State and finally starts a job and finishes the job successfully.
If you have followed the last blog article to create the httpd Beacon, you are good to test now.
- Login to a machine which has your Beacon installed.
- Stop the httpd service
- Monitor the event log on your master, if your event is showing up
- In SaltStack Config UI go to “Activity/Completed”
- You should see an action “service.start” completed successfully.
If you reach this step and you can see a job starts after you have triggered the event, your Beacon and Reactor config is working.
If your Reactor State file has a problem, for example a syntax problem. You might see a job called “runner.state.event” which is failed.
If the raw data, does not help you, check your master status, by logging in to you master and run the command “systemctl status salt-master”. The status command might be different on other operating systems.
By leveraging the event-driven architecture of Beacons and the automated response capabilities of Reactors, administrators can easily automate actions and respond to changes in their infrastructure in no time. With a little bit of setup and configuration, these tools can help save time and improve the overall efficiency of your environment. With proper use, it can be a valuable addition to your IT toolset, helping you to maintain a reliable, stable and secure infrastructure for your organization.
