It takes less than 3 minutes to configure a log forwarding trigger to send any number of AWS logs to Log Insight Cloud. If you want to monitor OS and Application logs in AWS you have the option to use the CloudWatch Agent. (Note you can also use the Log Insight agent and forward logs to a Cloud Proxy server). I’ve had some people ask about how to configure the CloudWatch Agent to collect application and OS logs. While this is documented, it’s one of those things where you might have 20 tabs open before you have completed the task! I figured I’d consolidate the information in this post while still including the pertinent links. Once you have these steps down you can automate the process so future deployments automatically are configured with the appropriate settings.
Step 1: Download the Agent
I’ll be installing this on Ubuntu.
wget https://s3.amazonaws.com/amazoncloudwatch-agent/debian/amd64/latest/amazon-cloudwatch-agent.deb
Step 2: Install the Agent
sudo apt install ./amazon-cloudwatch-agent.deb
Step 3: Attach the CloudWatch IAM Role
A. Find your VM(s) in EC2. Select the Name > Actions > Security > Modify IAM Role
B. Click on Create new IAM role
C. Click on Create Role
D. Select EC2 and Click on Next: Permissions
E. Search for CloudWatchAgentServerPolicy, select it, and click on Next: Tags. On the next screen add tags if desired, or simply click on next to review.
F. Name the role CloudWatchAgentRole (or similar) and click on Create Role. (Once you have this role created you can use it on future deployments.)
You should see that the role has been successfully created.
Step 5: Configure the Agent
You can run the agent configuration wizard or simply copy over the appropriate .json. I will run through the wizard to create the file and will copy the file for future deployments. For this deployment, I only want to collect logs and not metrics. Select the appropriate options for your specific requirements. You can view all the options here.
Run the configuration wizard:
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard
=============================================================
= Welcome to the AWS CloudWatch Agent Configuration Manager =
=============================================================
On which OS are you planning to use the agent?
1. linux
2. windows
3. darwin
default choice: [1]:
1
Trying to fetch the default region based on ec2 metadata...
Are you using EC2 or On-Premises hosts?
1. EC2
2. On-Premises
default choice: [1]:
1
Which user are you planning to run the agent?
1. root
2. cwagent
3. others
default choice: [1]:
1
Do you want to turn on StatsD daemon?
1. yes
2. no
default choice: [1]:
2
Do you want to monitor metrics from CollectD?
1. yes
2. no
default choice: [1]:
2
Do you want to monitor any host metrics? e.g. CPU, memory, etc.
1. yes
2. no
default choice: [1]:
2
Do you have any existing CloudWatch Log Agent (http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AgentReference.html) configuration file to import for migration?
1. yes
2. no
default choice: [2]:
2
Do you want to monitor any log files?
1. yes
2. no
default choice: [1]:
1
Log file path:
/var/log/apache2/**.log
Log group name:
default choice: [**.log]
moad_apache
Log stream name:
default choice: [{instance_id}]
{local_hostname}
Do you want to specify any additional log files to monitor?
1. yes
2. no
default choice: [1]:
1
Log file path:
/var/log/mysql/**.log
Log group name:
default choice: [**.log]
moad_mysql
Log stream name:
default choice: [{instance_id}]
{local_hostname}
Do you want to specify any additional log files to monitor?
1. yes
2. no
default choice: [1]:
1
Log file path:
/var/log/httpd/**.log
Log group name:
default choice: [**.log]
moad_httpd
Log stream name:
default choice: [{instance_id}]
{local_hostname}
Do you want to specify any additional log files to monitor?
1. yes
2. no
default choice: [1]:
2
Saved config file to /opt/aws/amazon-cloudwatch-agent/bin/config.json successfully.
Current config as follows:
{
"agent": {
"run_as_user": "root"
},
"logs": {
"logs_collected": {
"files": {
"collect_list": [
{
"file_path": "/var/log/apache2/**.log",
"log_group_name": "moad_apache",
"log_stream_name": "{local_hostname}"
},
{
"file_path": "/var/log/mysql/**.log",
"log_group_name": "moad_mysql",
"log_stream_name": "{local_hostname}"
},
{
"file_path": "/var/log/httpd/**.log",
"log_group_name": "moad_httpd",
"log_stream_name": "{local_hostname}"
}
]
}
}
}
}
Please check the above content of the config.
The config file is also located at /opt/aws/amazon-cloudwatch-agent/bin/config.json.
Edit it manually if needed.
Do you want to store the config in the SSM parameter store?
1. yes
2. no
default choice: [1]:
2
Program exits now.
I will use the file created to automate the log configuration for 100s of servers. With that in mind, I created a log_group with the name of the application I am monitoring. I am adding all the log file locations I will monitor for the overall application so I don’t have to manage several different config files. I am setting the log_stream_name as the local_hostname so I can easily filter and identify log messages using the server name.
Step 6: Apply the Configuration
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json
Step 7: Restart the Service
sudo systemctl restart amazon-cloudwatch-agent.service
Step 8: Verify Log Flow
If you don’t pre-create the log group, it will not be created until the service starts generating logs. To view the log group and configure retention settings go to CloudWatch > Log groups.
Click on the log group and verify there are messages.
Step 8: Add Log Groups to the vRealize Log Insight Cloud Trigger
If you haven’t previously configured the Lambda function, you’ll need to create it first. The full instructions are located under Log Sources within Log Insight Cloud. I also run through these steps in this video.
Add the newly created log groups to the trigger. Go to Lambda > Functions > Select the function
Once the trigger is created for the log group it will automatically send logs as you add additional agents with this config.
Click on Add Trigger
Select CloudWatch > select the log group > add descriptive filter name. Follow this step for each log group that you created for your application(s).
Step 9: Validate Messages are Flowing to Log Insight
Launch vRealize Log Insight Cloud and filter on the name of the loggroup(s) you created. Note the logstream contains the name of the source VM.
Summary
Once you have the IAM role created and the config.json with the appropriate settings, these can be deployed as part of your template. The logs will automatically be forwarded for all future deployments since the trigger was created during these steps.
If you’re not already using vRealize Log Insight Cloud, check out our free 30 day trial. To learn more about how to use Log Insight Cloud, please check out VMware Pathfinder.
