Updated: 9/18/2024
Editorial Note: this is a living blog featuring important service updates for VMware Cloud on AWS logs. Please check back frequently for the latest updates and additional answered FAQs. As this feature is in active development, all information in this document is subject to change.
What is Changing?
Today, VMware Cloud on AWS (VMC) leverages VMware Aria Operations for Logs SaaS for customers to monitor logs and troubleshoot their SDDC deployments. These logs are accessible via the Aria Operations for Logs SaaS Console and can subsequently be forwarded to other logging solutions for further analysis and archiving.
With the End of Availability (EOA) of the Aria SaaS portfolio of products including VMware Aria Operations for Logs SaaS, the VMC team is ending the existing integration of Aria Operations for Logs SaaS with VMC. To enable continued customer access to SDDC logs, VMC will provide customers with access to their SDDC logs from Amazon Simple Storage Service (Amazon S3), and customers must provide their own logging tool for collecting and processing those logs.
What is the Go Forward Plan?
Going forward, VMware Cloud on AWS SDDC logs will be deposited into an Amazon S3 bucket dedicated to each customer’s VMC organization. SDDC logs will be delivered to the Amazon S3 bucket automatically, and read-only access to this S3 bucket will be granted to the native AWS account(s) linked to their VMC organization through the Account Linking (CloudFormation) process. With read-only access to their SDDC logs from the S3 bucket, customers can configure a logging tool/solution of their choice that can collect logs from Amazon S3. This tool must be acquired independently from VMware Cloud on AWS for customers to collect, process/store, and analyze SDDC logs.
Once this feature is available, SDDC logs will remain available in the S3 bucket for 48 hours before they expire. It is essential that customers retrieve their logs from the S3 bucket at least every 48 hours as the logs will be permanently removed beyond that time window and will not persist.
Customers will be required to change to a new logging solution that is able to read logs from Amazon S3 prior to December 6, 2024. Specific instructions for customers to access their Amazon S3 bucket will be provided as soon as this information is available. This update is only applicable to the VMC on AWS commercial service.
Frequently Asked Questions
1. What is changing with log consumption for VMware Cloud on AWS?
Broadcom has announced End of Availability (EOA) of Aria SaaS products, including Aria Operations for Logs SaaS. To ensure uninterrupted access to SDDC logs, the VMware Cloud on AWS team is ending the existing integration with Aria Operations for Logs SaaS and providing customers with read-only access to an Amazon S3 bucket for retrieving their SDDC logs. Customers will then need to provide their own solution to collect and process/store those logs.
2. How will customers access their SDDC logs without Aria Operations for Logs SaaS?
The VMware Cloud on AWS service will provide customers with access to SDDC logs via a read-only Amazon S3 bucket, unique for each customer organization and for each region they have an SDDC deployed within their organization. SDDCs deployed in the same region within an organization will share the same S3 bucket, and each SDDC will have its own directory within the regional S3 bucket. VMC will employ a standardized naming convention for buckets and directories to simplify management for customers, described later in this document.
3. Will VMware Cloud on AWS provide a replacement tool to store and analyze SDDC logs?
No, VMC will only provide access to SDDC logs via an Amazon S3 bucket. Customers must provide their own tool to subsequently collect and process/store those SDDC logs from the Amazon S3 bucket. Any logging tool capable of ingesting from Amazon S3 can be used. To ensure uninterrupted access to their SDDC logs, customers must have a logging solution prior to December 6, 2024.
4. How long will logs be retained in the Amazon S3 bucket provided by the VMware Cloud on AWS service?
The S3 bucket will include SDDC logs from the last 48 hours. After 48 hours, older logs will be removed. Customers will need to collect and process/store their SDDC logs from the Amazon S3 bucket by leveraging common commercially available solutions.
5. What specific logs will be delivered to the S3 bucket?
At feature launch, SDDC audit logs generated by ESXi, vCenter, and NSX will be available via the S3 bucket. VMC Activity and Notification logs will be available at a later date.
6. Will existing log data and/or configurations be migrated from Aria Operations for Logs SaaS?
No, the Amazon S3 bucket provided by VMC will include only the most recent SDDC logs from the last 48 hours. For questions related to migrating from Aria Operations for Logs SaaS, please contact your Broadcom team. VMC customers will need to provide their own solution to collect and process/store logs from the Amazon S3 bucket and are responsible for migrating any logs required from Aria Operations for Logs SaaS to that solution, if supported.
7. Will customers manage and control the Amazon S3 bucket for their SDDC logs?
No. Although the Amazon S3 bucket is dedicated per VMC organization, it is managed by VMC and customers are provided with read-only access. The S3 bucket is created in the VMC organization’s managed AWS account and subsequently shared with the customer’s externally linked AWS account(s) for accessing SDDC logs.
8. How will customers get and control user access to the Amazon S3 bucket?
Read-only access to the Amazon S3 bucket will be automatically granted to the native AWS account(s) linked to the customer’s VMC organization through the Account Linking (CloudFormation) process. Customers will be able to control access to the S3 bucket through their AWS IAM policies in their connected account(s) and must further grant access to authenticated IAM users in their connected account(s). Details for managing access for cross-account S3 buckets can be found in AWS public documentation.
9. Can customers extend the log retention time or request additional features?
No, logs will be retained in the S3 bucket for 48 hours to provide VMC customers with enough time to collect and store them through their own logging solution. After 48 hours, older logs will be removed and replaced with the most recent logs. We recommend customers use commercially available solutions to collect, process, and store their SDDC logs from the Amazon S3 bucket. The S3 bucket is provided as is, and VMC will not offer different storage tiers or replication. It is intended only for time-limited access to logs so that customers can collect for consumption via their own tooling.
10. When does this transition to the new logging solution start and when must customers complete their transition from Aria Operations for Logs SaaS?
We expect to share more details for customer access to the new S3 bucket in October, but customers should start identifying their own logging tool now. Customers need to provide their own solution to collect and process/store their SDDC logs from Amazon S3 and complete their transition from Aria Operations for Logs SaaS prior to December 6, 2024.
11. When are log files created and at what intervals?
Log files are created in real-time once they are generated by the SDDC management appliances. As soon as logs are received by the VMC service, they are compressed and pushed to the S3 bucket. For small to medium-sized deployments, logs are pushed as often as every few seconds. The delay from log file generation by the SDDC management appliances and their subsequent push to S3 should be no longer than a few minutes.
12. What is the naming convention and format of the S3 buckets and the log files?
The Amazon S3 bucket will be regional and will follow the naming convention vmw.logs.<<org_id>>.<<region>>. Logs will be provided in JSON format as compressed files. The logs will be contained in SDDC-specific folders, under the directory logs/<<sddc_id>>/<<current_date>>/<<uuid.uuid4()>>_log.ndjson.gz. The <<uuid.uuid4()>> is a random UUID (example 4f54c40d-0699-45d2-9049-2083ae884dae).
The format for <<current_date>> is “yyy/mn/dd” and SDDC logs for the day will fall under this directory. Logs will be pushed to their directory whenever they are generated by the SDDC components for that day. The <<uuid.uuid4()>>_log.ndjson.gz will include logs for every batch of logs sent at a time for that day. Whenever logs are pushed from the SDDC components, a new <<uuid.uuid4()>>_log.ndjson.gz will be created for that batch. This information is subject to change, and any updates will be published prior to the release of the S3 bucket solution.
13. What permissions are granted for the S3 bucket to the linked AWS account?
Permissions granted to the customer’s externally linked AWS account will follow this policy structure:
{
“Effect”: “Allow”,
“Principal”: {
“AWS”: “arn:aws:iam::<linkedacc>:root”
},
“Action”: [
“s3:List*”,
“s3:Get*”
],
“Resource”: [
“arn:aws:s3:::<shadow acc bucket>”,
“arn:aws:s3:::<shadow acc bucket>/*”
]
}
14. When will customers have access to an S3 bucket with their SDDC logs to begin integration activities?
Customer S3 buckets will be enabled in waves, starting in October/November, but this timeline can change. Please check back to this blog for more updates.
15. What if customers have additional questions?
For additional questions related to VMware Cloud on AWS and the change to SDDC logs, please reach out directly to your Broadcom team. For questions specific to Aria Operations for Logs SaaS, please reach out to [email protected].
The post VMware Cloud on AWS – SDDC Logs Update appeared first on VMware Cloud Foundation (VCF) Blog.
