Why Splunk SmartStore Architecture?
Splunk software helps create hidden value from ever-growing machine data. As data volume increases and decisions are made based on this data, retention periods are getting longer. As a result, organizations are looking for more cost-effective storage solutions to manage their growing Splunk datastores. To address this, Splunk introduced a feature called SmartStore, which offers enhanced storage management functionality. SmartStore allows moving warm data buckets to S3-compatible object stores. Moving the data from expensive indexer storage achieves the following benefits:
- Decouple storage and compute layers
- Elastically scale compute on-demand for search and indexing workloads
- Grow storage independently to accommodate retention requirements
- Cost savings with more flexible storage options
Reference Architecture for Deploying Splunk SmartStore
With an integrated solution combining the traditional vSAN storage with vSAN DPp (Data Persistence platform) and object storage such as Cloudian, we can support hot data (high performance) and warm data (large capacity) to meet the demands of Splunk SmartStore architecture.
The above figure shows the Splunk SmartStore on Cloudian and VMware stack. Here, high-performance vSAN storage is used to deploy hot data buckets, and the warm buckets are moved to an S3-compatible object store. This solution helps move the data from expensive indexer storage to a less expensive S3 object store.
In this reference architecture, we validate that VMware Cloud Foundation with Tanzu can support this solution by deploying a Splunk distributed deployment in a VMware Cloud Foundation workload domain and Cloudian HyperStore on containers (an S3-compatible object storage system).
See the recently published Splunk SmartStore Solution on Cloudian and VMware for details, including deployment recommendations.
