VMware TanzuTM GemFire® is one of the distributed in-memory database implementations based on the open source software Apache GeodeTM. With a rich set of Tanzu GemFire features, any system developer can implement individual change data capture methodologies easily.
Here are four ways to implement CDC methodologies with Tanzu GemFire:
1. Timestamps/version numbers/status indicators on rows
In traditional relational database systems, system developers implement this CDC methodology by adding the following columns with each row to capture data changes:
- Timestamps: To log when the individual data is changed
- Version numbers: To record the actual version of the individual data, e.g., version 1.0, version 1.1 and so on
- Status indicators: To check the current status of the individual data, e.g., indicating whether the data is changed or not
Tanzu GemFire is based on key-value entry stores called regions, which are like simple tables that have key column and value column only in the context of traditional relational database systems. However, Tanzu GemFire can store any type of data as key and value in each region, such as simple string, numeric number, JSON document, complex Java object, etc.
One way to implement this CDC methodology with Tanzu GemFire is to prepare the following JSON files and store them as values on Tanzu GemFire.
2. Triggers on tables
In traditional relational database systems, system developers implement triggers on tables by storing data change histories into the dedicated table on demand every time there are updates with the target data tables. Typically, they use triggers called “log trigger mechanism.”
In the case of Tanzu GemFire, if there are any updates with data stored in regions, Tanzu GemFire automatically raises events including its CDC information. System developers can implement this CDC methodology with Tanzu GemFire by writing the code to consume those events with the push model. That CDC information is then stored in the dedicated region on demand using plug-in features, such as CacheListener.
3. Event programming
In traditional relational database systems, system developers may implement the event programming methodology using a trigger. For example, the “triggers on tables” methodology identifies data updates and triggers event processing. However, if developers want to achieve more complex and flexible event processing — which is difficult to implement with a trigger based on SQL semantics — they need to introduce additional packages such as Apache Kafka, VMware Tanzu RabbitMQ, etc.
As mentioned in the previous section, Tanzu GemFire automatically raises CDC events and provides plug-in features to consume and process those events, such as CacheListener. System developers can write the code to perform event processing using CacheListener flexibly based on several programming languages, e.g., Java, .NET C# and C++. That said, developers can implement this CDC methodology with Tanzu GemFire without introducing additional packages.
4. Log scanners
In traditional relational database systems, system developers can exploit transaction logs to implement log scanners to scan CDC information. Developers originally used transaction logs to roll back transactions when recovering from database failure situations. The actual format of transaction logs is generally undocumented and could be different depending on relational database implementations, e.g., MySQL, PostgreSQL, etc. Therefore, it’s challenging to exploit transaction logs with a log scanner methodology from scratch. The realistic way to implement this CDC methodology is to introduce additional CDC software solutions to support transaction logs for the target relational database implementation.
Since Tanzu GemFire is a distributed in-memory system, for optimized performance it only supports transactions and not transaction logs. Before committing a transaction, all the updates are stored in memory temporarily. However, Tanzu GemFire automatically raises events while performing transactions and provides plug-in features to consume and process those events, such as Transaction Listener. System developers can write the code to perform event processing using Transaction Listener to implement in a similar way with the log scanner methodology.
Learn from VMware experts
Leverage these four CDC methodologies with Tanzu GemFire to improve the performance of your CDC processing.
As a VMware App Modernization Technical Account Manager (TAM), I help VMware customers with these change data capture methodologies and other ways to get the most from their Tanzu investments. To learn more about how VMware App Modernization Technical Account Management can help your organization, contact your VMware Account Manager.
