Lakehouse Analytics: Querying & Ingesting Apache Iceberg Data with Tanzu Greenplum

Enterprise data architectures are converging: the traditional line between ‘data warehouse’ and ‘data lake’ is disappearing, not because data is being copied from one into the other, but because the warehouse itself can now operate directly on data sitting in the lake. Apache Iceberg is the key catalyst for this shift, and VMware Tanzu Greenplum embraces this model directly.

With the release of the Platform Extension Framework (PXF) 8, Tanzu Greenplum can read and write Apache Iceberg tables in place, on cloud object storage platforms like Native S3 in VMware vSAN, as if they were native warehouse tables.

The result isn’t about a separate lakehouse ecosystem bolted onto Tanzu Greenplum, it’s about a high-performance MPP SQL engine operating natively on open, cloud-scale storage. Users get elastic storage, open table interoperability, and full analytic performance, with no data duplication and no proprietary lock-in.

Why Apache Iceberg Matters

While traditional data lakes addressed scalable storage needs, they brought distinct operational hurdles, including inconsistent metadata, weak transactional guarantees, brittle partition handling, and restricted concurrency support.

Apache Iceberg changes that. As an open, high-performance table format designed for massive analytic datasets, Iceberg brings database-like table semantics to object storage by introducing:

ACID transactions
Snapshot isolation
Hidden partitioning
Schema evolution
Time travel
Metadata-driven query optimization

By bringing these capabilities to data stored in cloud object storage, Iceberg makes it possible for a warehouse engine to query and manage lake-resident data with the same reliability and structure as native tables, without ever needing to relocate it. This allows cloud object stores like Native S3 in VMware vSAN and Amazon Web Services (AWS) S3 to become the foundation for truly enterprise-grade analytical platforms.

The Challenge: High-Performance SQL on Cloud Object Storage

Object storage is massively scalable and cost-efficient, but it was never designed to behave like a traditional database filesystem. Analytics engines, however, need to solve key challenges such as distributed metadata management, schema reconciliation, efficient predicate pruning, and large-scale scan optimization.

Historically, organizations often copied files from data lakes into proprietary warehouses to achieve acceptable query performance. This is an expensive and cumbersome process – it incurs data duplication, adds ETL complexity, results in higher infrastructure costs, and fragmentation in data governance.

Modern analytics requires something better.

Tanzu Greenplum + PXF 8: Native Iceberg Analytics

PXF 8 allows Tanzu Greenplum to directly access Iceberg tables stored in cloud object stores.This facilitates reading Apache Iceberg metadata and manifests, querying Iceberg tables in place, pushing down filters efficiently, and writing data into Apache Iceberg-managed storage. In short, it empowers organizations to run Tanzu Greenplum’s massively parallel SQL engine directly on open lakehouse architectures.

Instead of requiring data migration into a separate warehouse, the analytical engine now functions seamlessly right on top of the lake. The lakehouse is no longer a separate ecosystem. With PXF 8, Tanzu Greenplum becomes part of it.

Enterprise Benefits of Tanzu Greenplum + Iceberg on S3

Running Tanzu Greenplum natively on Iceberg-backed S3 data isn’t just a technical convenience, it changes how organizations manage cost, governance, and vendor risk. Five benefits stand out.

Breaking Down Data Silos with Open Interoperability

A compelling advantage of this offering is its native compatibility with the broader data ecosystem. Data written by Apache Spark, Apache Flink, and AI/ML pipelines can be queried directly from Tanzu Greenplum without duplication. This shifts your team’s approach: rather than treating the Tanzu Greenplum analytical warehouse as a destination where data must be moved, Tanzu Greenplum becomes a lens through which existing data can be viewed and analyzed in place.

A Future-Proof, Vendor-Neutral Foundation

Iceberg’s open metadata and storage semantics mean organizations are never locked into a single vendor or engine. The freedom to choose the best compute layer for each workload, whether that’s Tanzu Greenplum for SQL analytics, Spark for transformation, or a specialized ML runtime is preserved by design. This flexibility is increasingly critical as the data tool landscape continues to evolve rapidly.

Analytical Scale Without Compromise

Tanzu Greenplum’s massively parallel processing (MPP) architecture ensures that querying Iceberg-backed S3 data isn’t a performance concession,it’s a full-scale analytical experience. Organizations can execute complex, large-scale SQL workloads directly against their lakehouse without sacrificing the speed or reliability they’d expect from a traditional warehouse.

Consistent Governance Across Hybrid Environments

By leveraging Iceberg’s metadata-centric design, organizations gain a uniform framework for data governance throughout hybrid analytics environments. Instead of administering access policies across disconnected tools, security and governance seamlessly follow the dataset itself. This centralized approach ensures robust compliance, simplifies auditing processes, and facilitates secure data sharing at enterprise scale.

Eliminating the ETL Burden

Query and write data where it already lives. PXF 8 expands Greenplum beyond simple lakehouse querying into a complete, bi-directional ELT engine. Data teams can read raw Iceberg datasets in place, execute complex MPP transformations, and append curated data products directly back to object storage using standard INSERT INTO statements. This eliminates intermediate staging files, third-party processing tools, and synchronization delays, redirecting engineering resources from data plumbing to high-value analytics.

Transaction-Level Atomicity Support

Beyond per-table snapshot isolation, PXF 8 delivers complete transaction-level atomicity across complex single and multi-statement operations. Managed seamlessly through the pxf.iceberg_autocommit parameter, multi-statement Greenplum transactions unify staged writes into a single, atomic Iceberg snapshot upon commit. If a transaction aborts or rolls back, all uncommitted data and staging artifacts are instantly cleaned up, eliminating partial writes, guaranteeing uncompromised data integrity, and providing the enterprise-grade reliability your mission-critical workflows demand.

Together, Tanzu Greenplum and Iceberg on S3 represent a practical path toward a modern lakehouse that is open, scalable, cost-efficient, and built to grow with the organization.

Accessing Iceberg Tables with Tanzu Greenplum

Accessing an Iceberg table on S3 with Tanzu Greenplum is a two-part process that supports both high-performance SQL queries and direct data appends: first PXF needs to know where the table’s metadata lives, then Tanzu Greenplum needs a foreign server and foreign table defined against that catalog.

Configuring the Iceberg Catalog

Before Tanzu Greenplum can access an Iceberg table, PXF needs to know where the table’s metadata lives. PXF 8 currently supports two catalog types for this purpose: 1. JDBC Catalog and 2. Hadoop Catalog. Both are configured in an iceberg-site.xml file that sits alongside the rest of the named PXF server’s configuration.

JDBC Catalog

Use a JDBC Catalog when you’d rather store an Iceberg catalog in a relational database, such as Tanzu Greenplum, PostgreSQL, or MySQL. The table metadata and data files continue to reside on S3:

<?xml version="1.0" encoding="UTF-8"?> <configuration> <property> <name>iceberg.catalog</name> <value>default</value> </property> <property> <name>iceberg.catalog.default.type</name> <value>jdbc</value> </property> <property> <name>iceberg.catalog.default.uri</name> <value>jdbc:postgresql://dbhost:5432/iceberg_catalog</value> </property> <property> <name>iceberg.catalog.default.jdbc.user</name> <value>iceberg_user</value> </property> <property> <name>iceberg.catalog.default.jdbc.password</name> <value>changeme</value> </property> <property> <name>iceberg.catalog.default.warehouse</name> <value>s3a://my-bucket/warehouse/iceberg</value> </property> </configuration>

Similarly, users can also configure hadoop catalog type by specifying an S3 warehouse path, a Hadoop Catalog co-locates Iceberg metadata directly with data files in S3, but it is not recommended for production environments due to non-atomic file system updates.

Creating Foreign Servers and Foreign Tables

We start by installing the PXF FDW extension, which automatically registers the iceberg_pxf_fdw foreign data wrapper. Any foreign server or foreign table defined against this wrapper is assumed to point to Iceberg catalogs and tables.

Next, we create a foreign server that tells Tanzu Greenplum which Iceberg configuration to use. The config option references a named PXF server configuration – in our example, iceberg-s3 — which contains the connection details for the Iceberg catalog and its underlying S3 storage. Note that the connector for Iceberg does not require a format option: Iceberg manages data file formats (Parquet and Avro) internally.

-- The 'iceberg-s3' config references a PXF server directory that includes -- both the Iceberg catalog configuration and an s3-site.xml file containing -- the S3 access key and secret key used to reach the underlying object storage. CREATE SERVER pxf_iceberg_s3 FOREIGN DATA WRAPPER iceberg_pxf_fdw OPTIONS (config 'iceberg-s3');

The iceberg-s3 server configuration includes an s3-site.xml file specifying the S3 credentials PXF uses to access the underlying object storage:

<configuration> <property> <name>fs.s3a.access.key</name> <value>YOUR_AWS_ACCESS_KEY_ID</value> </property> <property> <name>fs.s3a.secret.key</name> <value>YOUR_AWS_SECRET_ACCESS_KEY</value> </property> <property> <name>fs.s3a.fast.upload</name> <value>true</value> </property> </configuration>

We continue by establishing a user mapping, which associates a local database user with a specific foreign server. Since the S3 access key and secret key are already supplied through the server’s s3-site.xml configuration (shown below), no additional credentials need to be passed here — the mapping simply binds the current user to the server:

CREATE USER MAPPING FOR CURRENT_USER SERVER pxf_iceberg_s3;

This keeps the credentials visibly tied to the server config (where they actually live) rather than implying they’d appear in the CREATE USER MAPPING statement, which stays credential-free in this setup.

Next, we create the foreign table. You can choose any name for this table. You must define columns and data types that match the Iceberg table’s schema, which must be known in advance. Specify the server (catalog) the table belongs to, and use the resource option to define the remote table’s namespace and name in <namespace>.<table_name> format. For example, to map to the clickstream_events table in the analytics namespace via the pxf_iceberg_s3 server, we would execute the following:

CREATE FOREIGN TABLE ext_clickstream_events ( event_id integer, event_type text, event_time timestamp without time zone, user_id bigint, payload text ) SERVER pxf_iceberg_s3 OPTIONS ( resource 'analytics.clickstream_events' );

Writing and Querying Iceberg Tables via Greenplum SQL

Now you are ready to interact with your S3-backed data lake. First, you can append new records directly into the Iceberg table using standard SQL INSERT INTO statements:

INSERT INTO ext_clickstream_events VALUES (101, 'page_view', '2025-05-15 10:30:00', 45021, '{"page": "/home"}'), (102, 'checkout', '2026-05-15 10:32:15', 38910, '{"cart_id": 9921}');

Once data is appended, run standard SQL directly against your S3-backed data lake to query it:
SELECT event_type, count(*) FROM ext_clickstream_events WHERE event_time >= '2026-01-01' GROUP BY event_type;

Executing this query returns the aggregated output directly from the underlying Iceberg Parquet files on S3:

 event_type | count
------------+-------
 checkout   |     2
(1 row)

Performance Optimization Highlights

PXF doesn’t just blindly pull files over the network; it acts intelligently to optimize resources. This is achieved by applying the following optimization principles:

Column Projection: If the user query only requests event_type and user_id, PXF instructs Iceberg to pull only the specific columns out of the S3 Parquet files. This significantly cuts down network I/O.
Filter Pushdown: Standard logical and mathematical expressions (=, <, >, IN, IS NULL, AND, OR) are processed directly within the Iceberg API layer. These predicates enable precise partition pruning and data file skipping. By pruning irrelevant data blocks early on, this prevents massive amounts of unnecessary data from ever reaching Greenplum.
Hidden Partitioning and Evolution: Iceberg employs hidden partitioning, deriving partition columns from table data using transform functions like year(), month(), day(), hour(), bucket(), or truncate(). These do not need to be explicitly defined in the table schema. As layouts evolve, Iceberg tracks changes transparently. PXF manages these variations seamlessly, eliminating the need for complex query rules for historical data layouts.
Partition Pruning: PXF automatically performs partition pruning, skipping data files for partitions that do not match the query filter.

Modernize your Analytics with Native Apache Iceberg on Tanzu Greenplum

The future of enterprise analytics belongs to open, decoupled architectures. As organizations face exponentially growing datasets, the strategy of continually moving and copying files into proprietary data silos is fundamentally unsustainable.

Tanzu Greenplum with PXF 8 redefines this paradigm. By unlocking Apache Iceberg tables stored on native cloud object stores, it gives enterprises the ultimate architectural freedom: high-performance, massive-scale SQL analytics paired with completely open data interoperability.

Tanzu Greenplum now serves as a high-powered, zero-latency lens into your entire data ecosystem. It is a practical, cost-efficient, and uncompromising path forward for the modern data-driven enterprise.