Apache Iceberg is an open table format for huge analytic datasets. Iceberg adds tables to compute engines including Spark, Trino, PrestoDB, Flink and Hive using a high-performance table format that works just like a SQL table.
Tech tags:
Related shared contents:
-
product2026-05-12
The article discusses the General Availability of Catalog Commits, a significant upgrade that enhances the interoperability of Unity Catalog managed tables with Delta Lake. It highlights how Catalog Commits unify the lakehouse architecture by addressing challenges such as the 'split-brain' problem, multi-engine access sprawl, and the need for multi-table transaction coordination. The authors explain the evolution of Delta Lake and Unity Catalog, emphasizing the importance of a centralized governance model for data access and auditing across various engines. The article concludes with the benefits of enabling Catalog Commits for improved governance and performance in data management.
-
tech12026-06-25
The article discusses the implementation of agentic AI applications using a modern data mesh strategy on AWS. It emphasizes the need for fine-grained access control across multiple data sources to ensure governance and security. The authors outline a serverless architecture that integrates various AWS services, including Amazon S3, AWS Lake Formation, and Amazon Athena, to create a governed data mesh. Key architectural changes are highlighted, including the use of S3 Tables and S3 Vectors for optimized data handling and access control.
-
product2026-07-02
This article discusses the challenges of managing high-volume application logs and how to overcome them using AWS Glue and Apache Iceberg materialized views. It provides a detailed solution for building an application log pipeline that enhances query performance by utilizing pre-computed query results. The architecture leverages several AWS services, including Amazon CloudWatch Logs, AWS Lambda, and Amazon Data Firehose, to create a scalable and efficient data pipeline. The article also outlines deployment steps and best practices for maintaining fast analytics performance on large-scale log data.
-
project2026-03-03
The article details Yggdrasil Gaming's migration from Google BigQuery to an AWS-based lakehouse architecture, highlighting the challenges faced due to multi-cloud operational complexity and the need for a scalable analytics foundation. It outlines the phased approach taken to establish a new architecture using AWS services, including Amazon S3, Apache Iceberg, and Amazon Athena, which enabled real-time data ingestion and advanced analytics capabilities. The migration resulted in significant cost savings, improved data freshness, and enhanced governance for analytics workloads. The article serves as a case study for organizations looking to modernize their data architecture.
-
product2025-11-20
Google Cloud has announced the general availability of Iceberg REST Catalog support in BigLake metastore, enhancing open data interoperability across various data engines. This fully-managed, serverless metastore allows users to query data using their preferred engines, including Apache Spark and BigQuery, without the need for data duplication. The integration with Dataplex Universal Catalog provides comprehensive governance and lineage capabilities. Organizations like Spotify are already leveraging this technology to build modern lakehouse platforms.
-
product2024-12-10
The validation and remediation are interesting.
-
project2025-02-14
-
vision2024-12-20
-
vision2024-12-12
Good summarise the current problem for using Iceberg system, but the new S3 Table looks addressing all these pain points.
-
spike2024-12-03
Leverage of Iceberg table, Data is partitioned and stored in a way that aligns with the join keys, enabling highly efficient joins with minimal data movement for Spark job.
-
tech12024-12-04
S3 Table bucket handle the Iceberg compaction and catalog maintenance tasks for you.
-
project2024-11-22
Improving the data processing efficiency by implementing Apache Iceberg's base-2 file layout for S3.
-
product2024-12-09
Without Iceberg, there are lot of overhead works to implement WAP pattern.
-
spike2024-12-09
-
In productions with: