AWS Glue is a serverless data integration service.
Tech tags:
Related shared contents:
-
product2026-07-07
The article introduces Apache Spark Connect support in AWS Glue interactive sessions, enabling developers to use local Python environments while executing Spark applications on AWS Glue. This integration allows for enhanced development capabilities, such as debugging and linting, by decoupling the Spark client from the server through a lightweight gRPC protocol. The article also outlines how to connect to AWS Glue Spark Connect sessions using Amazon SageMaker Unified Studio and provides a detailed workflow for managing Spark sessions programmatically. Additionally, it discusses the benefits of serverless execution and monitoring capabilities within the Spark Live UI.
-
tutorial2026-07-15
The article discusses how Salesforce and AWS have collaborated to enable Zero Copy access to Apache Iceberg tables stored in Amazon S3. This integration allows users to query and analyze data across platforms without the need for data replication, thereby enhancing cost efficiency, scalability, and operational agility. It provides a detailed walkthrough for setting up this integration, including prerequisites and configuration steps. The article emphasizes the benefits of real-time data access and streamlined data architecture.
-
tech12026-07-13
The article discusses the challenges and solutions associated with building a multi-cloud lakehouse architecture on AWS specifically for Agentic AI applications. It emphasizes the need for interoperability among diverse data platforms and outlines a high-level architecture that integrates metadata catalogs and context layers to enable AI agents to access enterprise data seamlessly. The article also highlights best practices and trade-offs involved in unifying data access across various environments while ensuring governance and compliance. This is the first part of a series that will delve deeper into implementation and integration strategies.
-
project2026-06-22
The article discusses Avanse Financial Services' migration from an external analytics application to a cloud-native lakehouse architecture using Amazon SageMaker Unified Studio. This transition addressed several operational challenges, including data synchronization bottlenecks, high licensing costs, and limited auditability. By leveraging AWS services, Avanse improved their analytics capabilities, achieving faster report generation and better governance. The article outlines their migration journey, detailing the phases and outcomes of adopting a unified analytics environment.
-
product2026-07-02
This article discusses the challenges of managing high-volume application logs and how to overcome them using AWS Glue and Apache Iceberg materialized views. It provides a detailed solution for building an application log pipeline that enhances query performance by utilizing pre-computed query results. The architecture leverages several AWS services, including Amazon CloudWatch Logs, AWS Lambda, and Amazon Data Firehose, to create a scalable and efficient data pipeline. The article also outlines deployment steps and best practices for maintaining fast analytics performance on large-scale log data.
-
product2026-03-26
The article discusses the implementation of AWS Glue Data Quality pipelines using Terraform, highlighting two methods: ETL-based and Catalog-based Data Quality validation. It explains how these methods can ensure comprehensive data quality across data lakes and pipelines, utilizing a real-world dataset of NYC yellow taxi trips. The article emphasizes the benefits of Infrastructure as Code (IaC) practices for consistent and repeatable deployments, and provides a step-by-step guide for setting up the necessary resources in AWS.
-
project2026-03-03
The article details Yggdrasil Gaming's migration from Google BigQuery to an AWS-based lakehouse architecture, highlighting the challenges faced due to multi-cloud operational complexity and the need for a scalable analytics foundation. It outlines the phased approach taken to establish a new architecture using AWS services, including Amazon S3, Apache Iceberg, and Amazon Athena, which enabled real-time data ingestion and advanced analytics capabilities. The migration resulted in significant cost savings, improved data freshness, and enhanced governance for analytics workloads. The article serves as a case study for organizations looking to modernize their data architecture.
-
project2025-07-22
This Data Processing MCP Server can fully manage all the EMR, Athena and Glue services. You really don't need code any more...
-
spike2025-03-07
-
project2025-02-18
Very classic Glue job pipeline to feed the AWS Bedrock Knowledge Bases for a RAG use case.
-
vision2024-12-20
-
tech12024-12-04
S3 Table bucket handle the Iceberg compaction and catalog maintenance tasks for you.
-
project2024-12-05
Twitch has leveraged Views in their Data Lake to enhance data agility, minimize downtime, and streamline development workflows. By utilizing Views as interfaces to underlying data tables, they've enabled seamless schema modifications, such as column renames and VARCHAR resizing, without necessitating data reprocessing. This approach has facilitated rapid responses to data quality issues and supported efficient ETL processes, contributing to a scalable and adaptable data infrastructure.
-
product2024-12-20
-
product2024-12-09
Without Iceberg, there are lot of overhead works to implement WAP pattern.
-
product2024-12-12
Build a process to built the complete data lineage information by merging the partial lineage generated by dbt automatically.
-
project2022-06-22
-
project2024-10-29
With AWS LakeFormation, creating filter packages and controlling access. A filter package provides a restricted view of a data asset by defining column and row filters on the tables.
-
project2024-11-11
To search user profiles to remove, we use an AWS Lambda function that queries Aurora, DynamoDB, and Athena and places those locations in a DynamoDB table specifically for GDPR requests.
-
product2024-11-22
Classic RAG solution for this kind of application.
-