AWS Glue

AWS Glue is a serverless data integration service.

Tech tags:

• Data transformation
• Data streaming transformation
• Data schema

Related shared contents:

• Schema Change Management at Halodoc
  spike
  2025-03-07
• How Formula 1® uses generative AI to accelerate race-day issue resolution
  project
  2025-02-18

  Very classic Glue job pipeline to feed the AWS Bedrock Knowledge Bases for a RAG use case.
• Jumia builds a next-generation data platform with metadata-driven specification frameworks
  vision
  2024-12-20
• A First Look at S3 (Iceberg) Tables
  tech1
  2024-12-04

  S3 Table bucket handle the Iceberg compaction and catalog maintenance tasks for you.
• Views pwn Tables as data interfaces
  project
  2024-12-05

  Twitch has leveraged Views in their Data Lake to enhance data agility, minimize downtime, and streamline development workflows. By utilizing Views as interfaces to underlying data tables, they've enabled seamless schema modifications, such as column renames and VARCHAR resizing, without necessitating data reprocessing. This approach has facilitated rapid responses to data quality issues and supported efficient ETL processes, contributing to a scalable and adaptable data infrastructure.
• Amazon Q data integration adds DataFrame support and in-prompt context-aware job creation
  product
  2024-12-20
• Build Write-Audit-Publish pattern with Apache Iceberg branching and AWS Glue Data Quality
  product
  2024-12-09

  Without Iceberg, there are lot of overhead works to implement WAP pattern.
• Building end-to-end data lineage for one-time and complex queries using Amazon Athena, Amazon Redshift, Amazon Neptune and dbt
  product
  2024-12-12

  Build a process to built the complete data lineage information by merging the partial lineage generated by dbt automatically.
• BMW Cloud Data Hub: A reference implementation of the modern data architecture on AWS
  project
  2022-06-22
• How BMW streamlined data access using AWS Lake Formation fine-grained access control
  project
  2024-10-29

  With AWS LakeFormation, creating filter packages and controlling access. A filter package provides a restricted view of a data asset by defining column and row filters on the tables.
• Building a GDPR compliance solution with Amazon DynamoDB
  project
  2024-11-11

  To search user profiles to remove, we use an AWS Lambda function that queries Aurora, DynamoDB, and Athena and places those locations in a DynamoDB table specifically for GDPR requests.
• Introducing generative AI troubleshooting for Apache Spark in AWS Glue (preview)
  product
  2024-11-22

  Classic RAG solution for this kind of application.

In productions with: