Apache Spark

Spark is a unified analytics engine for large-scale data processing. It provides high-level APIs in Scala, Java, Python, and R, and an optimised engine that supports general computation graphs for data analysis.

Web site

Github repository

Tech tags:

• Data transformation
• Data streaming transformation
• SQL to dataset
• Distributed computation

Related shared contents:

• How Amazon Ads uses Iceberg optimizations to accelerate their Spark workload on Amazon S3
  project
  2024-11-22

  Improving the data processing efficiency by implementing Apache Iceberg's base-2 file layout for S3.
• Right-sizing Spark executor memory
  project
  2024-11-06

In productions with:

Uber Airbnb Yelp Funding Circle