Ray Data

Ray Data is a scalable data processing library for ML workloads. It provides flexible and performant APIs for scaling Offline batch inference and Data preprocessing and ingest for ML training. Ray Data uses streaming execution to efficiently process large datasets.

Web site

Tech tags:

• Data transformation
• Data streaming transformation

Related shared contents:

• Paradigm Shifts in Data Processing for the Generative AI Era
  vision
  2024-11-11

  "AI-centric" data processing focuses on preparing and managing large-scale, multimodal datasets efficiently for AI model training, fine-tuning, and deployment, rather than traditional database queries. It involves optimizing computation across heterogeneous resources (CPUs/GPUs), improving data flow efficiency, and enabling scalability—all crucial for building next-generation AI models.

In productions with: