Airflow

2 posts about Airflow. Every summary links to the original.

Grab · Mar 20, 2025
Improving Hugo stability and addressing oncall challenges through automation
Hugo, Grab’s data-ingestion platform for a data lake with more than 4,000 user-onboarded pipelines, faces failures that can leave users with limited visibility and on-call teams handling ad-hoc diagnosis, unclear resolution steps, and DPIs without clear RCA. The proposed automation architecture combines Signal, Diagnosis, RCA Table, Auto-resolution, Data Health API, and Data Health Workbench modules to monitor table health, identify root causes and assignees, execute fixes, and expose status to users and other platforms. Signals come from Airflow failure callbacks and Genchi’s SLA and data-completeness checks; diagnosis avoids parsing extensive logs, runs analysers in parallel according to signal type, and supports asynchronous custom handlers such as retry with backoff for transient errors. Reported outcomes include improved data visibility, faster issue resolution, reduced Hugo on-call workload, scalable handling of transient failures, and more precise data-contract incident management, while next steps include runtime optimisation, richer workbench details, and Flink diagnostics.
Shuguang Xiang
Ramp · Sep 15, 2023
How Ramp Accelerated Machine Learning Development to Simplify Finance
Ramp describes how its machine-learning organization used Metaflow to shorten the path from prototype to production for models spanning credit risk, fraud, growth, product, and Ramp Intelligence. Its initial riskiness model exposed long vendor-managed jobs, weak logging, limited Docker support, and substantial platform friction; the model took months to build. Metaflow, with developer-experience improvements, lets teams define flows in Python, run them locally or on AWS Batch, visualize results with cards, and connect production execution to Step Functions and Airflow. Ramp chose AWS-managed infrastructure, initially using Fargate before encountering startup and resource constraints, and later built a MetaflowOperator to simplify Airflow triggering and log access. After adoption, Ramp shipped eight additional models in ten months, recorded more than 6,000 Flow runs, and reports that data scientists can largely self-service while platform engineers spend less time debugging infrastructure.
Peyton McCullough

Improving Hugo stability and addressing oncall challenges through automation

Hugo, Grab’s data-ingestion platform for a data lake with more than 4,000 user-onboarded pipelines, faces failures that can leave users with limited visibility and on-call teams handling ad-hoc diagnosis, unclear resolution steps, and DPIs without clear RCA. The proposed automation architecture combines Signal, Diagnosis, RCA Table, Auto-resolution, Data Health API, and Data Health Workbench modules to monitor table health, identify root causes and assignees, execute fixes, and expose status to users and other platforms. Signals come from Airflow failure callbacks and Genchi’s SLA and data-completeness checks; diagnosis avoids parsing extensive logs, runs analysers in parallel according to signal type, and supports asynchronous custom handlers such as retry with backoff for transient errors. Reported outcomes include improved data visibility, faster issue resolution, reduced Hugo on-call workload, scalable handling of transient failures, and more precise data-contract incident management, while next steps include runtime optimisation, richer workbench details, and Flink diagnostics.

Shuguang Xiang

How Ramp Accelerated Machine Learning Development to Simplify Finance

Ramp describes how its machine-learning organization used Metaflow to shorten the path from prototype to production for models spanning credit risk, fraud, growth, product, and Ramp Intelligence. Its initial riskiness model exposed long vendor-managed jobs, weak logging, limited Docker support, and substantial platform friction; the model took months to build. Metaflow, with developer-experience improvements, lets teams define flows in Python, run them locally or on AWS Batch, visualize results with cards, and connect production execution to Step Functions and Airflow. Ramp chose AWS-managed infrastructure, initially using Fargate before encountering startup and resource constraints, and later built a MetaflowOperator to simplify Airflow triggering and log access. After adoption, Ramp shipped eight additional models in ten months, recorded more than 6,000 Flow runs, and reports that data scientists can largely self-service while platform engineers spend less time debugging infrastructure.

Peyton McCullough