How Airflow Orchestrates Data Pipelines
Airflow is a workflow orchestration platform that schedules, monitors, and manages data pipelines by defining tasks as code, executing them in dependency order, and tracking their success or failure.
Loading…
Topic
Airflow
10 posts about Airflow. Every summary links to the original.
- 10 posts
- 8 companies
- 1 explainer
See the idea
Airflow explainers
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
October 2026 newsletter
The October 2026 ClickHouse newsletter announces CostBench results where ClickHouse Cloud delivered 752× better performance per dollar than Databricks and 438× to 512× better than BigQuery on real-time analytics workloads. The 26.9 release extends LIMIT with AFTER and UNTIL boundary conditions, adds incremental refresh for materialized views, and introduces CREATE TOKEN for time-limited credentials. Executable user-defined functions are now generally available on ClickHouse Cloud, supporting precompiled Rust, Go, C++, and JavaScript binaries with outbound network access and a deterministic flag. Featured community member Hoang Minh V has had over 100 pull requests merged since May, including a fast path for PromQL aggregations and fixes to TimeSeries reads. The newsletter also covers the official Apache Airflow provider for ClickHouse, the chDB durable layer for agent memory, and a Postgres round-up on memory safety and local NVMe storage.
The official ClickHouse provider for Apache Airflow is now available
Apache Airflow users can now use an upstream ClickHouse provider, offering an officially maintained alternative to community plugins or custom integration code. The provider, apache-airflow-providers-clickhousedb, uses ClickHouse Connect over HTTP(S), integrates with Airflow’s common SQL framework, and supplies ClickHouseHook for bulk inserts, streaming, and client-specific operations. The post documents installation, connection setup, standard SQLExecuteQueryOperator DAGs, bulk insertion, and local or managed deployment through Astronomer, including TLS settings for ClickHouse Cloud. Production examples include Astronomer’s Astro Observe, Chartmetric’s 15-million-row playlist pipeline every five minutes, and ClickHouse’s warehouse, which schedules 76 DAGs across more than 40 sources and moves about 6 billion rows daily. The provider is available now, with future capabilities guided by community requests.
Refreshing the Travel-Time Map Behind Lyft’s Marketplace: Rebuilding Neighborhood Reachability…
Lyft’s Neighborhood Reachability Signals dataset supplies precomputed travel-time estimates between geohash-6 cells and a vocabulary of usable cells for pricing, driver heatmaps, and marketplace features. Production ETAs had remained largely based on 2018–2019 data, systematically underestimating travel times and omitting genuinely matchable pairs, while consumers maintained separate allowlists and denylists. The FORTOP team redesigned the Airflow-generated files by rebuilding the geohash universe from demand and supply history, estimating dense pairs from trip history, using Routing Simulation System (RSS) fallbacks for sparse pairs, and blending partial observations. In the second half of 2025, the refreshed dataset was adopted by Pricing after validation, with reported improvements in accuracy and completeness, a six-month refresh cadence, and plans for nine time-category files per region to represent traffic patterns.
Three principles for building a vector platform at Thumbtack
Thumbtack describes a vector platform that lets ML engineers stand up production vector search without separately negotiating database access, building ETL, or writing a query service. The design uses an existing Aurora Postgres cluster with pgvector, while Airflow jobs generate embeddings into BigQuery and managed reverse-ETL syncs them into vector tables; a query API validates inputs and returns ranked results. Three principles guide the layers: reuse trusted infrastructure, treat embeddings as ordinary data flowing through established pipelines, and lower adoption costs for subsequent teams. Production examples include semantic customer-support escalation using cosine similarity and an internal knowledge assistant combining vector retrieval, often with keyword search, before a language model responds; planned directions include self-serve provisioning, vector-backed feature retrieval, and framework-neutral agent memory.
Enabling Multi-Region Data Systems at Notion
Notion’s data residency work keeps customer data processed and stored in the region where it was generated, supporting EU requirements across Search, analytics, scheduled events, and AI. To make this possible, Notion modularized infrastructure around isolated private networks and workspace IDs, then deployed regional data lakes, Kafka clusters, Spark jobs, Elasticsearch, and vector databases. Debezium captures Postgres changes into regional Kafka, while Airflow remains a centralized orchestration layer that routes jobs to regional EMR clusters; event logging similarly routes traffic by workspace and sanitizes data before Snowflake. Centralized mapping tables, shared region helpers, Terraform, and localized sanitization let teams retain a unified Snowflake environment without exporting customer data. The work took months, and Notion says it is applying these patterns to Japan and South Korea while aiming to provision future regional data stacks within days rather than months.
Reducing experiment duration with predicted control variates
Etsy describes how its experimentation platform evolved from CUPED to CUPAC to shorten the time needed to reach statistically powered results. CUPED uses pre-experiment visitor data as covariates in linear regression, while CUPAC first uses a nonlinear model to predict the outcome and then uses that prediction in the variance-reduction correction. Etsy trained and tuned LightGBM models on more than 100 pre-experiment features in Vertex AI, generating predictions through parallel Dataflow jobs and applying the correction with Spark within an Airflow-orchestrated pipeline. CUPED reduced variance by 7% on average, whereas CUPAC achieved 27%, shortening average experiment duration by almost three additional days, although results ranged from 2% to 77% depending on the metric and experiment; Etsy is balancing CUPAC’s impact against its training and maintenance costs.
Building an Enterprise Data Warehouse on Heroku: From Complex ETL to Seamless Salesforce Integration
The article describes Heroku’s effort to unify Salesforce and data from more than 20 applications in a real-time analytics warehouse after traditional ETL created API, freshness, performance, and maintenance problems. Its architecture uses Heroku Postgres as an operational warehouse and staging layer, AWS Redshift for historical analytics, Heroku Connect for direct Salesforce replication, follower databases for read-only source access, and Apache Airflow for orchestration. Template-driven Python jobs and 200+ JSON definitions support incremental, bulk, and date-range loads, while 30+ DAGs and monitoring across 12 regional Heroku Connect instances provide automated retries and consistency. Reported outcomes include over 10 TB processed monthly, sub-minute freshness, 99.99% pipeline uptime, 60% lower integration costs, and 40% better customer analysis accuracy. Tableau and Dataclips provide real-time, historical, and ad hoc reporting, while planned or newly described capabilities include Postgres Advanced, expanded Connect synchronization, and zero-copy Data Cloud ingestion in beta.
Building Etsy Buyer Profiles with LLMs
Etsy is experimenting with large language models (LLMs) to create anonymous buyer profiles from browsing and purchasing behavior, capturing interests such as styles, categories, and shopping missions. The pipeline combines recent searches, item views, purchases, favorites, session data, and listing information from internal feature-store and BigQuery sources, then prompts an LLM to generate structured profiles with explanations and confidence information. To scale processing, Etsy moved listing data to clustered and partitioned BigQuery tables, shortened input history from two years to nine months, increased batch sizes and compute, added managed concurrency, and used Airflow to stagger tasks. For 10 million users, these changes reduced generation time from 21 days to three and cut estimated cost per million users by 94%; experiments apply profiles to query rewriting and refinement pills.
Improving Hugo stability and addressing oncall challenges through automation
Hugo, Grab’s data-ingestion platform for a data lake with more than 4,000 user-onboarded pipelines, faces failures that can leave users with limited visibility and on-call teams handling ad-hoc diagnosis, unclear resolution steps, and DPIs without clear RCA. The proposed automation architecture combines Signal, Diagnosis, RCA Table, Auto-resolution, Data Health API, and Data Health Workbench modules to monitor table health, identify root causes and assignees, execute fixes, and expose status to users and other platforms. Signals come from Airflow failure callbacks and Genchi’s SLA and data-completeness checks; diagnosis avoids parsing extensive logs, runs analysers in parallel according to signal type, and supports asynchronous custom handlers such as retry with backoff for transient errors. Reported outcomes include improved data visibility, faster issue resolution, reduced Hugo on-call workload, scalable handling of transient failures, and more precise data-contract incident management, while next steps include runtime optimisation, richer workbench details, and Flink diagnostics.
How Ramp Accelerated Machine Learning Development to Simplify Finance
Ramp describes how its machine-learning organization used Metaflow to shorten the path from prototype to production for models spanning credit risk, fraud, growth, product, and Ramp Intelligence. Its initial riskiness model exposed long vendor-managed jobs, weak logging, limited Docker support, and substantial platform friction; the model took months to build. Metaflow, with developer-experience improvements, lets teams define flows in Python, run them locally or on AWS Batch, visualize results with cards, and connect production execution to Step Functions and Airflow. Ramp chose AWS-managed infrastructure, initially using Fargate before encountering startup and resource constraints, and later built a MetaflowOperator to simplify Airflow triggering and log access. After adoption, Ramp shipped eight additional models in ten months, recorded more than 6,000 Flow runs, and reports that data scientists can largely self-service while platform engineers spend less time debugging infrastructure.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.