The GPU-Resident Astronomy Pipeline
A GPU-resident astronomy pipeline keeps every stage of image analysis — load, align, subtract, detect, review — inside GPU memory instead of round-tripping through the CPU.
Loading…
Topic
Data Pipelines
247 posts about Data Pipelines. Every summary links to the original.
- 247 posts
- 25 companies
- 9 explainers
See the idea
Data Pipelines explainers
BigQuery in the Right Lane: Five Checkpoints for AI Data Pipelines
A five-checkpoint workflow for keeping BigQuery in the right lane: land and shape data once, tune partitions and history windows, and federate instead of copying.
How to Make Apache Hudi Copy-On-Write Upserts Less Wasteful
Learn how Apache Hudi copy-on-write upserts can use row-level file and page locations to update changed data while avoiding unnecessary decompression and rewriting.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
How we evaluated the business impact of marketing campaigns
Grab needed a scalable, centralised way to evaluate the business impact of millions of marketing messages sent across expanding campaigns. Manual analysis by marketing analysts produced inconsistent results and failed to scale, leading to the creation of multi-touch attribution and controlled experiment methodologies. The team designed a Marketing attribution model (MAM) that credits touchpoints based on relevance and recency, while using stratified hold-out groups to assess single campaign lift against baseline metrics. Statistical significance calculations, implemented using the jStat library based on the central limit theorem, ensure observed business metric differences like gross merchandise value are not random. The system provides campaign managers with direct access to impact analyses while the team works on challenges surrounding smaller sample sizes and hold-out group constraints.
Ensuring data reliability and observability in risk systems
Grab's in-house risk management platform, GrabDefence, relies on ingesting large volumes of upstream data to drive real-time heuristic rules and data science models. To prevent data discrepancies and missing information from degrading fraud detection, the team implemented a real-time data observability pipeline. The solution uses Apache Flink SQL alongside custom table functions like JSONEXPLOAD to flatten nested JSON structures and aggregate metrics over 5-minute tumbling windows. Processed metrics are sent to Datadog, where counters are grouped by source stream and evaluated using anomaly detection algorithms. Alerts are routed directly to Slack, reducing anomaly detection latency from days or weeks down to within the same day or hour.
Grab Experiment Decision Engine - a Unified Toolkit for Experimentation
Grab's Experimentation platform tests thousands of experimental variants weekly, but analyses were largely handled manually using disparate tools unsuited to multi-sided marketplace nuances. To standardize workflows and eliminate reliability issues, Grab developed the GrabX Decision Engine, an internally open-sourced Python toolkit integrated with its Automated Experiment Analysis application. The framework includes a Trusted Advisor component that automates power analysis and sample size estimation while adjusting for clustered standard errors across metric data types. It also bundles a post-experiment statistical toolbox executing default and custom evaluations, ranging from non-parametric tests to regression models that control for confounders and fixed effects. This unified platform institutionalizes experimentation best practices across teams, streamlining collaboration and ensuring consistent statistical inference.
Iris - Turning observations into actionable insights for enhanced decision making
Standard infrastructure monitoring tools track CPU and memory usage at the host or container level, obscuring the precise resource consumption of individual Spark jobs on shared instances. To achieve granular observability, the Iris platform gathers job metrics directly at the Java Virtual Machine level without requiring changes to user application code. Data collection relies on Uber's JVM Profiler for deep JVM, thread, and memory pool diagnostics alongside sparkMeasure for stage and task execution statistics. Collected metrics route asynchronously through an Apache Kafka queue to avoid execution bottlenecks, feeding a Telegraf, InfluxDB, and Grafana stack for real-time monitoring while archiving to a data lake for offline analytics. This multi-tiered telemetry enables engineering teams to identify over- or under-provisioned jobs and optimize ETL resource allocations.
Enabling near real-time data analytics on the data lake
Traditional data lake setups using Parquet on Hive metastores struggle with frequent updates and long pipeline intervals, introducing significant latency for ad hoc queries. Grab solved this by implementing Apache Hudi to support near real-time analytics across bounded relational databases and unbounded Kafka streams. For high-throughput sources, Flink streams Avro log files to Merge On Read tables and generates compaction plans for asynchronous Spark writers. Low-throughput workloads leverage Copy On Write tables, while relational database sources ingest binlogs via Flink Change Data Capture connectors. This architecture reduced analytics data latency to the minute level without overloading production databases and Kafka clusters.
The journey of building a comprehensive attribution platform
Grab needed to modernize its marketing analytics from manual ad hoc queries and high data latency to a platform supporting real-time attribution for pricing models like cost per order. The engineering team initially deployed a pure stream-processing engine using Kappa architecture, Kafka, ScyllaDB, and Redis, which reduced latency from days to minutes and merged ads and promo touchpoints. However, stream-only processing faced high costs, out-of-order event issues, and difficulties running multi-touch models across longer historical windows. Grab transitioned to a Lambda architecture pairing Coban stream processing with Spark-based batch ETL and Amazon S3. This hybrid design separated real-time operational metrics from historical batch reporting, cutting real-time processing costs by approximately 25% while maintaining under 1% data discrepancy.
Rethinking Stream Processing: Data Exploration
Online data materialised in Kafka topics at Grab traditionally remained uninspected until ingested into the Data Lake as offline data, introducing latency and making stream processing applications difficult to develop. To enable earlier data exploration, Grab adopted Apache Zeppelin notebooks integrated with an internal data streaming platform. Zeppelin uses a Flink interpreter to convert interactive SQL queries into Flink jobs submitted to a Flink session cluster. The platform dynamically translates Protobuf schemas into SQL Data Definition Language statements to simplify table discovery. Security and compliance risks are addressed by combining mutual TLS authentication with Strimzi's Open Policy Agent to enforce read-only access policies and audit logging on Kafka topics.
Road localisation in GrabMaps
Grab needed to localise nearly 30 million road segments across more than 10,000 area boundaries in Southeast Asia to attach hyperlocal attributes like language, driving side, and vehicle access rules. Testing direct geometric inclusion between complex road polylines and boundary polygons proved computationally prohibitive for daily map generation pipelines. To optimize this process, Grab introduced geohashes as rectangular proxies, precomputing geohash coverage across borders and road segments before joining the datasets in parallel. To resolve misclassification errors near borders without sacrificing performance, the team evaluated geohash coverage percentages and refined boundary-touching geohashes with polygon clipping. A laptop benchmark on a subset of data reduced runtime from 38 minutes with geometric intersection to 78 seconds using the geohash approximation.
Scaling marketing for merchants with targeted and intelligent promos
Grab previously relied on globally assigned, heuristic promotional campaigns where all users could redeem offers until limits were reached. This lack of targeting and customisation failed to optimize promo spending or meet specific merchant business objectives. To solve this, the engineering team built Bullseye, an automated AI-driven promotional assignment system that customises and assigns offers to consumers. The architecture integrates an operations user interface, backend services, Amazon S3 storage, and Spark jobs executing a suite of predictive modules including eater segmentation, campaign impact simulation, and customer response modeling. Since deploying Bullseye in 2021, the system has increased food campaign sales while decreasing promo spend across food and Mart campaigns.
Building hyperlocal GrabMaps
Southeast Asia presents unique mapping challenges, including fast-changing road restrictions, new roads appearing rapidly, and severe traffic congestion. To address these conditions, GrabMaps integrates hyperlocal data derived from local team knowledge, rides and deliveries GPS pings, and KartaView imagery with OpenStreetMap data. The pipeline downloads OpenStreetMap .pbf files for Asia and converts them into Parquet format using a Parquetizer. GrabMaps processes spatial data using Apache Spark and bi-directional graphs divided by geohash to detect complex features such as intersection links based on road parallelism, naming, direction, and shape geometry. Tagging these intersection links enables downstream navigation services like GrabNav to calculate shorter routing alternatives and provide more accurate pricing.
Unsupervised graph anomaly detection - Catching new fraudulent behaviours
Fraud detection systems relying on historical training data often struggle to detect new fraudulent modus operandi because emerging patterns lack labeled supervision. Grab developed GraphBEAN, an unsupervised autoencoder model designed to detect anomalous patterns across bipartite interaction graphs representing consumers and merchants. The architecture uses graph convolution layers to encode node and edge attributes, then reconstructs features and edge existence through separate feature and structure decoders. High reconstruction errors flag rare, anomalous behaviors at both the node and edge levels. An automated pipeline applies heuristic fraud tags to these scores and routes them to human analysts and automated mitigation systems.
PII masking for privacy-grade machine learning
Data engineers at Grab require real-world streaming data to tune machine learning models, but user personal data cannot be exposed in non-production environments. To solve this, the data streaming team implemented an automated masking pipeline between production and staging Kafka clusters. Developers explicitly tag Personally Identifiable Information (PII) types in Protocol Buffers schemas, which a CI script validates against a keyword list before packaging the schemas into Scala JARs. An in-house Apache Flink application running in production consumes the unmasked streams, applies dynamic and consistent transformations like keyed HMACs or seeded random values, and produces sanitised records to staging. Untagged new fields in schema updates are automatically dropped by the deserialiser until the masking application is redeployed, preventing accidental data leaks.
ShopifyQL Notebooks: Simplifying Querying with Commerce Data Models
ShopifyQL Notebooks gives merchants a self-serve way to investigate changing business metrics, understand their causes, and share findings as data stories. It combines the ShopifyQL query language with commerce-specific data models designed for users without SQL expertise, covering domains such as products, orders, and customers. The modeling process starts with business questions, translates them into columns and an appropriate grain, validates the design with mock data and queries, and evaluates upstream data, freshness, and performance. To support large merchants, the implementation uses incremental models and roll-ups, which can reduce build and query times while adding maintenance, pipeline, and monitoring complexity.
The Complex Data Models Behind Shopify's Tax Insights Feature
Shopify Tax’s Tax Insights feature provides merchants with state-level information about sales-tax liability and approaching liability, amid U.S. rules that vary by jurisdiction, product, threshold, and evaluation period. The implementation modified existing data models, added four new ones, and represented changing tax rules with lookup data containing valid_from and valid_to dates, including thresholds, date ranges, inclusions, exclusions, and inclusive or exclusive comparisons. Teams prototyped models in BigQuery SQL, implemented them in PySpark and Spark jobs, and combined merchant monthly taxable-sales aggregates with the rule model to evaluate economic nexus; the architecture moved source data through GCS and a warehouse into a final model. Each run publishes merchant-keyed insights to Google Cloud Bigtable, where the core application retrieves them for the frontend, while warehouse data supports analytics and other processes.
The Hardest Part of Writing Tests is Getting Started
The article presents getting started with automated testing as the main obstacle, especially when a full suite requires helper tools, stubs, and fixtures that can overwhelm beginners. It argues that tests serve as executable documentation, preserve intended behavior and edge cases, and expose regressions before changes reach production. The recommended progression is to install a framework and verify both passing and failing tests, cover isolated functions, add stubs with sanitized API-response fixtures, and then build end-to-end tests while monitoring coverage. In an ETL example, API stubbing avoided expensive, slow calls, while database checks validated loaded and transformed data; the resulting tests covered more than 80 percent of the codebase and saved time for the author and team. The article also proposes E2E tests as a way to safely approach legacy code before refactoring it.
Graph service platform
Grab's GrabDefence team required a dedicated graph infrastructure to proactively identify mobile fraud patterns, such as multiple accounts operating on shared physical devices and suspicious financial loops. To address this, the team built a four-layer Platform as a Service that encapsulates graph database operations behind uniform RESTful APIs for OLTP search and OLAP analysis. The architecture utilizes Amazon S3 for raw data files, Amazon Neptune for graph storage, DynamoDB for schema and metadata configurations, and Kafka for streaming ingestion. When users trigger data loading tasks, the service validates entity attributes against schemas stored in DynamoDB before importing records into Neptune. This infrastructure allows investigators to traverse adjacent account IDs and visualize complex entity relationships without managing underlying database runtimes.
Supabase Wrappers, a Postgres FDW framework written in Rust
Supabase Wrappers is an Alpha framework for building PostgreSQL Foreign Data Wrappers that connect databases with external systems, including third-party APIs, through SQL. It initially supports Firebase and Stripe, while wrappers for ClickHouse, BigQuery, and Airtable are under development. The framework extends pgx, a Rust Postgres extension framework, with async support for RESTful API-based wrappers and limited push-down support currently provided as a proof of concept. The post presents Wrappers as a basis for QETL, allowing teams to query, extract, transform, and load remote data through Postgres, either on demand or into local tables, while joining it with operational data. It is released outside official platform support, with potential breaking changes.
3 (More) Tips for Optimizing Apache Flink Applications
Shopify presents three additional practices for optimizing large stateful Apache Flink applications, focusing on parallelism, sink capacity, and combining heterogeneous sources. It recommends starting with execution-environment parallelism, then matching task-manager capacity to the highest parallelism value; for example, parallelism 100 requires 25 task managers with four slots each. Sink bottlenecks can propagate backpressure upstream, so batching writes and correcting data skew—including with key bucketing when a low-cardinality or uneven key is unavoidable—can improve distribution, though bucketed results must later be combined. For archived Kafka topics, HybridSource exposes cloud-storage history and the live Kafka topic as one ordered logical DataStream, automatically switching after the archive is exhausted and using object-storage partitions to accelerate historical backfills.
SQL or NoSQL? Why not use both (with PostgreSQL)?
The post compares SQL and NoSQL storage for an application that records daily food, water, exercise, weight, and notes, focusing on the tension between easy document writes and later analysis. It proposes using PostgreSQL as a hybrid: keep core fields such as date, user_id, weight, and notes in relational columns, while storing food_log, water_log, and exercise_log as JSONB fields. For analysis, PostgreSQL's jsonb_array_elements expands arrays into rows; ->> extracts text, -> returns JSON, and numeric values such as calories must be cast to INTEGER before aggregation, with a common table expression used to work around the restriction on set-returning functions inside aggregate calls. The examples show daily calorie totals and a search for garlic bread returning 200 calories, supporting the conclusion that JSONB can simplify storage while SQL preserves querying and analysis capabilities.
Query expansion based on user behaviour
Grab developed a query expansion framework to resolve search issues such as typos, multi-language terms, missing alternatives, and word spacing. The corpus is generated through manual annotation for the top thousand fat-head queries and session-based data mining of user rewrite patterns. Offline pipelines extract adjacent query rewrites occurring within 30 seconds where only the second query generated a click, limiting candidates to three per query for serving efficiency. In production, expanded keywords pass through a query understanding, multi-recall, and result fusion flow with automated pruning of underperforming pairs. Online A/B testing across six Southeast Asian countries showed improvements in click-through and conversion rates, with data-mined expansions outperforming manual annotations alone.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.