The GPU-Resident Astronomy Pipeline
A GPU-resident astronomy pipeline keeps every stage of image analysis — load, align, subtract, detect, review — inside GPU memory instead of round-tripping through the CPU.
Loading…
Topic
Data Pipelines
247 posts about Data Pipelines. Every summary links to the original.
- 247 posts
- 25 companies
- 9 explainers
See the idea
Data Pipelines explainers
BigQuery in the Right Lane: Five Checkpoints for AI Data Pipelines
A five-checkpoint workflow for keeping BigQuery in the right lane: land and shape data once, tune partitions and history windows, and federate instead of copying.
How to Make Apache Hudi Copy-On-Write Upserts Less Wasteful
Learn how Apache Hudi copy-on-write upserts can use row-level file and page locations to update changed data while avoiding unnecessary decompression and rewriting.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Reducing BigQuery Costs: How We Fixed A $1 Million Query
During the infrastructure work for a marketing tool, Shopify's team found a BigQuery query that would have processed about 75,462,743,846 bytes per request and cost nearly $1 million monthly at an estimated 60 requests per minute. The query served a pipeline ingesting one billion rows through Apache Flink, with state managed by RocksDB and streaming requests from Apache Kafka; scaling beyond the release made ingestion unsustainable. To support general availability, the team evaluated an external SQL warehouse that could load Parquet atomically, handle 60 requests per minute, and export results to Google Cloud Storage, then clustered a dataset on two feature columns used in WHERE clauses. Running the same query on the clustered table reduced billed data to 508.1 MB and identified 108.3 MB scanned, lowering estimated monthly cost to about $1,370.67; the post also recommends selecting needed columns, partitioning tables, and using free previews instead of exploratory queries.
Using mobile sensor data to encourage safer driving
Dedicated vehicular inertial measurement units are prohibitively expensive to install across large fleets, while customer feedback lacks the coverage required to assess driving safety comprehensively. To monitor driver behavior at scale, Grab processes smartphone sensor data collected via a mobile telematics SDK at high sampling frequencies. A custom orientation algorithm calculates Euler roll and pitch angles from gravity and resolves yaw through principal component analysis to align phone measurements with vehicle axes. The pipeline applies low-pass or Kalman filtering to handle varying noise levels, followed by threshold peak-finding algorithms to detect harsh acceleration, braking, and cornering. Speeding events are identified by comparing GPS speeds against known map limits or aggregated speed profiles derived from free-flowing traffic.
The Engineering Story Behind Flex Comp
Shopify’s Flex Comp lets employees allocate total rewards among base salary, Restricted Stock Units (RSUs), and Stock Options, with allocation changes during windows and no one-year equity cliffs. Building the system required a prioritized backlog, cross-functional teams, early API contracts, and privacy controls around its code and data. For sensitive analytics events, the team encrypted entire Kafka payloads with RSA, using 4096-bit keys for ordinary events and 6144-bit keys for larger payloads, while later identifying hybrid encryption as a better design. On the frontend, typed hooks, mocked data, and artificial delays allowed business logic and interface work to proceed in parallel while contracts evolved. Version one shipped to Shopify’s employees, with documented technical and design debt and future decisions retained for subsequent iterations.
What is a Full Stack Data Scientist?
Shopify defines full stack data science as an ownership-oriented, end-to-end approach to solving business problems across the data science lifecycle. Rather than specializing in one stage, practitioners participate in discovery and analysis, acquisition, data modeling, pipeline development, and production delivery. The role combines business acumen, communication, Python and SQL programming, exploratory data analysis, data engineering, and machine learning, while treating machine learning as a tool rather than a default and starting simple before adding complexity. Shopify describes the model as generalist with T-shaped development: broad capability is paired with deeper expertise in a specific area, enabling individual strengths within teams to address complex challenges.
Automatic rule backtesting with large quantities of data
Evaluating new or modified risk rules previously required Grab analysts to run slow offline Presto queries, manually construct payloads, or run rules in shadow mode for days. To standardize and accelerate this workflow, Grab developed an automated backtesting system powered by an AWS EMR Spark pipeline. Historical events are continuously ingested via Kafka and a Kubernetes stream pipeline into S3 using Snappy-compressed Parquet. Users configure replay intervals and rule definitions directly in the rule engine UI, which triggers asynchronous Spark jobs through Amazon SQS and Lambda. This automated simulation replaces multi-week shadow mode runs and generates downloadable aggregation metrics on transactions, user counts, and treatment outcomes.
Focus On Behavior, Not State, for a More Maintainable Codebase
Shopify’s work adding pre-orders and “try before you buy” to its Purchase Options APIs exposed code that coupled selling plans too tightly to subscriptions, making future purchase models harder to add. The team reframed the domain around orders and behaviors, treating subscriptions as recurring orders and distinguishing recurring or deferred delivery and billing where those properties mattered. It introduced a wrapper around selling-plan Active Record objects, replaced those objects outside their defining component, and exposed methods such as whether deliveries are recurring while keeping policy details encapsulated. Because platform behavior and merchant reporting need different concepts, the team also added a translation table populated at order creation, with calculated or merchant-provided labels and optional trait records; the maintenance cost isolates reporting semantics from platform and data-team logic.
How we store and process millions of orders daily
The Grab Order Platform processes millions of food and mart transactions daily, requiring high throughput, fault tolerance, and reduced cloud costs across transactional and analytical workloads. To meet these demands, the engineering team decoupled their database architecture by using Amazon DynamoDB for critical OLTP queries and MySQL RDS for historical OLAP queries. DynamoDB handles online order lifecycles with strong consistency, utilizing sparse Global Secondary Indexes for ongoing orders and TTL configurations to limit storage growth. Updates propagate asynchronously to MySQL RDS through a Kafka ingestion pipeline backed by Amazon SQS retries and timestamp-based version checks. This dual-database approach isolated core transaction availability from analytical queries and delivered significant cloud cost savings.
Introducing ShopifyQL: Our New Commerce Data Querying Language
Shopify introduces ShopifyQL, a commerce-focused querying language intended to make data exploration accessible to Shopify Plus merchants without data science or coding backgrounds. It addresses the limits of dashboards and reports, which show what happened but may not explain why, and simplifies access to modeled commerce datasets that would otherwise require SQL and data-lifecycle knowledge. Its syntax uses clauses such as FROM, SHOW or VISUALIZE, BY, WHERE, SINCE, UNTIL, ORDER BY, COMPARE TO, and LIMIT, with direct charting, commerce-specific date ranges, and planned native period comparisons. ShopifyQL is available through the ShopifyQL Notebooks first-party app, while GraphQL APIs let partners submit queries and receive results from the underlying datasets. Shopify says it will use the language across Shopify Admin analytics and plans features including more visualizations, pivoting, and aggregate conditions.
Graph concepts and applications
Real-world systems generate dynamic, non-random connections that traditional statistical approaches fail to characterize or forecast. Graph models represent these structures through vertices and edges, abstracting complex networks into mathematically tractable relationships. Common data representation formats include the Resource Description Framework (RDF), which models subject-predicate-object triples with IRIs, literals, and blank nodes, and Labeled Property Graphs (LPGs), which store arbitrary key-value properties directly on nodes and edges. Graph databases, derived from the LPG model, treat relationships with equal weight to entities, delivering responsive traversals for highly interconnected systems. While they offer agility and explicit relationship modeling, graph databases lack a standardized query language and remain poorly suited for standard transaction-focused workloads compared to relational databases.
Automated Experiment Analysis - Making experimental analysis scalable
Manual ad-hoc analysis of online controlled experiments at Grab introduced operational inefficiencies, inconsistent quality control, and scalability barriers across teams. To resolve these issues, Grab extended its GrabX experimentation platform with an Automated Experiment Analysis system that standardises metrics and automates statistical evaluations. The architecture stores experiment configurations and metric definitions from Cosmos DB into Azure Data Lake as bronze datasets, uses Spark on Databricks via Azure Data Factory to process subjects into silver datasets, and applies an internal Python Decision Engine to generate final gold results. These gold datasets are stored in star-schema fact and dimension tables and presented directly in the GrabX interface using embedded Power BI visualisations. The automation eliminates repetitive data pipeline construction for analysts, ensures reproducible findings aligned with initial hypotheses, and accelerates product launch decisions.
Lessons Learned From Running Apache Airflow at Scale
Shopify describes operating Apache Airflow 2.2 on Kubernetes with the Celery executor and MySQL 8 for data extraction, machine-learning training, Iceberg maintenance, and DBT modeling. In its largest environment, Shopify runs over 10,000 DAGs, averages over 400 tasks running at a given moment, and executes over 150,000 runs per day, while facing slow file access, metadata growth, ownership tracking, DAG-author control, traffic bursts, and resource contention. The authors describe replacing per-pod GCSFuse access with an in-cluster NFS volume synchronized to GCS, retaining GCS as the user-facing source of truth; they also use 28-day metadata retention, a YAML manifest, DAG policies, and scheduling controls including pools, priority weights, and Celery queues. These measures are presented as ways to improve file performance, limit database growth, identify owners, enforce constraints, and reduce workload interference, though the retention policy does not support durable history-dependent features such as long-running backfills.
Double Entry Transition Tables: How We Track State Changes At Shopify
Shopify needed a reliable way to count merchants using Shopify Balance over time, where usage requires both an active Shopify Balance account and an active Shopify account. The data team built accounts_transition_facts with double entry transition tables, a format that represents each state change as a -1 row for the previous state and a +1 row for the new state. Individual attribute tables use account_id, transition_at, and an index that breaks timestamp ties; the pipeline merges attributes, carries forward missing values, and generates net_change rows. Because net_change is additive, queries can sum it while filtering statuses, group results by date, or feed daily aggregates, while additional attributes can be added without rewriting existing SQL or PySpark. The initial build required substantial effort but produced a reporting structure intended to scale with Shopify Balance’s growing complexity.
Director of Machine Learning Insights [Part 2: SaaS Edition]
Engineering directors across SaaS organizations share operational experiences regarding the integration, maintenance, and impact of machine learning systems. Deploying production ML introduces significant operational overhead beyond initial modeling, requiring teams to manage data drift, resolve siloed data sources, and adapt to vertical-specific domain shifts. Leaders emphasize that integrating ML effectively often requires prioritizing core business requirements and traditional ML techniques over newly released, complex algorithms. Allowing end users to incorporate domain knowledge and business controls directly into model interactions remains critical for trust and usability. Looking forward, these leaders anticipate substantial ML-driven operational improvements across enterprise workflows, cybersecurity threat detection, and connected healthcare infrastructure.
Opinion Classification with Kili and HuggingFace AutoTrain
Understanding mobile application user feedback often requires expensive and labor-intensive manual analysis. To streamline this process, an active learning pipeline was constructed using Kili for data annotation alongside HuggingFace AutoTrain and standard transformers for text classification. The workflow utilized approximately 40,000 Medium reviews collected from the Google Play Store, classifying content into categories such as Subscription, Content, Interface, and User Experience. Hyperparameter searches with Ray backend were compared against AutoTrain, which automated training within 30 minutes. Subsequent sentiment analysis of the categorized reviews revealed that while overall sentiment remained positive, subscription and interface categories contained predominantly negative feedback, especially in application version 4.5.
Data Is An Art, Not Just A Science—And Storytelling Is The Key
Data science combines science, mathematics, and art, but communication is presented as an often-neglected skill that helps stakeholders understand and act on analysis. Effective data storytelling goes beyond charts and dashboards by framing a business problem as the main character, supplying context, communicating in the audience’s language, and directing action through next steps. The article recommends investing in clean, conformed data foundations and formal support, including Shopify’s data storytelling working group, while also borrowing communication techniques such as Duarte’s Slidedocs visual reports. It further advocates interactive experiences that let audiences filter, drill into, and explore data, illustrated by Shopify’s BFCM 2021 Notebook, which helped merchants examine sales, products, days, and locations. The conclusion is that storytelling can build trust and increase data fluency.
Director of Machine Learning Insights
Directors of machine learning must balance complex mathematical modeling, system design, and sector-specific business operations across diverse industries. In digital media, practitioners leverage deep learning and Bayesian techniques to power assistive content creation tools and short-term interest recommendations while maintaining user privacy. Pharmaceutical teams employ natural language understanding, computer vision, and reinforcement learning across research and manufacturing, requiring strict GxP validation to navigate regulatory compliance and dataset diversity challenges. In the utility and energy sector, machine learning bridges disparate relational billing records, spatial GIS feeds, and physical grid machinery data to generate operational insights. Across these domains, leaders emphasize that successful integration depends on data quality validation, model monitoring safeguards, and assistive human-in-the-loop collaboration.
How Kafka Connect helps move data seamlessly
Grab's real-time data platform team, Coban, implemented a managed Kafka Connect ecosystem on Kubernetes to streamline moving data in and out of Apache Kafka. To resolve dual-write consistency issues and capture pre- and post-change data, the team integrated Debezium connectors to capture MySQL binlog events and accommodate database DDL migrations. For disaster recovery and stream migrations, Coban deployed MirrorMaker2 connectors managed via Terraform to handle message mirroring and consumer offset translation across AWS regions. Additionally, they developed a custom converter utilizing Confluent Schema Registry to transform Protobuf-serialized Kafka records into JSON for ingestion into Azure Event Hubs. This architecture enabled zero-downtime cluster migrations and robust cross-region disaster recovery.
Real-time data ingestion in Grab
Service teams at Grab historically had to dual-write transactional data into databases and Kafka, creating data integrity issues during transaction failures alongside substantial schema maintenance overhead. To overcome these limitations and eliminate burst reads from SQL-based queries, the Caspian team built a real-time ingestion platform synchronising MySQL, Aurora, and DynamoDB directly to Kafka. For MySQL and Aurora, the platform uses Debezium with Kafka Connect on ROW-format binlogs, while DynamoDB changes are captured via DynamoDB streams with auto-scaling AWS Lambda functions. Messages encoded in Protobuf are transported via Kafka and ingested into Amazon S3 using a Golang stream processor. This architecture supports search indexing in Elasticsearch, automated data lake pipelines, cross-region disaster recovery replication, and audit trails.
Abacus - Issuing points for multiple sources
Grab needed a centralised points management architecture to issue loyalty points across a growing catalog of products, membership tiers, and external partner exchanges. To address this, the engineering team built Abacus, an issuance platform designed to process millions of daily transactions with high availability. The system ingests completed transaction streams or API calls, dynamically computes points via configured multipliers, and passes calculations through Amazon Simple Queue Service queues. Once the Point Awarding module updates a persistent ledger, Abacus notifies consumers, emits events to Kafka for downstream consumers, and recalculates rolling point expiration dates.
Shopify's Playbook for Scaling Machine Learning
Shopify describes how its machine-learning practice grew from a first order-fraud product into a portfolio used across products such as Shopify Capital, product categorization, and Help Center search. The playbook starts with choosing a user-important problem, validating trustworthy and accessible data, tracing downstream dependencies, understanding existing solutions, and optimizing for product outcomes rather than model scores alone. To productionize models, Shopify recommends well-defined training, verification, and testing pipelines, deployment decisions based on volume and user commitment, and platform practices that encode validated standards without abstracting too early. Its fraud pipeline builds models in Python, serializes them with PMML for deployment in Ruby-based production systems, and uses Apache Airflow to schedule operations. The account concludes that scaling is a full-stack problem requiring data scientists and data engineers to align on one roadmap and goal.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.