---
title: "Spark"
description: "27 posts about Spark, summarised, each linking to the original."
---

# Spark
> 27 posts about Spark, summarised, each linking to the original.

## Articles

### [Approximate answers, exact decisions: New sketch functions for analytics](https://yomu.fyi/post/approximate-answers-exact-decisions-new-sketch-functions-for-analytics.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Daniel Tenedorio, Kent Marten, Gengliang Wang, Chenhao Li
- Published: Apr 29, 2026
- Type: Announcement

Databricks now supports four sketch function families, built on Apache DataSketches, for replacing expensive exact percentiles, distinct counts, set operations, top-K rankings, and metric aggregations with bounded-memory approximations. KLL sketches answer quantiles, Theta sketches support unions, intersections, and differences, approximate top-K sketches track frequent items, and Tuple sketches combine distinct counting with aggregated metrics such as customer revenue. The sketches can be built during ETL, stored as columns in Delta tables, and merged on read, allowing dashboards and streaming pipelines to avoid rescanning raw data; SQL, DataFrame, and Structured Streaming pipelines are supported. The stated trade-off is configurable 1–2% relative error, with the source citing a 1000x speedup for applicable workloads. Exact methods remain appropriate for financial auditing, compliance reporting, and other precision-required uses.


### [Stop hand-coding change data capture pipelines](https://yomu.fyi/post/stop-hand-coding-change-data-capture-pipelines.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Matt Jones, Zoé Durand, Phoebe Weiser, Bilal Aslam, Ray Zhu
- Published: Apr 22, 2026
- Type: Problem & solution

Hand-coded change data capture (CDC) and slowly changing dimension (SCD) pipelines become fragile as they handle out-of-order updates, duplicate events, deletes, late-arriving data, retries, backfills, and schema evolution. The post presents AutoCDC in Lakeflow Spark Declarative Pipelines as a declarative alternative: engineers specify keys, sequencing, delete conditions, and SCD Type 1 or Type 2 semantics while the platform manages ordering, state, deduplication, versioning, and incremental processing. Examples show a current-state table that keeps Mercedes’s latest city and removes deleted user 123, alongside a Type 2 table that preserves validity windows through \_\_START\_AT and \_\_END\_AT. The supplied comparison reports roughly 6–10 lines of declarative definition versus 40–200+ lines of custom logic, and says AutoCDC supports both change data feeds and snapshot-based CDC.


### [Balancing cost and reliability for Spark on Kubernetes](https://yomu.fyi/post/balancing-cost-and-reliability-for-spark-on-kubernetes.md)
- Company: [Notion](https://yomu.fyi/company/notion.md)
- Author: Justin Lee
- Published: Feb 12, 2026
- Type: Problem & solution

Notion describes how it reduced Spark compute costs on Kubernetes while maintaining reliability across production workloads. After moving from fixed EC2 clusters on Amazon EMR to EMR on EKS, it used Karpenter, EKS Auto Mode, and the MostAllocated scheduler for dynamic provisioning and bin-packing. Spot Instances then exposed a failure mode: one interruption could remove many bin-packed executors, causing shuffle loss and expensive recomputation. Spot Balancer addresses this by labeling executor creation through a Kubernetes webhook and maintaining each job’s configured spot-to-on-demand ratio, with per-job counts stored in Redis. Named stability weights make the cost-reliability trade-off explicit, and reported savings reached 60–90 percent while jobs that previously failed from interruptions finished reliably.


### [Reducing experiment duration with predicted control variates](https://yomu.fyi/post/reducing-experiment-duration-with-predicted-control-variates.md)
- Company: [Etsy](https://yomu.fyi/company/etsy.md)
- Author: Kelly McManus
- Published: Nov 25, 2025
- Type: Explainer

Etsy describes how its experimentation platform evolved from CUPED to CUPAC to shorten the time needed to reach statistically powered results. CUPED uses pre-experiment visitor data as covariates in linear regression, while CUPAC first uses a nonlinear model to predict the outcome and then uses that prediction in the variance-reduction correction. Etsy trained and tuned LightGBM models on more than 100 pre-experiment features in Vertex AI, generating predictions through parallel Dataflow jobs and applying the correction with Spark within an Airflow-orchestrated pipeline. CUPED reduced variance by 7% on average, whereas CUPAC achieved 27%, shortening average experiment duration by almost three additional days, although results ranged from 2% to 77% depending on the metric and experiment; Etsy is balancing CUPAC’s impact against its training and maintenance costs.


### [DispatchGym: Grab’s reinforcement learning research framework](https://yomu.fyi/post/dispatchgym-grab-s-reinforcement-learning-research-framework.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Tan Sien Yi
- Published: Jul 7, 2025
- Type: Explainer

DispatchGym is a research framework for reinforcement learning (RL) studies in dispatch systems that match bookings with drivers. It combines an RL algorithm, a dispatch-process simulation, and a Gymnasium API interface, allowing data scientists with basic RL and Python skills to modify and test ideas independently. The simulated environment models booking creation, driver dispatch and movement, and booking completion, prioritizing directional accuracy over exact metric matching; its modular Python architecture uses Numba to reduce runtime overhead. A command-line upload launches the modified code on internal infrastructure as a Spark job, avoiding infrastructure management while preserving flexibility across simulation and optimization components. The framework supported experiments with contextual bandits, including linear, neural-linear, and Gaussian-process variants alongside several action-sampling strategies, to identify combinations of levers and reward functions for dispatch hyperparameter tuning and improved fulfilment efficiency and reliability.


### [Building a Spark observability product with StarRocks: Real-time and historical performance analysis](https://yomu.fyi/post/building-a-spark-observability-product-with-starrocks-real-time-and-hi.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Huong Vuong
- Published: Mar 6, 2025
- Type: Problem & solution

Grab’s Iris project replaces a fragmented Spark observability stack with a centralised platform for real-time and historical analysis. The previous setup split data between Grafana, Superset, InfluxDB, and a data lake, creating usability, access-control, pipeline, metadata, and high-cardinality query challenges. The new architecture uses StarRocks as a unified store, ingesting Kafka directly, serving a custom Iris web application and Superset, and backing up to S3. Its relational data model links worker metrics through worker\_uuid and Spark event metrics through app\_id, while materialised views and dynamic partitioning support responsive queries and data retention as volumes grow. The resulting system consolidates metrics and metadata, simplifies ingestion and offline processing, and supports job-level showback/chargeback; the roadmap includes recommendations, advanced analytics, machine learning, and scalability improvements.


### [Categorizing Products at Scale](https://yomu.fyi/post/categorizing-products-at-scale.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Apr 30, 2020
- Type: Explainer

Shopify describes a product-categorization system for more than one billion products across the Google Product Taxonomy, which contains over 5,000 hierarchical categories. The model uses product titles, descriptions, collections, tags, vendors, and merchant-provided types, with text preprocessing and a fixed-length term-frequency HashingTF featurizer in PySpark rather than image features or vocabulary-based methods such as Word2Vec. Kesler’s Construction transforms multiclass training into a binary dataset by embedding each candidate class into feature tokens, while logistic regression provides a single, interpretable classifier that can use taxonomy relationships and reduce computational load. The resulting baseline supports more than 20 teams and downstream experiences, while schematized Kafka events and an internal annotation platform provide human feedback; the post identifies class imbalance, multilingual text, and image features as improvement areas.


[Newer posts](https://yomu.fyi/topic/spark.md)