---
title: "Apache Spark"
description: "8 posts about Apache Spark, summarised, each linking to the original."
---

# Apache Spark
> 8 posts about Apache Spark, summarised, each linking to the original.

## Articles

### [Introducing Apache Spark 4.2](https://yomu.fyi/post/introducing-apache-spark-4-2.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Wenchen Fan, Andreas Neumann, Serge Rielau, Szehon Ho, Gengliang Wang, Linhong Liu, Hyukjin Kwon, Jerry Peng, DB Tsai, Xiao Li, Reynold Xin
- Published: Jul 16, 2026

Apache Spark 4.2 extends the engine’s role in modern data and AI workloads with governed metrics, vector and top-K primitives, Arrow-first Python execution, native change data capture, and stronger streaming foundations. Metric views provide shared business definitions, while Spark Connect uses gRPC and Arrow to let remote clients submit logical plans without a full Spark runtime. Spark SQL adds vector similarity functions, NEAREST BY, geospatial types, sketches, and time-series features; Python interoperability includes Arrow UDFs and can move Spark DataFrames into supported Arrow-native tools without copying or serializing underlying data. Spark Declarative Pipelines adds Auto CDC for SCD Type 1 targets, while Data Source V2 standardizes change streams through CHANGES and expands row-level operations and schema evolution. The release also includes operational updates such as Web UI modernization, Kubernetes improvements, JDK 25 support, and dependency upgrades.


### [Ultra-Fast Anomaly Detection using Apache Spark Real-Time Mode](https://yomu.fyi/post/ultra-fast-anomaly-detection-using-apache-spark-real-time-mode.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Jitesh Soni
- Published: Jul 13, 2026

The post presents a reusable real-time guardrail pattern for flagging suspicious Ethereum blockchain transactions and routing them for downstream action. Its rules identify impossible blocks where gas\_used exceeds gas\_limit and scan extra\_data for email addresses, JWT tokens, or AWS access-key patterns, producing ALLOW or QUARANTINE decisions with reasons. The implementation uses Apache Spark Structured Streaming Real-Time Mode, whose continuous data flow, pipeline scheduling, streaming shuffle, pre-allocated execution pipelines, and asynchronous checkpointing target millisecond latency without a separate streaming engine. On a four-worker DBR 16.4 LTS cluster, the stateless Kafka-to-Kafka test processed about 23 million records at 69,713 rows per second, with P95 below 0.5 milliseconds and P99 at 1 millisecond. The post notes that RTM with a Kafka sink provides at-least-once delivery and that more complex stateful workloads may incur higher latency.


### [Azure Databricks at Data + AI Summit 2026 featuring Industry Leaders and Partners](https://yomu.fyi/post/azure-databricks-at-data-ai-summit-2026-featuring-industry-leaders-and.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Kiriana Stukas
- Published: Jun 11, 2026

Data + AI Summit 2026 brings Databricks and Microsoft leaders, partners, and customers together June 15–18, with in-person and virtual programming focused on Azure Databricks. The collaboration presents Azure Databricks as a first-party Microsoft offering for unifying data, analytics, and AI on a secure, scalable foundation, with sessions covering ecosystem integration, federated analytics, governance, modernization, and AI applications. One technical example introduces zero-copy federation between Azure Data Manager for Energy and Databricks compute, preserving ADME as the source of truth while avoiding large-scale data copies. Another shows Unity Catalog External Locations extending governed access to Microsoft OneLake without ETL pipelines, while customer sessions describe Apache Iceberg and Apache Spark integration, fragmented data consolidation, and production-grade finance workflows using Azure Document Intelligence.


### [Apache Spark Real-Time Mode for Gaming: A Better Way to Do Real-Time Sessionization](https://yomu.fyi/post/apache-spark-real-time-mode-for-gaming-a-better-way-to-do-real-time-se.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Neha Prabhu, Murali Talluri
- Published: Jun 3, 2026

Gaming platforms need sub-second session data for personalization, recommendations, content scheduling, and operational controls across millions of devices. The described pipeline uses Apache Spark Real-Time Mode with transformWithState to process Kafka events, maintain per-device session state, and emit scheduled heartbeats and timeout events without a separate streaming engine. Events are grouped by deviceId; a StatefulProcessor uses MapState keyed by session ID, while handleInputRows() handles starts and ends and handleExpiredTimer() registers and fires 30-second timers. The implementation was tested at about 500K input events per minute, 4M active sessions, and 8M heartbeat records, producing 16x output amplification. End-to-end latency reached 432 ms at p99, reported as 20x faster than micro-batch mode, and the source states that Real-Time Mode is generally available.


### [Data Science vs Data Analytics: Compare Careers, Skills, and Degrees](https://yomu.fyi/post/data-science-vs-data-analytics-compare-careers-skills-and-degrees.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Jun 2, 2026

Data analytics and data science are compared as data-focused career paths, with analytics interpreting existing, usually structured data for trends and business decisions, while science builds models and automated systems to predict future outcomes. The guide contrasts typical questions, outputs, tools, education paths, and roles: analysts use SQL, Excel, Tableau, or Power BI for dashboards and reports, whereas data scientists use Python, R, Apache Spark, and MLflow for predictive models and algorithms. It describes analytics types from descriptive through prescriptive, and a data science workflow spanning collection, feature engineering, training, validation, and deployment, including unstructured data such as text, images, and sensor streams. It also explains collaboration, including analysts defining problems and baselines before scientists build models, and offers portfolio suggestions and questions for choosing a path.


### [Expanded interoperability with Unity Catalog Open APIs](https://yomu.fyi/post/expanded-interoperability-with-unity-catalog-open-apis.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Alex Jiang, Tathagata Das
- Published: May 14, 2026

Unity Catalog’s expanded Open APIs address data silos by letting organizations use multiple compute engines while retaining centralized governance and a single copy of data. In beta, Apache Spark, Apache Flink, and DuckDB can create, read, write, and stream to or from UC managed Delta tables, with catalog commits providing serialized commits, transactional safety, and auditability. Delta Kernel, an open source Java and Rust library, abstracts low-level protocol details, helping connectors integrate external writes with catalog-managed commits while Predictive Optimization continues to run on accessed tables. Credential vending, now GA for tables, issues short-lived, scoped cloud credentials and supports M2M OAuth plus automatic refresh; volume credential vending is in Public Preview for unstructured data. The roadmap includes functionality for fine-grained row- and column-level ABAC on external reads, while external managed-table access remains in beta.


### [Shopify's Data Science & Engineering Foundations](https://yomu.fyi/post/shopify-s-data-science-engineering-foundations.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jun 18, 2020

Shopify’s Data Science & Engineering team describes the foundations it uses to provide daily insights to internal teams, merchants, and partners across a rapidly changing commerce ecosystem. Its data warehouse follows a shared dimensional-modelling philosophy, with modelled data built on Spark in a single GitHub repository and made broadly queryable through Presto, while ETL jobs are unit tested and prevented from failing silently. Centralized dashboards, reproducible vetted data points, peer review, product-aligned sub-teams, accessible communication, and cross-team collaboration extend those foundations into analysis and decision-making. Together, these practices let team members reuse existing work, produce trusted and understandable metrics, address the specialist gap through collaboration, and derive new metrics with fewer than 50 lines of SQL.


### [The Evolution of Kit: Automating Marketing Using Machine Learning](https://yomu.fyi/post/the-evolution-of-kit-automating-marketing-using-machine-learning.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Mar 4, 2020

Kit began as a heuristic, rules-based virtual employee that simplified Facebook advertising for Shopify merchants by selecting products and budgets while defaulting other settings to best practices. The post describes its evolution into an artificially intelligent assistant that uses merchant spending behavior and store signals such as traffic and sales to recommend marketing actions. Kit's machine-learning design combines regression to predict likely spending with classification to estimate whether a budget and store state are likely to generate sales; training uses feature engineering, Apache Spark, Google Cloud ML Engine, and monitored model metrics. For real-time predictions, Bigtable stores engineered features, while a Tornado web container delegates model computation to TensorFlow Serving. The resulting system powered one third of Kit's marketing campaigns.