---
title: "Apache Spark Real-Time Mode for Gaming: A Better Way to Do Real-Time Sessionization"
description: "Gaming platforms need sub-second session data for personalization, recommendations, content scheduling, and operational controls across millions of devices. The described pipeline uses Apache Spark Real-Time Mode with transformWithState to process Kafka events, maintain per-device session state, and emit scheduled heartbeats and timeout events without a separate streaming engine. Events are grouped by deviceId; a StatefulProcessor uses MapState keyed by session ID, while handleInputRows() handles starts and ends and handleExpiredTimer() registers and fires 30-second timers. The implementation was tested at about 500K input events per minute, 4M active sessions, and 8M heartbeat records, producing 16x output amplification. End-to-end latency reached 432 ms at p99, reported as 20x faster than micro-batch mode, and the source states that Real-Time Mode is generally available."
---

# Apache Spark Real-Time Mode for Gaming: A Better Way to Do Real-Time Sessionization

[Databricks](https://yomu.fyi/company/databricks) · Neha Prabhu, Murali Talluri · Jun 3, 2026

**Type:** Tutorial

## Summary

Gaming platforms need sub-second session data for personalization, recommendations, content scheduling, and operational controls across millions of devices. The described pipeline uses Apache Spark Real-Time Mode with transformWithState to process Kafka events, maintain per-device session state, and emit scheduled heartbeats and timeout events without a separate streaming engine. Events are grouped by deviceId; a StatefulProcessor uses MapState keyed by session ID, while handleInputRows() handles starts and ends and handleExpiredTimer() registers and fires 30-second timers. The implementation was tested at about 500K input events per minute, 4M active sessions, and 8M heartbeat records, producing 16x output amplification. End-to-end latency reached 432 ms at p99, reported as 20x faster than micro-batch mode, and the source states that Real-Time Mode is generally available.

## Context

Gaming platforms require sub-second processing for session data used in personalization, recommendation engines, dynamic content scheduling, device health signals, parental controls, and abnormal-session detection. Micro-batch Spark processing does not provide the required precision for timer-driven output, while adding Flink or custom services introduces infrastructure and operational complexity.

## Approach / What changed

The pipeline ingests console and PC session events from Kafka, groups them by deviceId, and applies transformWithState through a StatefulProcessor. MapState tracks active sessions, handleInputRows() processes starts and ends, and handleExpiredTimer() emits 30-second heartbeats and timeout events. Outputs are written as JSON to Kafka.

## Takeaways

- The session lifecycle covers starts, 30-second heartbeats, matching ends, and configurable maximum-duration timeouts.
- The test sustained about 500K input events per minute with 4M active sessions and 8M heartbeat records, creating 16x output amplification.
- Real-Time Mode achieved 432 ms p99 end-to-end latency, reported as 20x faster than micro-batch mode.

**Tags:** [Apache Spark](https://yomu.fyi/topic/apache-spark), [Kafka](https://yomu.fyi/topic/kafka), [Performance](https://yomu.fyi/topic/performance), [Streaming](https://yomu.fyi/topic/streaming)

- Source: [Databricks](https://www.databricks.com/blog/apache-spark-real-time-mode-gaming-better-way-do-real-time-sessionization)
- Source URL: https://www.databricks.com/blog/apache-spark-real-time-mode-gaming-better-way-do-real-time-sessionization
- Ingested by Yomu: 2026-08-31T03:31:50.434Z

[Read original post](https://www.databricks.com/blog/apache-spark-real-time-mode-gaming-better-way-do-real-time-sessionization)