Ultra-Fast Anomaly Detection using Apache Spark Real-Time Mode

Summary

The post presents a reusable real-time guardrail pattern for flagging suspicious Ethereum blockchain transactions and routing them for downstream action. Its rules identify impossible blocks where gas_used exceeds gas_limit and scan extra_data for email addresses, JWT tokens, or AWS access-key patterns, producing ALLOW or QUARANTINE decisions with reasons. The implementation uses Apache Spark Structured Streaming Real-Time Mode, whose continuous data flow, pipeline scheduling, streaming shuffle, pre-allocated execution pipelines, and asynchronous checkpointing target millisecond latency without a separate streaming engine. On a four-worker DBR 16.4 LTS cluster, the stateless Kafka-to-Kafka test processed about 23 million records at 69,713 rows per second, with P95 below 0.5 milliseconds and P99 at 1 millisecond. The post notes that RTM with a Kafka sink provides at-least-once delivery and that more complex stateful workloads may incur higher latency.

Context

The post addresses operational workloads in which immediate responses to suspicious or invalid events affect business outcomes. It uses Ethereum blockchain data to demonstrate real-time detection of physically impossible gas values and payloads containing recognizable PII or credential patterns, while noting that the same classification pattern applies to fraud, IoT monitoring, security operations, and personalization.

Approach / What changed

The pipeline uses Apache Spark Structured Streaming Real-Time Mode to evaluate each incoming event, apply data-quality and payload-hygiene rules, and emit an enriched event with an ALLOW or QUARANTINE decision and explanatory reasons. The validation was tested on approximately 23 million Ethereum messages loaded into Kafka across four partitions, using DBR 16.4 LTS and four i3.xlarge workers.

Takeaways

• Blocks with gas_used greater than gas_limit are treated as impossible under the Ethereum protocol and may indicate corruption, producer bugs, or schema parsing failures.
• The tested stateless pipeline processed about 23.2 million records at 69,713 rows per second; P0 through P95 rounded to zero milliseconds and P99 was 1 millisecond.
• Real-Time Mode with a Kafka sink provides at-least-once delivery, so downstream consumers need idempotent writes or deduplication to handle potential duplicates.

Related reading