3 (More) Tips for Optimizing Apache Flink Applications

Summary

Shopify presents three additional practices for optimizing large stateful Apache Flink applications, focusing on parallelism, sink capacity, and combining heterogeneous sources. It recommends starting with execution-environment parallelism, then matching task-manager capacity to the highest parallelism value; for example, parallelism 100 requires 25 task managers with four slots each. Sink bottlenecks can propagate backpressure upstream, so batching writes and correcting data skew—including with key bucketing when a low-cardinality or uneven key is unavoidable—can improve distribution, though bucketed results must later be combined. For archived Kafka topics, HybridSource exposes cloud-storage history and the live Kafka topic as one ordered logical DataStream, automatically switching after the archive is exhausted and using object-storage partitions to accelerate historical backfills.

Context

Shopify uses Apache Flink as a standard stateful streaming engine. Its Kafka topics have retention policies, and archived topics are copied to cloud object storage so users can still access data after Kafka retention expires. The guidance addresses scaling task execution, avoiding sink-driven backpressure and skew, and reading historical and real-time data in order.

Approach / What changed

Start with execution-environment-level parallelism and align task-manager slots with the highest parallelism. Reduce sink pressure through batch writes and address uneven key distribution with bucketing when necessary, then combine bucketed results. Use HybridSource to present archived cloud-storage data followed by live Kafka data as one ordered DataStream, with automatic source switching and high historical-read parallelism.

Takeaways

• For a parallelism value of 100, the recommended capacity example is 25 task managers with four slots each; task-manager count multiplied by slots should equal or slightly exceed the highest parallelism.
• Batch writes can improve sink throughput through better compression, lower network usage, and a smaller CPU hit, but sacrificing some latency may be necessary. Key bucketing can mitigate skew but requires a later result-combination strategy.
• HybridSource lets a Flink application read archived object-storage data and then switch automatically to a live Kafka topic as one logical, ordered DataStream; object-storage partitioning can reduce backfill time.

Related reading