7 Tips For Optimizing Apache Flink Applications

Summary

Shopify presents lessons from operating large, stateful Apache Flink applications on Kubernetes with Google Kubernetes Engine, RocksDB, and checkpoints and savepoints stored in Google Cloud Storage. The guidance begins with profiling tools, including async-profiler, VisualVM, jemalloc with jeprof, and Eclipse Memory Analyzer, to investigate CPU, heap, native-memory, and leak-related problems. Disabling Kryo fallback exposed unsupported serializers such as Scala BigDecimal and Scala algebraic data types; replacing them with supported alternatives increased throughput by 20 percent. The post also recommends separate configuration profiles for high-throughput backfills and low-latency steady state, while describing a RocksDB native-memory failure in which disabling the block cache stopped repeated out-of-memory crashes without affecting application performance.

Context

Shopify needed to keep large, stateful Flink applications performant and resilient across backfills, steady-state processing, and high-state workloads. Some applications store immense state, including 13 TB for sales data, and the team encountered serialization overhead, memory issues, and repeated Kubernetes container crashes.

Approach / What changed

The guidance combines JVM and native-memory profiling with workload-specific configuration. It recommends disabling Kryo fallback to expose serialization failures, replacing unsupported Scala types, tuning input parallelism for backfills, and using jemalloc and jeprof to inspect RocksDB memory. A custom RocksDBOptionsFactory was used to disable the block cache after profiling linked it to the memory failures.

Takeaways

• Disabling Kryo fallback with env.getConfig().disableGenericTypes() surfaced serialization failures; replacing Scala BigDecimal with Java BigDecimal and Scala ADTs with Scala enums contributed to a reported 20 percent throughput increase.
• Backfills prioritize throughput and rapid backlog reduction, while steady-state processing prioritizes low latency and output freshness; input-source partitioning should account for the backfill workload.
• jemalloc profiling showed RocksDB using 6.74 GB against 5.90 GB of configured Flink managed memory. Disabling the RocksDB block cache stopped the observed out-of-memory crash loop without changing performance after cache population.

Related reading