Modernizing Core Insurance Systems: Breaking the Batch Bottleneck

Summary

Migrating core insurance platforms from legacy relational databases with PL/SQL to Java and MongoDB Atlas frequently degrades batch processing performance. Like-for-like migrations can cause batch jobs to run 25 to 30 times slower due to high network round-trips, inefficient per-record operations, and under-utilized bulk capabilities. To resolve these bottlenecks, an optimization framework was developed that leverages MongoDB native bulkWrite operations, intelligent reference data prefetching, and parallel execution. An architecture combining a Spring Boot controller and a dedicated executor framework partitions workloads across threads or event processors. This framework recovered lost performance, completing previously failing batch jobs within service-level agreements and achieving 10 to 15 times better execution speeds than legacy systems in certain workloads.

Context

Migrating core platforms from legacy RDBMS running set-based PL/SQL to a separate application layer with Java and MongoDB Atlas caused batch jobs and ETL processes to slow down by 25 to 30 times or time out. The slowdown stemmed from high application-database network round-trips, per-record operations replacing set-based logic, under-utilization of bulk database features, and transformation computation overhead.

Approach / What changed

An extensible batch optimization framework was built combining bulkWrite operations, intelligent prefetching of reference data, configurable batch chunk sizes, and parallel execution using threads or event processors like the Disruptor pattern. In this architecture, a Spring Boot controller receives triggers, retrieves records from MongoDB, splits them into batches, and dispatches them to an executor framework that manages concurrency and ETL tasks before persisting results in bulk.

Takeaways

• Replacing database-native PL/SQL set operations with decoupled application logic can cause 25 to 30 times slower batch runs if network round-trips and individual record writes are not optimized.
• The optimization framework combines native MongoDB bulkWrite operations, reference data prefetching into memory, dynamic batch sizing, and thread partitioning to accelerate ETL throughput.
• Proper performance tuning requires balancing batch chunk sizes against memory, minding MongoDB transaction operation and document size limits, and sizing thread pools to prevent database overload.

Related reading