Optimizing Data Reliability: Heroku Connect & Drift Detection

Summary

Heroku Connect syncs Salesforce data with Heroku Postgres, but primary polling can miss changes when long-running Salesforce automations commit after the polling window advances, creating drift. The service uses secondary polls over the same bounds as primary polls to find and repair missed changes, with Bulk API or SOAP API selected according to dataset size and retries for failed polls. Recent improvements change Bulk API drift handling by creating one secondary bulk job for each primary bulk job, then adding corresponding batches instead of creating many jobs. Secondary tasks are now scheduled and executed with limited concurrency, reducing simultaneous database connections consumed while Postgres is updated and preserving capacity for applications. The post says these changes reduce load on Salesforce and improve operational reliability during heavy synchronization, while noting that Heroku Connect remains asynchronous rather than truly real-time.

Takeaways

1. 1

   Secondary polls reuse the primary poll bounds to catch Salesforce changes that the initial poll missed.

2. 2

   For Bulk API drift detection, one secondary bulk job is created for each primary bulk job, with secondary batches corresponding to primary batches.

3. 3

   Limiting concurrent secondary poll tasks reduces database connections used during synchronization and leaves more connections available to applications.