Using Triggers On Schemaless, Uber Engineering's Datastore Using MySQL

Summary

Schemaless triggers provides a scalable, fault-tolerant, and lossless way to listen for changes in Schemaless, Uber Engineering’s datastore using MySQL, and drives workflows including trip billing, analytics-warehouse ingestion, and cross-datacenter replication. Schemaless stores immutable cells and all their versions, allowing each shard to function as a partitioned change log with monotonically increasing added IDs while retaining random-access APIs. The framework tails shards through get_cells_for_shard, assigns them to worker processes under a leader, persists offsets in shared storage, and invokes column-specific callbacks such as bill_rider; handlers must be idempotent because retries can occur. Failed workers’ shards are reassigned, repeatedly failing cells can move to a separate queue, processing continues during leader or shared-storage outages although redistribution pauses, and replicas insulate triggers from database-node failures.

Takeaways

1. 1

   Schemaless retains every cell version and assigns cells monotonically increasing added IDs within each shard, supporting both random-access retrieval and log-style tailing.

2. 2

   Trigger callbacks are guaranteed to run at least once for each cell, so handlers must be idempotent to safely handle retries caused by application or transient external errors.

3. 3

   Workers process assigned shards in write order, and a leader can reassign shards after worker failure; repeated failures may be isolated in a separate queue to avoid stalling a shard.