The So-fine Real-time ML Paradigm

Summary

During Etsy’s CodeMosaic hackathon, a team investigated stateful training and online learning to reduce the roughly 40-hour delay between user actions and model-weight updates. They split the work across real-time training data, an incremental-learning service, and evaluation, using Etsy’s Beacon Main Kafka stream, Kafka SQL, Rivulet, and TensorFlow-based Ads models. The data team joined multiple sources despite Avro schema difficulties, while the service team loaded an older model and nearly implemented incremental weight updates; the planned head-to-head evaluation against daily batch training was not completed. The project estimated that replacing daily cold-start training for four ads models could save about $212,000 annually in Google Cloud costs, with possible metric gains from earlier events. Production would still require infrastructure and safeguards against sudden performance degradation.

Takeaways

1. 1

   The team generated real-time training data by joining multiple sources from Beacon Main and Rivulet, although Avro schema discovery and serialization issues prevented writing the result to a new Kafka topic.

2. 2

   The planned evaluation compared a continuously trained model with a daily batch-trained model by testing which surfaced a high-performing Etsy ad first, but the comparison was not completed during the hackathon.

3. 3

   The project estimated about $212,000 in annual Google Cloud savings by replacing daily cold-start training for four Ads models, while noting that production would need safeguards against performance degradation.