---
title: "The So-fine Real-time ML Paradigm"
description: "During Etsy’s CodeMosaic hackathon, a team investigated stateful training and online learning to reduce the roughly 40-hour delay between user actions and model-weight updates. They split the work across real-time training data, an incremental-learning service, and evaluation, using Etsy’s Beacon Main Kafka stream, Kafka SQL, Rivulet, and TensorFlow-based Ads models. The data team joined multiple sources despite Avro schema difficulties, while the service team loaded an older model and nearly implemented incremental weight updates; the planned head-to-head evaluation against daily batch training was not completed. The project estimated that replacing daily cold-start training for four ads models could save about $212,000 annually in Google Cloud costs, with possible metric gains from earlier events. Production would still require infrastructure and safeguards against sudden performance degradation."
---

# The So-fine Real-time ML Paradigm

[Etsy](https://yomu.fyi/company/etsy) · Kyle Gallatin · Sep 5, 2023

**Type:** Problem & solution

## Summary

During Etsy’s CodeMosaic hackathon, a team investigated stateful training and online learning to reduce the roughly 40-hour delay between user actions and model-weight updates. They split the work across real-time training data, an incremental-learning service, and evaluation, using Etsy’s Beacon Main Kafka stream, Kafka SQL, Rivulet, and TensorFlow-based Ads models. The data team joined multiple sources despite Avro schema difficulties, while the service team loaded an older model and nearly implemented incremental weight updates; the planned head-to-head evaluation against daily batch training was not completed. The project estimated that replacing daily cold-start training for four ads models could save about $212,000 annually in Google Cloud costs, with possible metric gains from earlier events. Production would still require infrastructure and safeguards against sudden performance degradation.

## Context

Etsy’s existing machine-learning pipelines use scheduled batch jobs, so user actions take at least 40 hours to affect model weights. The project aimed to investigate whether stateful training and online learning could make model updates near-real-time while reducing retraining costs.

## Approach / What changed

The team divided the work into real-time training data, incremental model updating, and evaluation. It used the Beacon Main Kafka stream, Kafka SQL, Rivulet, and TensorFlow-based Ads models; the intended system would load model weights, consume streamed data, update the model, and push artifacts for online serving.

## Takeaways

- The team generated real-time training data by joining multiple sources from Beacon Main and Rivulet, although Avro schema discovery and serialization issues prevented writing the result to a new Kafka topic.
- The planned evaluation compared a continuously trained model with a daily batch-trained model by testing which surfaced a high-performing Etsy ad first, but the comparison was not completed during the hackathon.
- The project estimated about $212,000 in annual Google Cloud savings by replacing daily cold-start training for four Ads models, while noting that production would need safeguards against performance degradation.

**Tags:** [Google Cloud](https://yomu.fyi/topic/gcp), [Kafka](https://yomu.fyi/topic/kafka), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Streaming](https://yomu.fyi/topic/streaming)

- Source: [Etsy](https://www.etsy.com/codeascraft/the-so-fine-real-time-ml-paradigm)
- Source URL: https://www.etsy.com/codeascraft/the-so-fine-real-time-ml-paradigm
- Ingested by Yomu: 2026-09-09T16:01:59.474Z

[Read original post](https://www.etsy.com/codeascraft/the-so-fine-real-time-ml-paradigm)