Kafka App? There’s a Skill for That

Summary

Etsy uses Kafka streams to provide fresh features and embeddings for machine learning models, which infer shopper intent and listing meaning from visits and listing data. Previously, engineers manually assembled Java and YAML applications, selected schemas and base classes, and handled testing and local execution. Etsy introduced conversational Kafka Skills that use language models to interpret requests, pass structured inputs into versioned scaffolding and validation tools, run applications locally, and open pull requests. Telemetry records Skill invocation and pull-request completion through protobuf messages in Kafka and BigQuery, while least-privilege ACLs, repository permissions, and deployment controls constrain agent actions. Initial adoption was low because developers did not know the Skills existed, so Etsy added vocabulary-specific and repository-level entry points; it now has Skills for seven Kafka workflows and is focusing on discoverability.

Takeaways

1. 1

   Kafka Skills pass structured inputs into versioned scaffolding and validation tools rather than having agents invent Kafka applications from scratch.

2. 2

   Skill telemetry emits started and completion events as protobuf messages through Kafka and stores them in BigQuery for adoption and workflow analysis.

3. 3

   Etsy added vocabulary-specific, lifecycle-stage, copy-paste, and repository-level entry points after finding that developers were using coding agents without discovering the Skills.