Building a Service-oriented Architecture with Rails and Kafka

Summary

Service-oriented Rails applications can use Kafka to move high-volume events between producers and consumers without coupling downstream availability to an upstream request. Kafka persists events for a configured period, lets consumers manage offsets, and distributes replicated data across broker partitions, while ordering is guaranteed within a partition rather than across a topic. The post shows a Rails setup using the ruby-kafka gem, an asynchronous producer, JSON payloads, topics, and partition keys, then applies the event stream to order creation, email, and fulfillment workflows. It also addresses operational concerns including consumer scaling, lag metrics, duplicate delivery, idempotent processing with UPSERTs or event identifiers, and schema evolution, contrasting JSON’s accessibility with Avro’s stronger schema support.

Takeaways

1. 1

   Kafka consumers manage their own offsets, allowing them to resume from a specific event or read from the beginning or end of a stream.

2. 2

   Kafka guarantees ordering within a partition, not across an entire topic; a partition_key can keep related events in the same partition.

3. 3

   At-least-once delivery requires consumers to tolerate duplicates, for example through UPSERT operations or unique event identifiers.