---
title: "Streaming"
description: "123 posts about Streaming, summarised, each linking to the original."
---

# Streaming
> 123 posts about Streaming, summarised, each linking to the original.

## Articles

### [How We Simplified Our Data Ingestion & Transformation Process](https://yomu.fyi/post/how-we-simplified-our-data-ingestion-transformation-process.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Yichao Wang
- Published: Mar 3, 2019
- Type: Problem & solution

Grab evolved its real-time data ingestion pipeline after an initial architecture built on Spark Streaming and Python encountered operational complexity, node failures, and data loss from S3 eventual consistency. Because the streaming workload primarily handled event partitioning and ORC file generation, the team consolidated these tasks directly into an existing Golang processing service. They implemented sharded concurrent maps for high-throughput partitioning and optimized heap allocations to resolve memory bottlenecks. This refactor removed intermediate Avro conversions and intermediate storage hops. The simplified Go pipeline eliminated data loss and reduced processing lag from up to 13 minutes down to approximately 1 minute.


### [Querying Big Data in Real-time with Presto & Grab's TalariaDB](https://yomu.fyi/post/querying-big-data-in-real-time-with-presto-grab-s-talariadb.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Roman Atachiants
- Published: Jan 2, 2019
- Type: Problem & solution

Grab developed TalariaDB to support real-time SQL querying over high-velocity event streams while maintaining predictable sub-second latencies and low infrastructure costs. The distributed time-series store retains only the most recent hour of data and integrates directly with Presto via its PrestoThriftService interface. Internally, TalariaDB uses the Go-based Badger key-value store to maintain an in-memory key index of metric names and timestamps while mapping columnar event payloads directly to disk. Ingestion occurs by processing pre-partitioned event batches written to Amazon S3 via SQS notifications. By combining a zero-copy decoder with parallel split evaluation across gossiping cluster nodes, the architecture scales horizontally while serving millions of events per second.


### [Mohammed Ridwanul Islam: How Mentorship, the T Model and a Pen Are the Keys to His Success](https://yomu.fyi/post/mohammed-ridwanul-islam-how-mentorship-the-t-model-and-a-pen-are-the-k.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: Toni Akinwumi
- Published: Oct 3, 2018
- Type: Explainer

Mohammed Ridwanul Islam's career story connects his upbringing in Noakhali, Bangladesh, varied work and music experience, and interest in engineering with his current role on Shopify's Eventscale team. He describes how trying to build a game led him to write C# and later study Electrical Engineering at the University of Waterloo, where an internship became a full-time position. Eventscale develops tools, libraries, and infrastructure for collecting event-oriented streaming data across web, backend, and mobile platforms, supporting analytics and operational needs. Mentorship during a difficult team project taught him project management concepts, while habits, VIM, Java, and writing designs on paper shape his working practices. His advice is to build broad competence alongside deep expertise, and to prioritize problem-solving, values, social impact, and continued growth.


[Newer posts](https://yomu.fyi/topic/streaming/page/6.md)