---
title: "Data Engineering"
description: "23 posts about Data Engineering, summarised, each linking to the original."
---

# Data Engineering
> 23 posts about Data Engineering, summarised, each linking to the original.

## Articles

### [Data mesh at Grab part I: Building trust through certification](https://yomu.fyi/post/data-mesh-at-grab-part-i-building-trust-through-certification.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Chun Rong Phang
- Published: Aug 19, 2025
- Type: Problem & solution

Grab began its Signals Marketplace data mesh journey in 2024 because its expanding, varied data ecosystem had ownership gaps, duplicated pipelines, central Data Engineering bottlenecks, broken downstream dependencies, inconsistent definitions, and uneven practitioner expertise. The decentralised model treats data as a product owned by domain teams, with Business Data Owners and Technical Data Owners accountable for upkeep, documentation, usability, and SLOs. Certification formalises expectations through data contracts covering schema, freshness, completeness, retention, change notice, and communication, while automated Data Production Incidents flag breached reliability guarantees for investigation. Using the percentage of queries hitting certified assets as its north star metric, Grab reports 75% of queries targeting certified assets, a 400% year-over-year increase in deprecated tables, and a 58% reduction in P80 datasets.


### [Search indexing optimisation](https://yomu.fyi/post/search-indexing-optimisation.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Weilun Wu
- Published: Sep 27, 2021
- Type: Problem & solution

Grab Deliveries uses MySQL as canonical storage and Elasticsearch as derived search storage, with Food-Puxian synchronising updates through Kafka. The original incremental-sync design treated Kafka events mainly as notifications, reloaded records from MySQL, and used separate consumers and in-memory event buffering, creating heavy database traffic, data-loss gaps, coupling to table schemas, redundant Elasticsearch updates, and duplicate cascade updates. A CDC system based on MySQL row-based Binlog provides common events containing Operation, PayloadBefore, and PayloadAfter, enabling filtering of irrelevant changes, partial Elasticsearch updates, event merging in the buffer, and deduplication of cascade updates through a dedicated Kafka stream. The reported results were an 80% reduction in database load from the first optimisation and an 80% reduction in cascade-update events. Overall, DSP saved about 91% of database reads and 90% of Elasticsearch reads, while stream-consumer throughput rose from 200 to 800 QPS, with peaks above 1000 and lower synchronisation latency.


### [Mohammed Ridwanul Islam: How Mentorship, the T Model and a Pen Are the Keys to His Success](https://yomu.fyi/post/mohammed-ridwanul-islam-how-mentorship-the-t-model-and-a-pen-are-the-k.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: Toni Akinwumi
- Published: Oct 3, 2018
- Type: Explainer

Mohammed Ridwanul Islam's career story connects his upbringing in Noakhali, Bangladesh, varied work and music experience, and interest in engineering with his current role on Shopify's Eventscale team. He describes how trying to build a game led him to write C# and later study Electrical Engineering at the University of Waterloo, where an internship became a full-time position. Eventscale develops tools, libraries, and infrastructure for collecting event-oriented streaming data across web, backend, and mobile platforms, supporting analytics and operational needs. Mentorship during a difficult team project taught him project management concepts, while habits, VIM, Java, and writing designs on paper shape his working practices. His advice is to build broad competence alongside deep expertise, and to prioritize problem-solving, values, social impact, and continued growth.


[Newer posts](https://yomu.fyi/topic/data-engineering.md)