---
title: "Latest reads"
description: "The engineering internet, summarised so you can actually read it."
---

# Latest reads
> The engineering internet, summarised so you can actually read it.

## Articles

### [Democratising Fare Storage at Scale Using Event Sourcing](https://yomu.fyi/post/democratising-fare-storage-at-scale-using-event-sourcing.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Sourabh Suman
- Published: Nov 23, 2020

Grab's legacy system stored booking and fare details in a single relational table, creating a bloated booking entity that tracked only the latest fare state and hindered rapid feature iteration. To resolve scalability, stability, and debugging challenges across millions of daily bookings, the team developed Fare Storage using the Event Sourcing pattern. The new architecture persists all fare modification events chronologically in DynamoDB, backed by a cache for eventually consistent reads and message streaming for downstream processing. The platform employs optimistic locking with versioning to manage concurrent updates, enforces idempotency through client-generated transaction UUIDs, and delegates metadata serialization to an SDK to prevent storage API changes.


### [Postgres Views](https://yomu.fyi/post/postgres-views.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: Paul Copplestone
- Published: Nov 18, 2020

Postgres views serve as query shortcuts that execute underlying SQL statements upon retrieval without generating new tables or persisting duplicate data. By encapsulating complex multi-table joins, standard views provide query consistency across applications, simplify repetitive calls, improve logical schema organization, and enhance security by restricting sensitive columns. In contrast, materialized views physically store query results on disk, dramatically reducing read latency for heavy queries spanning millions of rows. Because materialized views introduce the trade-off of stale data, administrators must periodically run the refresh command based on workload tolerances for use cases like analytics and internal dashboards. Materialized views should not substitute query optimization, as underlying query efficiency remains essential.


### [Leveraging Pre-trained Language Model Checkpoints for Encoder-Decoder Models](https://yomu.fyi/post/leveraging-pre-trained-language-model-checkpoints-for-encoder-decoder.md)
- Company: huggingface.co
- Author: Patrick von Platen
- Published: Nov 9, 2020

Pre-training sequence-to-sequence transformer models incurs massive computational costs, limiting development primarily to large institutions. To mitigate these expenses, encoder-decoder architectures can be warm-started using existing pre-trained checkpoints from encoder-only or decoder-only models like BERT and GPT-2. This walkthrough details the methodology and implementation of warm-starting sequence-to-sequence architectures using Hugging Face Transformers. By utilizing the EncoderDecoderModel framework alongside Seq2SeqTrainer, practitioners can construct and fine-tune models such as BERT2BERT on datasets like CNN/DailyMail. The resulting fully trained BERT2BERT model achieves a ROUGE-2 score of 18.22 on the full evaluation set, matching competitive sequence generation baselines at a fraction of standard pre-training costs.


### [Porting fairseq wmt19 translation system to transformers](https://yomu.fyi/post/porting-fairseq-wmt19-translation-system-to-transformers.md)
- Company: huggingface.co
- Author: Stas Bekman
- Published: Nov 3, 2020

The fairseq WMT19 news translation system was ported to the Hugging Face transformers library under the FSMT architecture. Development began with an initial proxy notebook emulating the transformers API before implementing native configuration, modeling, tokenization, and checkpoint conversion scripts. The author first targeted English-Russian models to handle complex dual-vocabulary requirements and directly evaluate output quality during testing, later extending the implementation to merged-vocabulary language pairs. Although model ensembling was omitted due to library limitations, removing fairseq optimizer states reduced the final model download footprint from 13GB to 1.1GB. The implementation was validated locally against converted fairseq checkpoints and merged into the main codebase after two weeks of iterative review.


### [Supabase Alpha October 2020](https://yomu.fyi/post/supabase-alpha-october-2020.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: Paul Copplestone
- Published: Nov 2, 2020

Eight months into building Supabase, the team released product updates focused on stability, reliability, and performance while preparing to transition from Alpha to Beta. The release introduces supabase-js version 1.0, incorporating extensive community feedback to deliver major developer experience improvements across client libraries. Platform interface updates include tabs and full-screen mode for the SQL Editor, alongside new keyboard shortcuts to help power users navigate and manipulate tables rapidly. For user authentication, Supabase added magic link capabilities, enabling passwordless login flows for both new and existing users either as a standalone method or alongside passwords. Under continuous improvement efforts, the team converted realtime-js to TypeScript, heavily optimized dashboard routing performance, and established a repository to benchmark all integrated open source tools.


### [Hyperparameter Search with Transformers and Ray Tune](https://yomu.fyi/post/hyperparameter-search-with-transformers-and-ray-tune.md)
- Company: huggingface.co
- Author: system
- Published: Nov 2, 2020

Hugging Face Transformers models often require hyperparameter tuning to achieve high performance, but practitioners frequently rely on basic grid search or bypass tuning entirely. The Transformers 3.1 release integrates Ray Tune into the Hugging Face Trainer to provide advanced hyperparameter optimization without losing framework customizability. By invoking hyperparameter\_search with the Ray backend, users can configure trial budgets, parallelize runs across multiple GPUs, and plug in search algorithms like HyperOptSearch alongside schedulers like ASHAScheduler. Experimental benchmarks on a BERT model using the RTE dataset show that Population-based Training reached 70.5% test accuracy in 48 GPU minutes, outperforming Grid Search at 65.4% test accuracy. The integration also supports experiment tracking tools like Weights and Biases out of the box.


### [Supabase.js 1.0](https://yomu.fyi/post/supabase-js-1-0.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: Paul Copplestone
- Published: Oct 30, 2020

Supabase has launched Supabase.js 1.0, updating its client library ecosystem with breaking API changes, scalability enhancements, and new documentation tooling. The release introduces native TypeScript implementations across all underlying sub-libraries, including supabase-js, postgrest-js, gotrue-js, and realtime-js, which also enables auto-generated reference docs. In response to community feedback, database and authentication operations now return structured error objects alongside data payloads instead of throwing runtime exceptions. Realtime client scalability is improved by consolidating multiple active subscriptions into a single socket connection per Supabase client. Additionally, the release updates authentication workflows via the newly created gotrue-js library to support third-party logins and OAuth providers alongside revised method names.


### [Keeping 170 Libraries Up to Date on a Large Scale Android App](https://yomu.fyi/post/keeping-170-libraries-up-to-date-on-a-large-scale-android-app.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Lucas Nelaupe
- Published: Oct 30, 2020

Grab's passenger Android superapp relies on more than 170 in-house and open-source libraries, incorporating five to ten library bumps into each weekly release. Although developers often avoided updates due to the fear of leaking defects or crashes into production, the engineering team established a risk-assessment framework based on codebase usage and the volume of incoming changes. To minimize update diffs and avoid accumulating large issues, libraries are updated incrementally on a weekly cadence matching upstream release schedules. Automated UI test cases written in Gherkin syntax execute on CI for every merge request, while high-risk bumps trigger targeted QA manual testing. This structured updating process prevents defect leakage while keeping the application compliant with evolving Google Play target API requirements.


### [Optimally Scaling Kafka Consumer Applications](https://yomu.fyi/post/optimally-scaling-kafka-consumer-applications.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Shubham Badkur
- Published: Oct 13, 2020

Grab's Coban platform runs Golang-based stream processing pipelines on Kubernetes, servicing roughly 400 billion events weekly from Kafka. The initial Horizontal Pod Autoscaler setup caused resource waste and uneven load distribution across Kafka partitions during scale-in and scale-out events. To resolve this, Grab moved to a fixed pod count matching the topic's partition count and adopted Vertical Pod Autoscaling, reducing resource usage versus requests by approximately 45%. The team also introduced Kubernetes priority classes to segment latency-sensitive workloads onto On-Demand nodes and non-critical jobs onto Spot instances. Additionally, overprovisioning via low-priority placeholder pods managed by Cluster Proportional Autoscaler enabled rapid pod rescheduling and reduced deployment delays.


### [Transformer-based Encoder-Decoder Models](https://yomu.fyi/post/transformer-based-encoder-decoder-models.md)
- Company: huggingface.co
- Author: Patrick von Platen
- Published: Oct 10, 2020

Natural language generation tasks map input sequences to target sequences whose lengths cannot be known in advance and vary by content. Standard deep neural networks struggle with these variable mappings because their inputs and targets require fixed-dimensional vector representations. While recurrent neural networks addressed this challenge by generating target sequences auto-regressively from a compressed context state, transformer-based encoder-decoder architectures became the standard paradigm in modern natural language processing. The framework processes sequence-to-sequence problems by encoding source sequences and auto-regressively decoding target sequences token by token using conditional probability distributions. A step-by-step implementation demonstrates greedy decoding with Hugging Face Transformers, passing cached encoder hidden states alongside previously generated target tokens to iteratively generate German translations from English inputs.


### [Supabase Alpha September 2020](https://yomu.fyi/post/supabase-alpha-september-2020.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: Paul Copplestone
- Published: Oct 3, 2020

Seven months into development, Supabase announced a series of platform updates across authentication, database tooling, and client libraries. The release introduced OAuth logins supporting Bitbucket, GitHub, GitLab, and Google, alongside table cloning and one-click Postgres extension management. In the SQL editor, users can now save favorite queries and access locally stored query histories directly within the browser. The web dashboard adopted Next.js automatic static optimization for improved responsiveness, while postgrest-js migrated to TypeScript and an isomorphic gotrue-js TypeScript library was built for Netlify GoTrue integration. Supabase is prioritizing a transition from Alpha to Beta by tracking open-source tool performance in a dedicated benchmarks repository.


### [Our Journey to Continuous Delivery at Grab (Part 1)](https://yomu.fyi/post/our-journey-to-continuous-delivery-at-grab-part-1.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Sylvain Bougerel
- Published: Sep 23, 2020

Around the end of 2018, Grab's backend architecture consisted of roughly 270 services managed through fragmented, manual deployment workflows. Engineers copied release parameters between build logs, wiki pages, Slack bots, and multiple Jenkins jobs, leading to high operational friction and an average of 10 business days between production updates for a service. To streamline delivery, Grab built Conveyor, an internal automation platform built on top of open-source Spinnaker. Conveyor introduced a custom user interface focused on pipeline visibility and a pipeline-as-code DSL called Artificer using Jsonnet files in the monorepository. The platform automatically registers build artifacts with commit metadata to eliminate manual parameter entry and automatically provisions integration, staging, and production pipelines.


### [Supabase Hacktoberfest 2020](https://yomu.fyi/post/supabase-hacktoberfest-2020.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: Thor Schaeff
- Published: Sep 11, 2020

Supabase announced its participation in Hacktoberfest 2020 by organizing a series of community events and open-source contribution activities. Contributors can attend scheduled live broadcasts, including a GitHub Open Source Friday stream featuring @kiwicopple on Twitch alongside an online introductory meetup hosted on YouTube. To coordinate development, Supabase launched a GitHub project board containing tasks and idea notes, requiring participants to label new issues with the hacktoberfest tag for official challenge qualification. Available engineering efforts span creating framework examples for Blitz.js and Redwood.js as well as writing specialized Postgres templates for diverse use cases. Non-code contribution pathways are also supported, encouraging community members to submit documentation improvements, user experience adjustments, illustrations, custom loading pages, or Lottie animations.


### [Block Sparse Matrices for Smaller and Faster Language Models](https://yomu.fyi/post/block-sparse-matrices-for-smaller-and-faster-language-models.md)
- Company: huggingface.co
- Author: François Lagunas
- Published: Sep 10, 2020

Full dense layers in neural networks are often unnecessarily large and can be pruned without sacrificing precision, but available sparse algebra tools and PyTorch native implementations lack efficiency. To resolve this issue, the pytorch\_block\_sparse extension introduces BlockSparseLinear as a drop-in replacement for standard linear layers alongside BlockSparseModelPatcher for on-the-fly model modifications. The library uses NVIDIA CUTLASS C++ CUDA templates derived from Yulhwa Kim's cutlass tilesparse proof of concept. Although baseline sparse operations remain roughly twice as slow as cuBLAS-optimized dense counterparts, performance scales with sparsity, making a 75% sparse matrix approximately two times faster than a dense layer while reducing memory consumption by four times. Future updates will target dynamic sparsity optimization and NVIDIA Ampere Tensor Core primitives.


### [Uncovering the Truth Behind Lua and Redis Data Consistency](https://yomu.fyi/post/uncovering-the-truth-behind-lua-and-redis-data-consistency.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Allen Wang
- Published: Sep 7, 2020

Grab experienced replica CPU usage spikes following service deployments in their master/replica Redis cluster, which caused failovers to spike to 100% CPU. Investigation revealed that a post-deployment Lua monitor script executed separately on both nodes and relied on non-deterministic HGETALL key ordering. Redis encodes hash objects as either ziplists or hashtables, and restoring from an RDB snapshot initializes small hashes as ziplists even if the master previously converted them to hashtables. This encoding discrepancy caused key ordering to diverge, preventing secondary data from deleting correctly and bloating dataset sizes. Grab resolved the issue by sorting the outputs of HKEYS and HGETALL within the Lua script to guarantee deterministic execution across nodes.


### [Supabase Alpha August 2020](https://yomu.fyi/post/supabase-alpha-august-2020.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: Paul Copplestone
- Published: Sep 3, 2020

Six months into developing its hosted database platform, Supabase announced updates focusing on authentication, table view management, and community developer tooling. Users can now create database tables and columns directly from the table view interface as well as invite team members to their organizations. The authentication system introduced configurable email confirmations for new user registrations, allowing administrators to toggle confirmations and edit email templates directly from the dashboard. Community contributors added TypeScript support to Supabase, implemented UMD support in supabase-js, and began building Python client libraries starting with postgrest-py. Supabase is preparing to transition from Alpha to Beta by stabilizing its Auth API, improving dashboard performance, and benchmarking platform systems.


### [Securing and Managing Multi-cloud Presto Clusters with Grab’s DataGateway](https://yomu.fyi/post/securing-and-managing-multi-cloud-presto-clusters-with-grab-s-datagate.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Vinnson Lee
- Published: Aug 24, 2020

Grab's data engineering team needed to manage data access across growing Presto workloads spanning hundreds of users, thousands of tables, and multiple clouds without exposing infrastructure churn to clients. Out-of-the-box Presto lacked fine-grained schema- and table-level access controls, unified endpoint routing, and seamless cluster switching. To solve this, the team built DataGateway, a smart HTTP proxy and abstraction layer deployed as microservices on Kubernetes. The platform parses incoming SQL statements, verifies user access control lists against target schemas and tables, and dynamically routes queries to appropriate AWS or Azure Presto clusters without requiring endpoint updates. This architecture enabled zero-disruption cluster migrations across 40 clusters and unified access management across a multi-cloud data lake environment.


### [Go Modules- A Guide for monorepos (Part 2)](https://yomu.fyi/post/go-modules-a-guide-for-monorepos-part-2.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Michael Cartmell
- Published: Aug 12, 2020

Managing dependencies in a multi-module monorepo created developer friction at Grab due to unexpected changes from previous vendoring attempts and accidental imports. Because Go modules were not yet enabled directly for builds, the team implemented a continuous integration check that executes go mod vendor and rejects merge requests if any diffs exist in go.mod or the vendor directory. Adopting this CI check required configuring SSH deploy keys for private repositories, adding retry logic for network-related false positives, and standardizing on a single Go version to prevent checksum discrepancies. To streamline ongoing maintenance across hundreds of dependencies, the team developed an automated tool named AutoVend Bot. The bot runs go list -m -u all to detect updates and opens a scheduled batch of merge requests each day for human review.


### [Supabase Auth](https://yomu.fyi/post/supabase-auth.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: Paul Copplestone
- Published: Aug 5, 2020

Following an early community launch on Hacker News, the open-source Firebase alternative Supabase faced heavy developer demand for a dedicated authentication and authorization solution. In response, the team evaluated existing open-source tools such as KeyCloak and Kratos before landing on an architecture combining a forked Netlify GoTrue server with PostgreSQL and PostgREST. The resulting Supabase Auth system provides client methods like signUp and signIn alongside authorization powered by PostgreSQL Row Level Security and policies. By leveraging SQL policies to enforce access rules directly in the database, developers avoid repetitive middleware filtering while maintaining granular access controls. Future roadmap plans include adding third-party OAuth providers, custom email templates, and simplified policy management interfaces for non-technical users.


### [Continuous PostgreSQL Backups using WAL-G](https://yomu.fyi/post/continuous-postgresql-backups-using-wal-g.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: Angelico de los Reyes
- Published: Aug 2, 2020

Continuous PostgreSQL backups require coordinating physical base snapshots with ongoing write-ahead log archiving to support point-in-time recovery. The open-source WAL-G utility simplifies this workflow by managing backup pushes and fetches to cloud storage services such as Amazon S3. In a typical setup on Ubuntu with PostgreSQL 12, envdir supplies AWS credentials and storage prefixes to WAL-G commands embedded directly in the PostgreSQL archive configuration. Regular base backups can be automated via cron jobs, which minimizes the volume of WAL archives that must be replayed during a restore. When recovering to a new instance, administrators fetch the latest base backup, define a restore command with targeted recovery timestamps in the configuration, and initiate replay using a recovery signal file.


[Newer posts](https://yomu.fyi/page/69.md) · [Older posts](https://yomu.fyi/page/71.md)