---
title: "Latest reads"
description: "The engineering internet, summarised so you can actually read it."
---

# Latest reads
> The engineering internet, summarised so you can actually read it.

## Articles

### [Using Server Sent Events to Simplify Real-time Streaming at Scale](https://yomu.fyi/post/using-server-sent-events-to-simplify-real-time-streaming-at-scale.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Nov 30, 2022
- Type: Problem & solution

Shopify’s 2022 BFCM Live Map needed to deliver real-time sales and product data to thousands of concurrent users while processing data from millions of merchants. Its 2021 architecture used WebSocket delivery to a presentation layer, which stored messages in a mailbox that clients polled at least every 10 seconds, creating latency and bottlenecks. Shopify replaced that path with a streamlined Flink-based pipeline and a Golang SSE server that subscribes to Kafka topics and pushes JSON data to registered, authenticated clients as soon as it is available. The SSE server ran horizontally behind Nginx load balancers, with a configurable Java client used to load-test concurrent connections. The 2022 system maintained 100 percent uptime, delivered SSE data within milliseconds of availability, and visualized data within 21 seconds of creation, including pipeline processing.


### [How We Built Fly Postgres](https://yomu.fyi/post/how-we-built-fly-postgres.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: Nov 29, 2022
- Type: Explainer

Fly.io describes how it built Fly Postgres as an automated database app rather than a managed database service. After initially resisting durable storage and database work, the company added attached disk storage and designed a deployable Postgres application with convenience commands, using Postgres streaming replication, Stolon, and Consul to support high availability. A two-node cluster can designate a writable leader and asynchronous standby, while Stolon’s sentinels, keepers, and proxies coordinate state, elect a replacement leader, and route connections; Consul introduces an additional failure mode. Backups use point-in-time, block-level snapshots of the volume, a platform capability that can also support other file-system databases. The post concludes that automated Postgres serves many apps, while managed database partnerships may suit users seeking a simpler experience.


### [How to Export Datadog Metrics for Exploration in Jupyter Notebooks](https://yomu.fyi/post/how-to-export-datadog-metrics-for-exploration-in-jupyter-notebooks.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Nov 29, 2022
- Type: Tutorial

Datadog dashboards can constrain metric analysis because they offer a limited set of visualizations and lack tooling for complex work such as statistical modeling. They also aggregate data over wider time ranges: the examples contrast one-second metrics across 15 minutes with two-hour intervals across 30 days, potentially hiding interesting events. The guide uses Datadog’s REST API and Python in Jupyter notebooks, requiring an API key, an APP key, and a metric query such as CPU utilization over time. It defines a time range, splits it into buckets whose width is controlled by time\_delta, requests each window in a loop, appends the results, and converts them into a dataframe. The resulting data can be examined with tools such as seaborn; a KDE plot is used to inspect the distribution of system CPU utilization, with the exported data offering greater granularity than the dashboard example.


### [Cost Efficient Snowflake CI](https://yomu.fyi/post/cost-efficient-snowflake-ci.md)
- Company: [Ramp](https://yomu.fyi/company/ramp.md)
- Author: Kevin Chao
- Published: Nov 28, 2022
- Type: Problem & solution

Ramp’s data team used Snowflake CI checks that created a full duplicate production database for each pull request, causing rising warehouse costs and up to 30 minutes of developer idle time as pull-request volume and model count grew. They changed the process by storing dbt manifest.json artifacts in S3, using dbt’s state selector to build modified models, and generating an upstream-model list for cloning. A custom dbt macro uses Snowflake’s information schema to create zero-copy clones of required production tables in the pull-request database, preserving a single namespace for debugging instead of relying on defer across databases. The resulting CI command sequence builds and tests modified models plus direct downstream dependencies, and the post reports a dramatic, mostly sustained dip in the rolling seven-day average warehouse cost despite higher pull-request velocity.


### [SQL or NoSQL? Why not use both (with PostgreSQL)?](https://yomu.fyi/post/sql-or-nosql-why-not-use-both-with-postgresql.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: burggraf
- Published: Nov 24, 2022
- Type: Explainer

The post compares SQL and NoSQL storage for an application that records daily food, water, exercise, weight, and notes, focusing on the tension between easy document writes and later analysis. It proposes using PostgreSQL as a hybrid: keep core fields such as date, user\_id, weight, and notes in relational columns, while storing food\_log, water\_log, and exercise\_log as JSONB fields. For analysis, PostgreSQL's jsonb\_array\_elements expands arrays into rows; ->> extracts text, -> returns JSON, and numeric values such as calories must be cast to INTEGER before aggregation, with a common table expression used to work around the restriction on set-returning functions inside aggregate calls. The examples show daily calorie totals and a search for garlic bread returning 200 calories, supporting the conclusion that JSONB can simplify storage while SQL preserves querying and analysis capabilities.


### [Graph for fraud detection](https://yomu.fyi/post/graph-for-fraud-detection.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Min Chen
- Published: Nov 24, 2022
- Type: Problem & solution

Rapid expansion across multiple business verticals introduced diverse, evolving fraud patterns at Grab, making traditional rule engines and decision trees inadequate due to their heavy reliance on extensive feature engineering and labeled data. To address cold starts and detect unknown fraud, the engineering team implemented a semi-supervised Relational Graph Convolutional Network (RGCN) across millions of connected ecosystem entities. The model propagates information along neighborhood nodes through graph convolutional layers to produce node embeddings and output fraud probabilities. Trained on graphs where only a small percentage of nodes are labeled, the RGCN achieved an AUROC close to 1 and distinctly separated fraudulent embeddings from genuine ones. The approach also offers high explainability by surfacing dense clusters of shared physical devices and addresses ongoing challenges in real-time prediction and noisy connections.


### [Our Solution for Measuring React Native Rendering Times](https://yomu.fyi/post/our-solution-for-measuring-react-native-rendering-times.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Nov 24, 2022
- Type: Tutorial

Shopify needed a reliable way to measure React Native rendering times and verify that mobile apps remained as fast as native while adopting React Native. The open-source @shopify/react-native-performance library measures app startup, navigation, and screen re-render times by timing from native startup, a user action, or a UI event until a screen is fully rendered and interactive. It uses an invisible marker view with a native counterpart to capture end-to-end timing, including React Native-to-native bridge communication, and models incremental rendering as interactive or non-interactive render passes in a state machine. Reports can be sent through onReportPrepared to analytics tools such as Amplitude, where percentile TTI dashboards compare screens over time, expose bottlenecks, and help catch regressions; Shopify also uses screen-level Apdex internally.


### [Real-Time Collaboration with Replicache and Fly-Replay](https://yomu.fyi/post/real-time-collaboration-with-replicache-and-fly-replay.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: Nov 23, 2022
- Type: Problem & solution

Distributed real-time collaboration becomes difficult when a single server coordinating document changes is overloaded and documents must be split across machines without exposing routing complexity to clients. This demo combines Replicache, a custom WebSocket endpoint, Fly Machines, and the Fly-Replay header to route each document’s connection through a shared router to the machine responsible for that document. The server keeps connections by spaceID, sends initial and incremental state, and tracks messages already delivered; the client overrides Replicache’s push and pull handlers to exchange data through the persistent socket. The resulting application has React frontend, Express router, and Replicache/WebSocket backend layers, with each todo list assigned to one backend machine. The author cautions that the approach is demonstrated only in a basic demo and should receive serious production testing and tuning.


### [Flutter Authorization with RLS](https://yomu.fyi/post/flutter-authorization-with-rls.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: tyler\_shukert
- Published: Nov 22, 2022
- Type: Tutorial

This tutorial extends a Flutter real-time chat app by adding authorization and one-to-one conversations backed by Supabase. It introduces a rooms page for starting conversations and listing existing rooms, while flutter\_bloc coordinates profile and message data across the rooms and chats pages. The database changes add rooms and room\_participants tables, connect messages to rooms, expose create\_new\_room for reusing or creating a two-user room, and register deep links for email confirmation on iOS and Android. SQL RLS policies then restrict room, participant, and message reads to participants and require message inserts to use the caller's profile ID, producing a real-time chat app with authentication and authorization.


### [Implementing Server-Driven UI Architecture on the Shop App](https://yomu.fyi/post/implementing-server-driven-ui-architecture-on-the-shop-app.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Nov 22, 2022
- Type: Explainer

The Shop Store team implemented a server-driven UI architecture for the Shop App’s Store Screen to personalize sections and layouts for different merchants. Previously, the client rendered a fixed layout, limiting customization, experiment timing, and the speed of fixes after weekly app releases. The new design uses template processing, orchestration, data loading, and a shop-server GraphQL layer, with ProductsSection and CollectionsSection types and configurable GridLayout or ShelfLayout options. On the client, ServerDrivenStoreScreen and StoreSectionContainer render the sections returned by the server, while default layouts allow older app versions to handle unfamiliar section or layout types. The architecture enables backend-controlled experiments and section changes without requiring an app release, while giving the product team a system intended to grow with the store experience.


### [Fetching and caching Supabase data in Next.js 13 Server Components](https://yomu.fyi/post/fetching-and-caching-supabase-data-in-next-js-13-server-components.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: jonmeyers\_io
- Published: Nov 17, 2022
- Type: Tutorial

The article demonstrates data fetching and caching with Supabase in Next.js 13’s experimental app directory, focusing on Server Components and Suspense. It creates a posts table, configures a Supabase client, and uses async components to fetch data without manually managing loading states. The examples cover static generation, dynamic routes with generateStaticParams, time-based revalidation, client-side fetching, and a realtime pattern that combines server-rendered initial data with a client subscription to Postgres changes. The article notes that Next.js 13 is stable while the app directory remains in beta, and concludes that separating caching from fetching lets applications change freshness requirements without replacing their data-fetching logic.


### [What We Learned from Open-Sourcing FlashList](https://yomu.fyi/post/what-we-learned-from-open-sourcing-flashlist.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Nov 17, 2022
- Type: Explainer

FlashList is Shopify’s React Native list library, created after Shopify’s migration from native technology exposed poor list performance in React Native. The post explains how a four-person team prototyped a library that combined FlatList’s familiar API with RecyclerListView’s performance, while addressing blank spacing and low frame rates on some devices. It also describes a launch planned six months ahead around a landing page, documentation, blog storytelling, social media, conference exposure, and feedback from external users. The post says FlashList was used in production on some Shopify app screens, and the launch achieved 1,000 GitHub stars in less than 24 hours. The broader conclusion is that successful open-source projects require product thinking, user empathy, community building, and a long-term authority-building strategy in addition to code.


### [Query expansion based on user behaviour](https://yomu.fyi/post/query-expansion-based-on-user-behaviour.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Shuailong Liang
- Published: Nov 16, 2022
- Type: Problem & solution

Grab developed a query expansion framework to resolve search issues such as typos, multi-language terms, missing alternatives, and word spacing. The corpus is generated through manual annotation for the top thousand fat-head queries and session-based data mining of user rewrite patterns. Offline pipelines extract adjacent query rewrites occurring within 30 seconds where only the second query generated a click, limiting candidates to three per query for serving efficiency. In production, expanded keywords pass through a query understanding, multi-recall, and result fusion flow with automated pruning of underperforming pairs. Online A/B testing across six Southeast Asian countries showed improvements in click-through and conversion rates, with data-mined expansions outperforming manual annotations alone.


### [Caching Without Marshal Part 2: The Path to MessagePack](https://yomu.fyi/post/caching-without-marshal-part-2-the-path-to-messagepack.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Nov 15, 2022
- Type: Problem & solution

Rails caching relied heavily on Marshal, but cached Ruby-specific objects could break after code changes when old payloads were loaded by code without the former classes. MessagePack provided a generic binary format whose core types exclude Ruby-specific Object and instance-variable encodings, while extension types allowed production-specific serializers such as the compact Date representation. During a roughly six-month migration, the team ran both formats, prefixed MessagePack payloads with a version byte, and used encoding failures to identify unsupported cache values. To scale beyond 128 extension types, they added an Object catchall based on as\_pack and from\_pack, plus Struct and T::Struct modules whose attribute digests detect refactors and turn stale data into cache misses. After logs showed that Marshal was no longer used, the core monolith moved exclusively to MessagePack, with the Paquito gem extracting much of the migration work.


### [Logbook: November 7 to November 14, 2022](https://yomu.fyi/post/logbook-november-7-to-november-14-2022.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: Nov 14, 2022
- Type: Announcement

The November 7–14, 2022 logbook presents updates intended to make Elixir applications easier to deploy and monitor in production, alongside documentation and storage guidance. Mark’s contribution explains how to set up Github Actions for Elixir CI, including the work needed to automate deployment workflows. A separate guide uses OpenTelemetry to monitor a production Elixir deployment and trace an N+1 query performance problem, while the Django section reports community progress toward more comprehensive Fly documentation. The logbook also points to a MinIO deployment guide for self-hosted, S3-compatible object storage, noting that colocating storage in the same datacenter as an application can matter although using an S3 host such as AWS S3 is described as easier in some cases.


### [Caching Without Marshal Part 1: Marshal from the Inside Out](https://yomu.fyi/post/caching-without-marshal-part-1-marshal-from-the-inside-out.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Nov 11, 2022
- Type: Explainer

Rails caching commonly relies on Ruby’s Marshal serialization, which can encode complex objects but also embeds their class names in cache entries. Shopify describes an incident in which a beta-flag refactor changed classes while old and new code overlapped during deployment; old code then failed on cached instances containing unfamiliar class names and methods, despite passing CI. The article examines Marshal through Ruby’s marshal.c implementation, covering atomic and composite types, instance variables, object references, circularity via TYPE\_LINK, and core-type subclasses via TYPE\_UCLASS. It presents MessagePack as a more compact, stricter, and controllable alternative intended to make caching safer by default, while deferring the cache migration and implementation details to the series’ next part.


### [Apollo Cache is Your Friend, If You Get To Know It](https://yomu.fyi/post/apollo-cache-is-your-friend-if-you-get-to-know-it.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Nov 9, 2022
- Type: Explainer

Shopify’s migration from Apollo GraphQL client 2 to client 3 prompted a closer examination of the Apollo InMemoryCache after past bugs were linked to misunderstanding or misuse. The cache stores an in-memory representation of queried data for the current browser session, and its lifecycle covers fetching, normalization, updating and merging, then garbage collection and eviction. Fetch policies determine whether data comes from the cache, the network, or both; under the default cache-first policy, incomplete data triggers a network request. Normalization breaks responses into objects, assigns cache identifiers usually from \_\_typename and id or configured key fields, and stores them in a flattened structure. Automatic UI updates depend on matching identifiers and suitable query or mutation responses, while objects made unreachable by changed identifiers can remain until garbage collection removes them.


### [Authentication in Ionic Angular with Supabase](https://yomu.fyi/post/authentication-in-ionic-angular-with-supabase.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: simon\_grimm
- Published: Nov 8, 2022
- Type: Tutorial

The tutorial builds a simple realtime messaging app with Ionic Angular and Supabase, covering account authentication and database protection. It defines Postgres tables for users, groups, and messages, enables Row Level Security with policies for public reads and authenticated operations, and adds a trigger to copy newly registered users into a public users table. Angular services, route guards, and Supabase client logic handle email/password registration and login, session state, protected navigation, and magic links. For native apps, Capacitor URL events extract access and refresh tokens from a custom-scheme callback before establishing the session and navigating to the groups page. The tutorial concludes that Supabase can support Ionic Angular authentication with magic links, while phone authentication through Twilio is another possible option.


### [Logbook: October 29 to November 6, 2022](https://yomu.fyi/post/logbook-october-29-to-november-6-2022.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: Nov 8, 2022
- Type: Announcement

This weekly logbook covering October 29 through November 6, 2022 reports updates across Elixir, Postgres, Laravel Livewire, Redis, and Fly Machines. Chris McCord describes how React inspired Elixir LiveView, while revamped Elixir documentation and Livewire tutorials cover client-side pagination, grouping, sorting, and browser notifications from the server. Fly’s revised Postgres documentation distinguishes the tools it provides for provisioning and managing a cluster from the responsibilities customers retain, and points readers to managed Postgres services. The Fly Redis dashboard command opens the Upstash Redis console with instance statistics and connection instructions, while a Fly Machines preview supports defining multiple processes to run inside one container; the log also notes updated documentation and a required CLI version update.


### [How to Categorize and Prevent Risks of Sensitive Links in URLScan](https://yomu.fyi/post/how-to-categorize-and-prevent-risks-of-sensitive-links-in-urlscan.md)
- Company: [Tinder](https://yomu.fyi/company/tinder.md)
- Author: Tinder
- Published: Nov 7, 2022
- Type: Problem & solution

Security integrations with URLScan can accidentally index sensitive links when API calls are executed with public visibility. Tinder Security Labs constructed targeted queries to uncover exposed links across file-sharing services, enterprise ticketing platforms, and single sign-on portals. Investigations revealed that exposed links included document magic links containing contracts and bank details, as well as active new-hire registration links that allowed attackers to complete registration workflows and configure two-factor authentication. In one SSO query alone, researchers discovered 1,300 indexed invitation links, with 20 to 50 new links appearing daily. The researchers collaborated with URLScan to purge the exposed indexes from public searches and advised organizations to configure security scanning tools to use private or unlisted visibility settings.


[Newer posts](https://yomu.fyi/posts/page/201.md) · [Older posts](https://yomu.fyi/posts/page/203.md)