Loading…
Latest reads
Every post, summarised
Grounded summaries of engineering blogs, newest first. Each links to the original.
- 4,535 posts
- 20 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Latest posts
Zero trust with Kafka
Grab's real-time data platform team transitioned their large-scale Kafka infrastructure from basic network access controls to a zero-trust architecture. The platform implements mutual Transport Layer Security (mTLS) for offline peer authentication and encryption, driven by HashiCorp Vault's PKI engine and Strimzi on Kubernetes. Policy-Based Access Control is enforced using dedicated Open Policy Agent deployments per cluster, backed by GitOps workflows where topic owners approve JSON authorization rules. To simplify client integration, the team enhanced their Go SDK to handle ephemeral in-memory certificates, automatic renewals, and configurable retries. While the security posture improved, the Java encryption and decryption overhead caused a drop in streaming throughput.
The Supabase Content Storm
The post announces Supabase’s Content Storm, a community campaign created to promote Launch Week 6 through a coordinated release of external content. Supabase invited well-known YouTubers, community members, makers, the SupaSquad, and others building with Supabase to publish related material on the same day, while giving participants an opportunity to build personal brands or promote products and startups. More than 30 creators took part, producing videos, livestreams, tutorials, project write-ups, and a Twitter thread covering topics such as authentication, migrations, Edge Functions, Storage, GraphQL, Flutter, Rust, and TypeScript integrations. The content was published on Tuesday, December 6, and the post provides a complete list of works and authors while directing readers to bookmark the collection, give feedback, and follow contributors.
Logbook: November 14 to December 5, 2022
An edition of Fly’s Logbook covering November 14 to December 5, 2022 collects updates and tutorials for building and operating applications on Fly. It features a real-time collaboration example that connects Replicache with WebSockets and Fly-Replay, plus a LiveWire walkthrough for streaming server content to browsers and another for unloading data during large-dataset pagination. A separate guide explains how to run Supercronic from Fly Dockerfiles when precise cron timing is needed beyond Fly Machines’ monthly, weekly, daily, and hourly scheduling options. The edition also recounts how Fly Postgres was built, despite Fly’s stated preference for off-platform database services such as RDS, CrunchyData, and PlanetScale. The post notes that the update followed a longer break while many Fly staff spent time with family over Thanksgiving.
Planning in Bets: Risk Mitigation at Scale
Preparing for BFCM and other high-traffic events, Shopify describes risk mitigation for a globally distributed, complex, interconnected infrastructure platform that peaked at 75.98M requests per minute, or 1.27M per second. At this scale, identifying and mitigating every possible failure is impossible, so the process starts with “what could go wrong” exercises, then uses voting, expert review of likelihood and severity, and assigned ownership to prioritize action. Decision makers are empowered to weigh risks and rewards, while summarized findings, including key risks and single points of failure, are shared to maintain alignment and awareness. The post frames engineering choices as probabilistic bets: decision quality should be judged separately from outcomes, and uncertainty cannot be eliminated, only addressed through thoughtful prioritization and communication.
How KartaCam powers GrabMaps
Grab's Geo team required an efficient, low-cost way to collect fresh street-level imagery across Southeast Asia, where professional mapping equipment is prohibitively expensive and smartphone crowdsourcing yields inconsistent quality. To address this, Grab built KartaCam, a custom mapping device equipped with a 12MP sensor, dual-band GNSS, 4G LTE, and onboard edge AI. Edge machine learning models evaluate scene suitability, check image quality, filter for map-relevant objects, and blur personal data directly on the device prior to upload. Deploying a four-camera KartaCam 360 array delivers panoramic coverage and point-of-interest data comparable to commercial mapping rigs at roughly one-twentieth the hardware cost.
Using Server Sent Events to Simplify Real-time Streaming at Scale
Shopify’s 2022 BFCM Live Map needed to deliver real-time sales and product data to thousands of concurrent users while processing data from millions of merchants. Its 2021 architecture used WebSocket delivery to a presentation layer, which stored messages in a mailbox that clients polled at least every 10 seconds, creating latency and bottlenecks. Shopify replaced that path with a streamlined Flink-based pipeline and a Golang SSE server that subscribes to Kafka topics and pushes JSON data to registered, authenticated clients as soon as it is available. The SSE server ran horizontally behind Nginx load balancers, with a configurable Java client used to load-test concurrent connections. The 2022 system maintained 100 percent uptime, delivered SSE data within milliseconds of availability, and visualized data within 21 seconds of creation, including pipeline processing.
How We Built Fly Postgres
Fly.io describes how it built Fly Postgres as an automated database app rather than a managed database service. After initially resisting durable storage and database work, the company added attached disk storage and designed a deployable Postgres application with convenience commands, using Postgres streaming replication, Stolon, and Consul to support high availability. A two-node cluster can designate a writable leader and asynchronous standby, while Stolon’s sentinels, keepers, and proxies coordinate state, elect a replacement leader, and route connections; Consul introduces an additional failure mode. Backups use point-in-time, block-level snapshots of the volume, a platform capability that can also support other file-system databases. The post concludes that automated Postgres serves many apps, while managed database partnerships may suit users seeking a simpler experience.
How to Export Datadog Metrics for Exploration in Jupyter Notebooks
Datadog dashboards can constrain metric analysis because they offer a limited set of visualizations and lack tooling for complex work such as statistical modeling. They also aggregate data over wider time ranges: the examples contrast one-second metrics across 15 minutes with two-hour intervals across 30 days, potentially hiding interesting events. The guide uses Datadog’s REST API and Python in Jupyter notebooks, requiring an API key, an APP key, and a metric query such as CPU utilization over time. It defines a time range, splits it into buckets whose width is controlled by time_delta, requests each window in a loop, appends the results, and converts them into a dataframe. The resulting data can be examined with tools such as seaborn; a KDE plot is used to inspect the distribution of system CPU utilization, with the exported data offering greater granularity than the dashboard example.
Cost Efficient Snowflake CI
Ramp’s data team used Snowflake CI checks that created a full duplicate production database for each pull request, causing rising warehouse costs and up to 30 minutes of developer idle time as pull-request volume and model count grew. They changed the process by storing dbt manifest.json artifacts in S3, using dbt’s state selector to build modified models, and generating an upstream-model list for cloning. A custom dbt macro uses Snowflake’s information schema to create zero-copy clones of required production tables in the pull-request database, preserving a single namespace for debugging instead of relying on defer across databases. The resulting CI command sequence builds and tests modified models plus direct downstream dependencies, and the post reports a dramatic, mostly sustained dip in the rolling seven-day average warehouse cost despite higher pull-request velocity.
SQL or NoSQL? Why not use both (with PostgreSQL)?
The post compares SQL and NoSQL storage for an application that records daily food, water, exercise, weight, and notes, focusing on the tension between easy document writes and later analysis. It proposes using PostgreSQL as a hybrid: keep core fields such as date, user_id, weight, and notes in relational columns, while storing food_log, water_log, and exercise_log as JSONB fields. For analysis, PostgreSQL's jsonb_array_elements expands arrays into rows; ->> extracts text, -> returns JSON, and numeric values such as calories must be cast to INTEGER before aggregation, with a common table expression used to work around the restriction on set-returning functions inside aggregate calls. The examples show daily calorie totals and a search for garlic bread returning 200 calories, supporting the conclusion that JSONB can simplify storage while SQL preserves querying and analysis capabilities.
Graph for fraud detection
Rapid expansion across multiple business verticals introduced diverse, evolving fraud patterns at Grab, making traditional rule engines and decision trees inadequate due to their heavy reliance on extensive feature engineering and labeled data. To address cold starts and detect unknown fraud, the engineering team implemented a semi-supervised Relational Graph Convolutional Network (RGCN) across millions of connected ecosystem entities. The model propagates information along neighborhood nodes through graph convolutional layers to produce node embeddings and output fraud probabilities. Trained on graphs where only a small percentage of nodes are labeled, the RGCN achieved an AUROC close to 1 and distinctly separated fraudulent embeddings from genuine ones. The approach also offers high explainability by surfacing dense clusters of shared physical devices and addresses ongoing challenges in real-time prediction and noisy connections.
Our Solution for Measuring React Native Rendering Times
Shopify needed a reliable way to measure React Native rendering times and verify that mobile apps remained as fast as native while adopting React Native. The open-source @shopify/react-native-performance library measures app startup, navigation, and screen re-render times by timing from native startup, a user action, or a UI event until a screen is fully rendered and interactive. It uses an invisible marker view with a native counterpart to capture end-to-end timing, including React Native-to-native bridge communication, and models incremental rendering as interactive or non-interactive render passes in a state machine. Reports can be sent through onReportPrepared to analytics tools such as Amplitude, where percentile TTI dashboards compare screens over time, expose bottlenecks, and help catch regressions; Shopify also uses screen-level Apdex internally.
Real-Time Collaboration with Replicache and Fly-Replay
Distributed real-time collaboration becomes difficult when a single server coordinating document changes is overloaded and documents must be split across machines without exposing routing complexity to clients. This demo combines Replicache, a custom WebSocket endpoint, Fly Machines, and the Fly-Replay header to route each document’s connection through a shared router to the machine responsible for that document. The server keeps connections by spaceID, sends initial and incremental state, and tracks messages already delivered; the client overrides Replicache’s push and pull handlers to exchange data through the persistent socket. The resulting application has React frontend, Express router, and Replicache/WebSocket backend layers, with each todo list assigned to one backend machine. The author cautions that the approach is demonstrated only in a basic demo and should receive serious production testing and tuning.
Flutter Authorization with RLS
This tutorial extends a Flutter real-time chat app by adding authorization and one-to-one conversations backed by Supabase. It introduces a rooms page for starting conversations and listing existing rooms, while flutter_bloc coordinates profile and message data across the rooms and chats pages. The database changes add rooms and room_participants tables, connect messages to rooms, expose create_new_room for reusing or creating a two-user room, and register deep links for email confirmation on iOS and Android. SQL RLS policies then restrict room, participant, and message reads to participants and require message inserts to use the caller's profile ID, producing a real-time chat app with authentication and authorization.
Implementing Server-Driven UI Architecture on the Shop App
The Shop Store team implemented a server-driven UI architecture for the Shop App’s Store Screen to personalize sections and layouts for different merchants. Previously, the client rendered a fixed layout, limiting customization, experiment timing, and the speed of fixes after weekly app releases. The new design uses template processing, orchestration, data loading, and a shop-server GraphQL layer, with ProductsSection and CollectionsSection types and configurable GridLayout or ShelfLayout options. On the client, ServerDrivenStoreScreen and StoreSectionContainer render the sections returned by the server, while default layouts allow older app versions to handle unfamiliar section or layout types. The architecture enables backend-controlled experiments and section changes without requiring an app release, while giving the product team a system intended to grow with the store experience.
Fetching and caching Supabase data in Next.js 13 Server Components
The article demonstrates data fetching and caching with Supabase in Next.js 13’s experimental app directory, focusing on Server Components and Suspense. It creates a posts table, configures a Supabase client, and uses async components to fetch data without manually managing loading states. The examples cover static generation, dynamic routes with generateStaticParams, time-based revalidation, client-side fetching, and a realtime pattern that combines server-rendered initial data with a client subscription to Postgres changes. The article notes that Next.js 13 is stable while the app directory remains in beta, and concludes that separating caching from fetching lets applications change freshness requirements without replacing their data-fetching logic.
What We Learned from Open-Sourcing FlashList
FlashList is Shopify’s React Native list library, created after Shopify’s migration from native technology exposed poor list performance in React Native. The post explains how a four-person team prototyped a library that combined FlatList’s familiar API with RecyclerListView’s performance, while addressing blank spacing and low frame rates on some devices. It also describes a launch planned six months ahead around a landing page, documentation, blog storytelling, social media, conference exposure, and feedback from external users. The post says FlashList was used in production on some Shopify app screens, and the launch achieved 1,000 GitHub stars in less than 24 hours. The broader conclusion is that successful open-source projects require product thinking, user empathy, community building, and a long-term authority-building strategy in addition to code.
Query expansion based on user behaviour
Grab developed a query expansion framework to resolve search issues such as typos, multi-language terms, missing alternatives, and word spacing. The corpus is generated through manual annotation for the top thousand fat-head queries and session-based data mining of user rewrite patterns. Offline pipelines extract adjacent query rewrites occurring within 30 seconds where only the second query generated a click, limiting candidates to three per query for serving efficiency. In production, expanded keywords pass through a query understanding, multi-recall, and result fusion flow with automated pruning of underperforming pairs. Online A/B testing across six Southeast Asian countries showed improvements in click-through and conversion rates, with data-mined expansions outperforming manual annotations alone.
Caching Without Marshal Part 2: The Path to MessagePack
Rails caching relied heavily on Marshal, but cached Ruby-specific objects could break after code changes when old payloads were loaded by code without the former classes. MessagePack provided a generic binary format whose core types exclude Ruby-specific Object and instance-variable encodings, while extension types allowed production-specific serializers such as the compact Date representation. During a roughly six-month migration, the team ran both formats, prefixed MessagePack payloads with a version byte, and used encoding failures to identify unsupported cache values. To scale beyond 128 extension types, they added an Object catchall based on as_pack and from_pack, plus Struct and T::Struct modules whose attribute digests detect refactors and turn stale data into cache misses. After logs showed that Marshal was no longer used, the core monolith moved exclusively to MessagePack, with the Paquito gem extracting much of the migration work.
Logbook: November 7 to November 14, 2022
The November 7–14, 2022 logbook presents updates intended to make Elixir applications easier to deploy and monitor in production, alongside documentation and storage guidance. Mark’s contribution explains how to set up Github Actions for Elixir CI, including the work needed to automate deployment workflows. A separate guide uses OpenTelemetry to monitor a production Elixir deployment and trace an N+1 query performance problem, while the Django section reports community progress toward more comprehensive Fly documentation. The logbook also points to a MinIO deployment guide for self-hosted, S3-compatible object storage, noting that colocating storage in the same datacenter as an application can matter although using an S3 host such as AWS S3 is described as easier in some cases.
Top topics
- Architecture830
- Performance812
- LLMs648
- Machine Learning617
- Open Source561
- AI Agents542
- AI454
- Developer Experience446
Weekly digest
The best engineering reads, once a week.