---
title: "Latest reads"
description: "The engineering internet, summarised so you can actually read it."
---

# Latest reads
> The engineering internet, summarised so you can actually read it.

## Articles

### [Query expansion based on user behaviour](https://yomu.fyi/post/query-expansion-based-on-user-behaviour.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Shuailong Liang
- Published: Nov 16, 2022

Grab developed a query expansion framework to resolve search issues such as typos, multi-language terms, missing alternatives, and word spacing. The corpus is generated through manual annotation for the top thousand fat-head queries and session-based data mining of user rewrite patterns. Offline pipelines extract adjacent query rewrites occurring within 30 seconds where only the second query generated a click, limiting candidates to three per query for serving efficiency. In production, expanded keywords pass through a query understanding, multi-recall, and result fusion flow with automated pruning of underperforming pairs. Online A/B testing across six Southeast Asian countries showed improvements in click-through and conversion rates, with data-mined expansions outperforming manual annotations alone.


### [How to Categorize and Prevent Risks of Sensitive Links in URLScan](https://yomu.fyi/post/how-to-categorize-and-prevent-risks-of-sensitive-links-in-urlscan.md)
- Company: [Tinder](https://yomu.fyi/company/tinder.md)
- Author: Tinder
- Published: Nov 7, 2022

Security integrations with URLScan can accidentally index sensitive links when API calls are executed with public visibility. Tinder Security Labs constructed targeted queries to uncover exposed links across file-sharing services, enterprise ticketing platforms, and single sign-on portals. Investigations revealed that exposed links included document magic links containing contracts and bank details, as well as active new-hire registration links that allowed attackers to complete registration workflows and configure two-factor authentication. In one SSO query alone, researchers discovered 1,300 indexed invitation links, with 20 to 50 new links appearing daily. The researchers collaborated with URLScan to purge the exposed indexes from public searches and advised organizations to configure security scanning tools to use private or unlisted visibility settings.


### [Using mobile sensor data to encourage safer driving](https://yomu.fyi/post/using-mobile-sensor-data-to-encourage-safer-driving.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Laiyi Lin
- Published: Oct 25, 2022

Dedicated vehicular inertial measurement units are prohibitively expensive to install across large fleets, while customer feedback lacks the coverage required to assess driving safety comprehensively. To monitor driver behavior at scale, Grab processes smartphone sensor data collected via a mobile telematics SDK at high sampling frequencies. A custom orientation algorithm calculates Euler roll and pitch angles from gravity and resolves yaw through principal component analysis to align phone measurements with vehicle axes. The pipeline applies low-pass or Kalman filtering to handle varying noise levels, followed by threshold peak-finding algorithms to detect harsh acceleration, braking, and cornering. Speeding events are identified by comparing GPS speeds against known map limits or aggregated speed profiles derived from free-flowing traffic.


### [Automatic rule backtesting with large quantities of data](https://yomu.fyi/post/automatic-rule-backtesting-with-large-quantities-of-data.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Chao Wang
- Published: Sep 8, 2022

Evaluating new or modified risk rules previously required Grab analysts to run slow offline Presto queries, manually construct payloads, or run rules in shadow mode for days. To standardize and accelerate this workflow, Grab developed an automated backtesting system powered by an AWS EMR Spark pipeline. Historical events are continuously ingested via Kafka and a Kubernetes stream pipeline into S3 using Snappy-compressed Parquet. Users configure replay intervals and rule definitions directly in the rule engine UI, which triggers asynchronous Spark jobs through Amazon SQS and Lambda. This automated simulation replaces multi-week shadow mode runs and generates downloadable aggregation metrics on transactions, user counts, and treatment outcomes.


### [PostgREST v10: EXPLAIN and Improved Relationship Detection](https://yomu.fyi/post/postgrest-v10-explain-and-improved-relationship-detection.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: steve\_chavez
- Published: Aug 19, 2022

PostgREST v10 adds HTTP and supabase-js access to PostgreSQL execution plans for reads, writes, and RPC requests, with the feature restricted to development-oriented use through db-plan-enabled. Plans can be returned as text or JSON, analyzed to show execution time and RLS filtering, and queried in verbose form for the pg\_stat\_statements query identifier and related settings. The release also improves relationship detection: unique or foreign-key primary keys produce one-to-one JSON objects, while computed relationships let SQL functions define efficient links for views and materialized views. Many-to-many detection now requires a join table’s primary key to include both foreign-key columns, a breaking change; the post says PostgREST 10 was not yet on the Supabase Platform and was planned for release there over the following month.


### [Supabase Vault](https://yomu.fyi/post/supabase-vault.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: michel
- Published: Aug 19, 2022

Supabase Vault is a Postgres extension for managing secrets and encryption inside the database, presented as a thin usability layer over pgsodium. The announcement contrasts it with pgcrypto, which accepts raw encryption keys directly and lacks public key signing, key derivation APIs, and streaming encryption. Vault stores secrets in an encrypted text column, while a unique per-database root key is managed outside SQL by pgsodium and libsodium; rows expose an internally derived key ID rather than the raw key. The vault.decrypted\_secrets view decrypts values on demand, leaving stored data encrypted on disk and in backups, and users can create additional keys with pgsodium.create\_key(). Vault uses Authenticated Encryption with Associated Data, binding the associated column to the authentication signature, and is positioned for API keys, access tokens, and environment variables rather than larger PII datasets.


### [Realtime: Multiplayer Edition](https://yomu.fyi/post/realtime-multiplayer-edition.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: wenbo,stas
- Published: Aug 18, 2022

Supabase announces the general availability of Realtime’s multiplayer features, Broadcast and Presence, alongside a new channel-based architecture and supabase-js support. Realtime originally used Postgres logical replication, rather than LISTEN/NOTIFY, to avoid the latter’s 8,000-byte payload limit and the latency of querying each change from the database. In the new design, Channels act as rooms that support ephemeral publisher/subscriber Broadcast messages, shared Presence state, and Postgres change events; Presence uses Phoenix Tracker, a delta-based CRDT, for eventually consistent synchronization and removes disconnected users automatically. Realtime now runs as a distributed Elixir and Phoenix cluster on Fly, with clients connecting to the nearest node and nodes communicating across a backbone network to improve connection speed, stability, and latency. Free-plan active projects and projects created from August 18 have access, while other projects are scheduled for migration over the following weeks.


### [Supabase is SOC2 compliant](https://yomu.fyi/post/supabase-is-soc2-compliant.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: inian,joel
- Published: Aug 17, 2022

Supabase reports achieving SOC2 Type 1 compliance and later updates the announcement to say it is SOC2 Type 2 compliant. The post explains SOC2’s five Trust Services Criteria and distinguishes Type 1’s point-in-time audit from Type 2’s observation period, while framing customer expectations and data stewardship as the main reasons for pursuing certification. Supabase used Vanta to monitor controls and collect evidence, selected an auditor familiar with SaaS companies and Vanta, and formalized practices covering device security, access control, policies, logging, monitoring, vendor reviews, and repository protection. The process still required substantial manual evidence, and the company found that audit scope and auditor expectations mattered more than achieving a perfect automated-tool score. Supabase also launched a security center and identified HIPAA certification as a next step.


### [supabase-js v2](https://yomu.fyi/post/supabase-js-v2.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: inian,alaister
- Published: Aug 16, 2022

Supabase announces supabase-js v2, a release focused on developer quality-of-life improvements and designed around incremental changes rather than a large rewrite. Its type system uses CLI-generated database definitions at the client level, while authentication adds explicit signInWithPassword() and signInWithOtp() methods, makes most Auth operations asynchronous, and moves server-side methods under auth.admin. Data mutations return minimal by default unless select() is appended, constructor options are scoped by library, and Edge Functions automatically detect request and response content types. The release also introduces typed errors, a channel() interface for presence and PostgreSQL change subscriptions, and a preparatory path toward multiplayer features; older realtime syntax remains supported for now. Developers can install v2, consult updated documentation and migration guidance, while v1 continues receiving security fixes and maintenance patches for three months.


### [Supabase CLI v1 and Management API Beta](https://yomu.fyi/post/supabase-cli-v1-and-management-api-beta.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: soedirgo,qiao
- Published: Aug 15, 2022

Supabase CLI v1 is moving out of beta with capabilities for managing database migrations and generating TypeScript types, alongside a beta Management API. The CLI can manage organizations and projects, link projects locally, run schema and manual migration workflows, support CI/CD through local and GitHub Actions workflows, and generate types from local or remote databases. Migration changes include improved manually written migrations, an experimental migra-backed db diff that is faster and produces more concise DDL, and local branching for testing DDL without data loss; the source notes that generated scripts are not perfect. The REST-based Management API, previously called the Admin API, is consumed by the CLI and exposes programmatic management of organizations, projects, Edge Functions, and more, with endpoints intended for CI/CD and test-environment creation.


### [How we store and process millions of orders daily](https://yomu.fyi/post/how-we-store-and-process-millions-of-orders-daily.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Xi Chen
- Published: Aug 15, 2022

The Grab Order Platform processes millions of food and mart transactions daily, requiring high throughput, fault tolerance, and reduced cloud costs across transactional and analytical workloads. To meet these demands, the engineering team decoupled their database architecture by using Amazon DynamoDB for critical OLTP queries and MySQL RDS for historical OLAP queries. DynamoDB handles online order lifecycles with strong consistency, utilizing sparse Global Secondary Indexes for ongoing orders and TTL configurations to limit storage growth. Updates propagate asynchronously to MySQL RDS through a Kafka ingestion pipeline backed by Amazon SQS retries and timestamp-based version checks. This dual-database approach isolated core transaction availability from analytical queries and delivered significant cloud cost savings.


### [Supabase Series B](https://yomu.fyi/post/supabase-series-b.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: Paul Copplestone
- Published: Aug 12, 2022

Supabase announced an $80 million Series B in May, bringing total funding to $116 million, and reported growth to more than 150,000 hosted PostgreSQL databases and 110,000 developers. Community metrics also increased, with Discord membership rising from 4,000 to 8,000, Twitter followers from 18,000 to 37,000, and GitHub stars from 19,000 to 36,000. The round is led by Felicis, with Coatue, Lightspeed, and Square Peg Capital joining, while Supabase describes its open-source support through contributor payments, maintainer hiring, sponsorship, and community collaboration. The company is moving into its third phase, focused on cloud-native PostgreSQL capabilities including branching, scalable storage, distribution, ephemeral compute, snapshots, and time travel. It invested $500,000 in OrioleDB and hired a developer to work on it, while stating that OrioleDB is not currently used on the Supabase platform and that the company will not run a PostgreSQL fork.


### [Launch Week 5 Hackathon](https://yomu.fyi/post/launch-week-5-hackathon.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: Ant Wilson
- Published: Aug 10, 2022

Supabase’s Launch Week 5 Hackathon invites individuals and teams of up to five to build a new open-source project using Supabase during a 10-day event. It starts at 8:00am PT on Friday 12 August 2022, with submissions due by 11:59pm PT on Sunday 21 August; projects may use any language or framework and must be created during the event. The best overall project receives $1,500 in GitHub sponsorship paid as $500 per month for three months, while the listed category prizes include Gold Supabase Caps, with winner and runner-up prizes and caps for every winning team member. Supabase-js v2.0 and Flutter SDK v1.0 are announced during Launch Week, and using them earns extra credit; the supabase-js release candidate is installable with npm install @supabase/supabase-js@rc.


### [Slack Consolidate: a slackbot built with Python and Supabase](https://yomu.fyi/post/slack-consolidate-a-slackbot-built-with-python-and-supabase.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: Rodrigo Mansueli
- Published: Aug 9, 2022

Slack Consolidate is a Python Slackbot that watches configured Slack channels and reposts messages to destination channels according to routing rules stored in Supabase. The tutorial sets up Slack app permissions from a manifest, installs the Python SDKs, configures environment variables for Supabase and Slack tokens, and creates slack\_channels and slack\_watcher PostgreSQL tables. Its polling loop loads channel mappings, fetches up to 20 recent messages, converts Slack timestamps to ISO-compatible values, inserts messages into the watcher table, builds message links, and posts through a bot client; it waits 1.4 seconds between channel processing and rescans channel configuration hourly. Public-channel posts contain a link, while private-channel posts include the message text as well. The project also provides a Google Colab notebook for trying the example.


### [Supabase Beta July 2022](https://yomu.fyi/post/supabase-beta-july-2022.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: Ant Wilson
- Published: Aug 3, 2022

Supabase published its July 2022 beta update alongside the announcement of Launch Week 5 scheduled for August 15-19. Key feature releases include a developer preview of Flutter SDK 1.0 with an emphasis on developer experience and Auth Helpers supporting server-side rendering environments like SvelteKit. The team introduced pg\_jsonschema, an experimental 10-line Postgres extension created with pgx to enforce structure on json and jsonb columns. Security and identity features were expanded with an hCaptcha integration in authentication settings to block bot attacks. Additionally, email one-time passwords now accommodate lengths between 6 and 10 digits, while the GenerateLink method returns explicit verification metadata.


### [Supabase Flutter SDK 1.0 Developer Preview](https://yomu.fyi/post/supabase-flutter-sdk-1-0-developer-preview.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: Tyler Shukert
- Published: Aug 2, 2022

Supabase released the Developer Preview of version 1.0 for its Flutter SDK, prioritizing improvements to developer experience across Dart applications. The new version eliminates the need for boilerplate state classes, automating authentication state persistence and deep link processing directly after initialization. To better align with Dart conventions, the SDK now throws exceptions across auth, storage, and Postgrest instead of returning error objects, while also deprecating the .execute() method for database queries. Switching to the app\_links library expands deep linking support to macOS and Windows alongside existing mobile and web platforms. Supabase also introduced a companion Auth UI library for prebuilt authentication interfaces and announced plans to integrate the Multiplayer Realtime engine in the stable release.


### [Implementing "seen by" functionality with Postgres](https://yomu.fyi/post/implementing-seen-by-functionality-with-postgres.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: Victor
- Published: Jul 18, 2022

Tracking unique post views within Postgres presents trade-offs between counter accuracy, row bloat, and concurrent write performance. To evaluate options under an architecture constraint preventing external dependencies, a benchmark suite generated synthetic users, skewed post distributions, and replayed view actions. The benchmark tested four implementations: a naive counter column, an hstore key-value approach, an association table, and HyperLogLog (HLL). Benchmark results demonstrated that simple-hstore achieved the lowest average latency among deduplicating approaches at 2.15 milliseconds, closely followed by HLL at 2.16 milliseconds. Despite hstore's raw performance in prototypes, HLL is recommended for production because it circumvents row bloat as view counts grow while avoiding expensive row counts.


### [Revamped Auth Helpers for Supabase (with SvelteKit support)](https://yomu.fyi/post/revamped-auth-helpers-for-supabase-with-sveltekit-support.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: Andrew Smith
- Published: Jul 13, 2022

Supabase has released an updated version of its framework-specific Auth Helpers libraries, introducing official support for SvelteKit alongside existing React and Next.js tooling to simplify server-side rendering authentication. The updated libraries transition from the @supabase/supabase-auth-helpers namespace to @supabase/auth-helpers within a dedicated monorepo managed by Turborepo. Structuring the codebase with Turborepo and changesets enables publishing isolated packages per framework through GitHub Actions, ensuring each helper contains only its own relevant dependencies. Versioning has also been reset to sub-0.x to reflect early framework lifecycles like pre-1.0 SvelteKit and manage expectations for future breaking changes. Meanwhile, the team has documented migration steps from previous packages and announced that Remix Auth Helpers are currently in active development.


### [How we automated FAQ responses at Grab](https://yomu.fyi/post/how-we-automated-faq-responses-at-grab.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Preeti Karkera
- Published: Jul 13, 2022

Internal engineering on-call engineers at Grab spent substantial working hours handling repetitive questions in Slack channels, such as how-to inquiries and access permission requests. To resolve this without building an in-house tool, the team conducted an anonymized vendor comparison and selected OneBar through an employee voting process and a phased proof-of-concept. Initial rollouts were restricted by contract to 20 channels, leading the team to prioritize deployment based on Slack message volume and member counts. Populating the knowledge base required roughly a quarter of consistent updates alongside tech talks, while a targeted crowdsourcing campaign among new onboarders expanded the glossary and grew usage to approximately 3,000 users.


### [Typeahead Search at Nextdoor](https://yomu.fyi/post/typeahead-search-at-nextdoor.md)
- Company: [Nextdoor](https://yomu.fyi/company/nextdoor.md)
- Author: Jerry Tian
- Published: Jul 6, 2022

Nextdoor built a proximity-based autocomplete service to power typeahead search and mention features across its hyperlocal platform for hundreds of millions of entities, including users and businesses. The system shards geographic data using Uber's open-source H3 geohashing library and stores prefix indexes in memory using Redis sorted sets. By adopting a Command Query Responsibility Segregation architecture, ingestion writes are processed on Redis primary nodes and replicated to read-only search nodes with under 10 milliseconds of replication lag. Dedicated APIs handle indexing, typeahead lookups, and ranking before returning hydrated results. Operating since August 2021, the service processes hundreds of millions of monthly typeahead queries while maintaining a P95 search latency below 30 milliseconds.


[Newer posts](https://yomu.fyi/page/40.md) · [Older posts](https://yomu.fyi/page/42.md)