Loading…
Latest reads
The engineering internet, summarised so you can actually read it.
Supabase ·
Supabase Beta December 2020
Supabase transitioned to Beta following ten months of development concentrated on performance, security, and reliability. The platform introduced inline table documentation updates, storing descriptions directly as PostgreSQL comments and exposing them through OpenAPI specifications. Developers can now utilize a table view featuring immediate realtime update reflection alongside pagination for handling large datasets. Performance measurements show a 26 percent increase to 1460 requests per second in regions supporting AWS Graviton instances. Furthermore, the update introduces a new region in Sao Paulo, editing support for native Postgres arrays in the grid editor and side panel, and broader compatibility for custom database types.
Paul CopplestoneGrab ·
Pharos - Searching Nearby Drivers on Road Network at Scale
Ride-hailing allocation requires identifying nearby drivers based on actual routing distance and estimated time of arrival rather than straight-line haversine distance. Grab developed Pharos, a distributed in-memory spatial microservice that evaluates road networks to match fast-moving drivers with pick-up requests. The system partitions OpenStreetMap road graphs by city and vehicle vertical while indexing driver positions and edge-based nodes using Adaptive Radix Trees. To locate candidates, Pharos projects driver coordinates onto road segments as phantom nodes and runs Incremental Network Expansion to compute isochrone reaches. Read-write concurrency is managed by switching tree root snapshots, allowing concurrent reads while updates produce new roots for subsequent lookups.
Hao WuGrab ·
Reflecting on the Five Years of Bug Bounty at Grab
Grab launched a private bug bounty programme on HackerOne before opening it publicly in 2017 to complement internal product security efforts. Prior to going public, the team conducted security sweeps, established policies, and expanded the testing scope to prepare for an influx of reports. Operationally, the team opted to pay full bounties immediately upon report triage rather than waiting for vulnerability resolution to incentivize security researchers. Noise reduction was handled using HackerOne Triage, Human-Augmented Signal, and by blocking automated scanning networks targeting Grab's infrastructure. In addition, rotating security engineers weekly and integrating HackerOne APIs with PagerDuty ensured fast initial response times.
Ajay SrivastavaSupabase ·
Making the Supabase Dashboard Supa-fast
Adding features to the Supabase single-page application dashboard risked performance regressions due to increasing JavaScript bundle sizes. To establish a baseline, the engineering team introduced next-bundle-analyzer and tracked Real User Monitoring alongside Core Web Vitals using Sentry. Optimization efforts focused on pruning dependencies, which included swapping Moment.js for day-js, replacing Joi with ajv, reverting crypto-js to version 3.3.0, and moving the 388 KB zxcvbn password module to a backend API. The team also implemented Next.js dynamic imports for heavy components like spreadsheet parsing, removed legacy server-side props to unlock Automatic Static Optimization, and configured long cache headers on assets. These architectural adjustments brought Core Web Vitals within recommended thresholds and lowered client-side page transition payloads below 200 KB of JavaScript.
Inian ParameshwaranGrab ·
How Grab is Blazing Through the Superapp Bazel Migration
Grab's mobile superapp scaled past 2.5 million lines of code across both Android and iOS, leading to unsustainable local and CI build times under Gradle and Xcode. To address these bottlenecks, the engineering team analyzed their dependency trees and introduced an internal tool to calculate and optimize the build critical path. They also deployed a Kubernetes-autoscaled remote build system using Mainframer for Android and implemented Test Impact Analysis to run only affected tests in pre-merge validation. While dependency decoupling yielded modest 7% to 10% gains and iOS remote builds proved unscalable on Apple hardware, Android remote builds reduced local compile times by up to 50%, and targeted test execution reduced pre-merge pipeline durations by more than 30%.
Sergii GrechukhaSupabase ·
Monitoro Built a Web Crawler Handling Millions of API Requests
Supabase showcases numerous customer case studies illustrating how companies across various industries utilize its platform to build products, migrate infrastructure, and improve operational metrics. Several organizations report substantial cost and efficiency improvements, such as Shotgun achieving an 83% data infrastructure cost reduction and Good Tape saving 60% after moving to managed Postgres and authentication. In terms of developer velocity, Kayhan Space experienced an 8x boost in speed, while Quilia reduced development time by 75% through Data API and Row Level Security features. Other highlighted migrations include Mobbin moving 200,000 users from Firebase, Bree transitioning from Fauna, and Firecrawl switching from Pinecone to Supabase Vector for PostgreSQL vector embeddings.
Supabase ·
TAYFA Built a No-Code Website Builder in Seven Days
Supabase compiles case studies detailing how organizations across industries build applications, scale infrastructure, and migrate databases. Organizations adopt the platform for capabilities including managed Postgres, Authentication, Row Level Security, Data APIs, and Supabase Vector for PostgreSQL vector embeddings. Reported operational outcomes include Shotgun reducing data infrastructure costs by 83 percent, Quilia cutting development time by 75 percent, and Good Tape achieving a 60 percent cost reduction. Several teams completed platform migrations to Supabase, including Mobbin migrating 200,000 users from Firebase, Bree moving from Fauna, and Firecrawl transitioning from Pinecone. Furthermore, organizations like Replenysh implemented one-time passwords within 24 hours, while Xendit shipped a production solution in under one week.
Supabase ·
Xendit Built a Counter-Fraud Watchlist for the Fintech Industry
Payment processor Xendit needed a fast, cost-effective counter-fraud watchlist system to screen transactions against international sanctions lists in real time without relying on expensive providers like Worldcheck or Refinitiv. Leveraging their existing familiarity with Postgres, the team loaded parsed sanctions lists from the United Nations and the Indonesian government into Supabase. They implemented full-text search using the Postgres Trigram extension, enabling relevance scoring across queried records. A dedicated database function handles the search queries directly via Xendit's Python clients, allowing the entire solution to reach production in under one week. The deployment has operated reliably in production for nine months without issues, with potential future iterations planned around machine learning techniques.
Supabase ·
Supabase Partners With Strive School To Help Teach Open Source
Supabase has partnered with fellow Y Combinator Summer 20 alumni Strive School to teach upcoming software engineers open-source development practices. Strive School operates an eight-month training program in Europe covering full-stack web and artificial intelligence engineering, with students paying tuition only after securing employment. Through this collaboration, Supabase aims to improve relational database instruction using Postgres while producing exclusive educational content for the program. In addition to creating learning materials, Supabase is hiring a Strive School graduate as an intern to integrate new engineering talent into its team. Supabase is also introducing a Student Developer Pack offering two years of free platform credits, Slack community access, founder office hours, and merchandise.
Rory WildingSupabase ·
Supabase Alpha November 2020
After nine months of development, Supabase introduced several platform updates as it progressed closer to its beta release. The dashboard now supports manual user creation alongside administrative actions such as deleting users and sending password reset emails or magic links. Database management gained direct browser support through a full Monaco SQL editor and new foreign key controls within the table editor. Platform reliability and security were addressed through a DigitalXRAID audit, default email confirmations on signups, and a dedicated status page tracking uptime and latency. Additional progress included community integration work with Vercel, Stripe, and Redwood, as well as an updated benchmarking suite for realistic server workloads.
Paul CopplestoneGrab ·
Democratising Fare Storage at Scale Using Event Sourcing
Grab's legacy system stored booking and fare details in a single relational table, creating a bloated booking entity that tracked only the latest fare state and hindered rapid feature iteration. To resolve scalability, stability, and debugging challenges across millions of daily bookings, the team developed Fare Storage using the Event Sourcing pattern. The new architecture persists all fare modification events chronologically in DynamoDB, backed by a cache for eventually consistent reads and message streaming for downstream processing. The platform employs optimistic locking with versioning to manage concurrent updates, enforces idempotency through client-generated transaction UUIDs, and delegates metadata serialization to an SDK to prevent storage API changes.
Sourabh SumanSupabase ·
Postgres Views
Postgres views serve as query shortcuts that execute underlying SQL statements upon retrieval without generating new tables or persisting duplicate data. By encapsulating complex multi-table joins, standard views provide query consistency across applications, simplify repetitive calls, improve logical schema organization, and enhance security by restricting sensitive columns. In contrast, materialized views physically store query results on disk, dramatically reducing read latency for heavy queries spanning millions of rows. Because materialized views introduce the trade-off of stale data, administrators must periodically run the refresh command based on workload tolerances for use cases like analytics and internal dashboards. Materialized views should not substitute query optimization, as underlying query efficiency remains essential.
Paul Copplestonehuggingface.co ·
Leveraging Pre-trained Language Model Checkpoints for Encoder-Decoder Models
Pre-training sequence-to-sequence transformer models incurs massive computational costs, limiting development primarily to large institutions. To mitigate these expenses, encoder-decoder architectures can be warm-started using existing pre-trained checkpoints from encoder-only or decoder-only models like BERT and GPT-2. This walkthrough details the methodology and implementation of warm-starting sequence-to-sequence architectures using Hugging Face Transformers. By utilizing the EncoderDecoderModel framework alongside Seq2SeqTrainer, practitioners can construct and fine-tune models such as BERT2BERT on datasets like CNN/DailyMail. The resulting fully trained BERT2BERT model achieves a ROUGE-2 score of 18.22 on the full evaluation set, matching competitive sequence generation baselines at a fraction of standard pre-training costs.
Patrick von Platenhuggingface.co ·
Porting fairseq wmt19 translation system to transformers
The fairseq WMT19 news translation system was ported to the Hugging Face transformers library under the FSMT architecture. Development began with an initial proxy notebook emulating the transformers API before implementing native configuration, modeling, tokenization, and checkpoint conversion scripts. The author first targeted English-Russian models to handle complex dual-vocabulary requirements and directly evaluate output quality during testing, later extending the implementation to merged-vocabulary language pairs. Although model ensembling was omitted due to library limitations, removing fairseq optimizer states reduced the final model download footprint from 13GB to 1.1GB. The implementation was validated locally against converted fairseq checkpoints and merged into the main codebase after two weeks of iterative review.
Stas BekmanSupabase ·
Supabase Alpha October 2020
Eight months into building Supabase, the team released product updates focused on stability, reliability, and performance while preparing to transition from Alpha to Beta. The release introduces supabase-js version 1.0, incorporating extensive community feedback to deliver major developer experience improvements across client libraries. Platform interface updates include tabs and full-screen mode for the SQL Editor, alongside new keyboard shortcuts to help power users navigate and manipulate tables rapidly. For user authentication, Supabase added magic link capabilities, enabling passwordless login flows for both new and existing users either as a standalone method or alongside passwords. Under continuous improvement efforts, the team converted realtime-js to TypeScript, heavily optimized dashboard routing performance, and established a repository to benchmark all integrated open source tools.
Paul Copplestonehuggingface.co ·
Hyperparameter Search with Transformers and Ray Tune
Hugging Face Transformers models often require hyperparameter tuning to achieve high performance, but practitioners frequently rely on basic grid search or bypass tuning entirely. The Transformers 3.1 release integrates Ray Tune into the Hugging Face Trainer to provide advanced hyperparameter optimization without losing framework customizability. By invoking hyperparameter_search with the Ray backend, users can configure trial budgets, parallelize runs across multiple GPUs, and plug in search algorithms like HyperOptSearch alongside schedulers like ASHAScheduler. Experimental benchmarks on a BERT model using the RTE dataset show that Population-based Training reached 70.5% test accuracy in 48 GPU minutes, outperforming Grid Search at 65.4% test accuracy. The integration also supports experiment tracking tools like Weights and Biases out of the box.
systemSupabase ·
Supabase.js 1.0
Supabase has launched Supabase.js 1.0, updating its client library ecosystem with breaking API changes, scalability enhancements, and new documentation tooling. The release introduces native TypeScript implementations across all underlying sub-libraries, including supabase-js, postgrest-js, gotrue-js, and realtime-js, which also enables auto-generated reference docs. In response to community feedback, database and authentication operations now return structured error objects alongside data payloads instead of throwing runtime exceptions. Realtime client scalability is improved by consolidating multiple active subscriptions into a single socket connection per Supabase client. Additionally, the release updates authentication workflows via the newly created gotrue-js library to support third-party logins and OAuth providers alongside revised method names.
Paul CopplestoneGrab ·
Keeping 170 Libraries Up to Date on a Large Scale Android App
Grab's passenger Android superapp relies on more than 170 in-house and open-source libraries, incorporating five to ten library bumps into each weekly release. Although developers often avoided updates due to the fear of leaking defects or crashes into production, the engineering team established a risk-assessment framework based on codebase usage and the volume of incoming changes. To minimize update diffs and avoid accumulating large issues, libraries are updated incrementally on a weekly cadence matching upstream release schedules. Automated UI test cases written in Gherkin syntax execute on CI for every merge request, while high-risk bumps trigger targeted QA manual testing. This structured updating process prevents defect leakage while keeping the application compliant with evolving Google Play target API requirements.
Lucas NelaupeGrab ·
Optimally Scaling Kafka Consumer Applications
Grab's Coban platform runs Golang-based stream processing pipelines on Kubernetes, servicing roughly 400 billion events weekly from Kafka. The initial Horizontal Pod Autoscaler setup caused resource waste and uneven load distribution across Kafka partitions during scale-in and scale-out events. To resolve this, Grab moved to a fixed pod count matching the topic's partition count and adopted Vertical Pod Autoscaling, reducing resource usage versus requests by approximately 45%. The team also introduced Kubernetes priority classes to segment latency-sensitive workloads onto On-Demand nodes and non-critical jobs onto Spot instances. Additionally, overprovisioning via low-priority placeholder pods managed by Cluster Proportional Autoscaler enabled rapid pod rescheduling and reduced deployment delays.
Shubham Badkurhuggingface.co ·
Transformer-based Encoder-Decoder Models
Natural language generation tasks map input sequences to target sequences whose lengths cannot be known in advance and vary by content. Standard deep neural networks struggle with these variable mappings because their inputs and targets require fixed-dimensional vector representations. While recurrent neural networks addressed this challenge by generating target sequences auto-regressively from a compressed context state, transformer-based encoder-decoder architectures became the standard paradigm in modern natural language processing. The framework processes sequence-to-sequence problems by encoding source sequences and auto-regressively decoding target sequences token by token using conditional probability distributions. A step-by-step implementation demonstrates greedy decoding with Hugging Face Transformers, passing cached encoder hidden states alongside previously generated target tokens to iteratively generate German translations from English inputs.
Patrick von Platen
