Loading…
Latest reads
The engineering internet, summarised so you can actually read it.
Supabase ·
Continuous PostgreSQL Backups using WAL-G
Continuous PostgreSQL backups require coordinating physical base snapshots with ongoing write-ahead log archiving to support point-in-time recovery. The open-source WAL-G utility simplifies this workflow by managing backup pushes and fetches to cloud storage services such as Amazon S3. In a typical setup on Ubuntu with PostgreSQL 12, envdir supplies AWS credentials and storage prefixes to WAL-G commands embedded directly in the PostgreSQL archive configuration. Regular base backups can be automated via cron jobs, which minimizes the volume of WAL archives that must be replayed during a restore. When recovering to a new instance, administrators fetch the latest base backup, define a restore command with targeted recovery timestamps in the configuration, and initiate replay using a recovery signal file.
Angelico de los ReyesSupabase ·
Supabase Alpha July 2020
Supabase announced the release of Supabase Auth following five months of development, providing resources and a video tutorial on implementing authentication with PostgreSQL Row Level Security. The Table Editor now allows users to drill multiple levels deep into table relationships as well as add, delete, and download rows directly. Newly created projects also include access to additional Postgres extensions, including pgsql-http, pgjwt, plpgsql_check, and pljava. Further platform updates include an or filter in the client library, expanded capabilities for the postgres-meta REST management server, and persistence for open tabs in the Table View. Additionally, the auto-generated documentation now integrates authentication guides, and visual page transition glitches have been resolved.
Paul CopplestoneGrab ·
The Journey of Deploying Apache Airflow at Grab
Engineering and data teams across Grab originally operated independent Apache Airflow instances, causing duplicate maintenance overhead and frequent job failures around scaling, logging, and dependency management. To resolve this, a dedicated team developed a centralized orchestration platform that runs isolated, containerized Airflow instances per team on Amazon EKS. The platform categorizes deployments into three size tiers and provisions dedicated Redis brokers, RDS metadata stores, and Vault secret sidecars using Terraform and custom Helm charts. Teams customize container images using shared GitLab CI/CD templates, while worker scaling is handled via Kubernetes Horizontal Pod Autoscalers. Today, the platform runs roughly 20 Airflow instances executing between 1,000 and 60,000 daily jobs per instance.
Chandulal KavarSupabase ·
Alpha Launch Postmortem
An unexpected viral submission on Hacker News prompted sudden alpha traffic for Supabase, attracting 30,000 visitors and over 1,400 signups in one week. The surge strained initial infrastructure, which initially relied on a single server running Docker Compose for middleware and Digital Ocean for customer databases. System bottlenecks emerged rapidly, including hit limits on Digital Ocean droplet counts, frequent physical server faults, and a three-hour provisioning outage when Cloudflare reached its 1,000-subdomain limit. In response, the team migrated middleware to multiple globally load-balanced ECS clusters via AWS Global Accelerator. Supabase also transitioned 1,800 servers from Digital Ocean to AWS t3a instances, nearly halving database startup times from 90 seconds to 50 seconds.
Paul CopplestoneSupabase ·
What are PostgreSQL Templates?
PostgreSQL bases every new database creation on an existing template database within the cluster, defaulting to template1. While administrators can directly modify template1 with tables, data, extensions, or procedural languages, altering it risks breaking future database creation commands if mistakes happen. Setting an existing database as a custom template using the ALTER DATABASE command allows users with the CREATEDB privilege to instantiate customized databases without polluting system templates. Any database creation from a template requires zero active connections on that template at execution time, making pg_dump the preferred tool for replicating active production environments. The immutable template0 system database serves as a fallback to recreate corrupted templates, restore clean dumps, or initialize databases with alternative character encodings.
Angelico de los ReyesSupabase ·
Physical vs Logical Backups in PostgreSQL
PostgreSQL backups fall into two distinct categories: logical backups, which convert data into SQL command files, and physical backups, which copy the underlying file system storage. Logical backups generated with pg_dump or pg_dumpall allow single-database targeting and provide the only practical route for migrating across major Postgres versions with differing internal storage formats. Conversely, physical backups suit larger databases where long-running logical exports can degrade concurrent query performance and risk failure. Physical backups also integrate with Write Ahead Log files through tools like WAL-G to support Point in Time Recovery and minimize Recovery Point Objectives. Selecting between these two strategies depends on whether administrative simplicity, version portability, cluster size, or precise disaster recovery takes precedence.
Angelico de los Reyeshuggingface.co ·
The Reformer - Pushing the limits of language modeling
Standard transformer models hit memory bottlenecks on long sequence modeling tasks due to the quadratic asymptotic memory complexity of global self-attention and oversized positional embedding matrices. The Reformer architecture overcomes these constraints to train sequences of up to half a million tokens using under 8GB of RAM. It re-engineers transformer operations using local and Locality Sensitive Hashing self-attention, chunked feed forward layers, reversible residual layers, and axial positional encodings. In empirical benchmarks using google/reformer-crime-and-punishment, axial positional encodings reduce the model parameter count from over 136 million to approximately 2.58 million by factorizing the positional dimensions. This architectural change cuts inference memory consumption from 959 MB down to 447 MB for evaluated benchmark workloads.
Patrick von PlatenSupabase ·
Supabase Alpha June 2020
Four months into development, Supabase released its June 2020 alpha update focused on simplifying PostgreSQL management and eliminating regional performance bottlenecks. The release introduces a spreadsheet-like interface for drilling down into relational tables alongside automated detection and editing support for JSON and JSONB columns. To address latency issues previously caused by hosting all projects exclusively in Singapore, the platform now allows developers to select specific database regions. In line with its zero lock-in principle, Supabase exposed daily backups on the dashboard and enhanced the SQL editor with improved syntax highlighting. The team also expanded by hiring PostgREST maintainer Steve Chavez while continuing efforts toward future multi-region replication and WAL-G backup integration.
Paul CopplestoneGrab ·
How We Built Our In-house Chat Platform for the Web
Grab extended its in-house chat platform to the web to support its internal Customer Support portal. Because the existing TCP gateway only supports unicast connections with one active connection per user, opening multiple browser tabs would repeatedly disconnect previous tabs. Rather than undertaking a complex migration to multicast connections on the server, the team adopted a hybrid client-side strategy using SharedWorker and BroadcastChannel APIs. The implementation uses a SharedWorker to maintain a single WebSocket connection per domain while a BroadcastChannel syncs events across all open tabs. A custom wrapper over the worker manages version transitions during deployments to avoid race conditions across tabs.
Vasu KrishnamoorthySupabase ·
Supabase Alpha May 2020
Supabase announced its May 2020 alpha development updates following acceptance into a fully remote Y Combinator cohort and unexpected attention from Hacker News. Operating under a ship early and often philosophy, the team continued deploying functional changes despite ongoing alpha bugs and platform instability. This milestone includes a revamped user interface configured to support incoming product capabilities alongside an initial, experimental Table View designed to rival tools like Airtable. Furthermore, Supabase rolled out realtime listeners that allow developers to subscribe to database changes comparable to Firebase functionality. The platform also added instant RESTful APIs and auto-updating documentation generated dynamically by introspecting modifications to the underlying Postgres schema.
Paul CopplestoneGrab ·
Go Modules- A Guide for monorepos (Part 1)
Grab transitioned its large Go monorepo dependency management from Glide to Go modules while retaining an existing vendor directory structure. The team generated root go.mod configurations from glide.yaml and used go mod vendor without directly enabling module-mode builds. Incompatible nested sub-vendor paths were excluded by placing empty go.mod files, relying on the rule that modules cannot contain other modules. Post-migration maintenance revealed challenges with dependency inheritance and implicit go.mod updates during builds, which engineers investigated using go mod graph and digraph to trace dependency paths.
Michael CartmellSupabase ·
Supabase Alpha April 2020
The April 2020 alpha release of Supabase introduces developer tools designed to make Postgres as straightforward to use as Firebase. Users can configure a Postgres instance in less than two minutes and execute database queries directly from the dashboard interface. The platform provides auto-generated RESTful APIs along with auto-updating documentation created through database schema introspection. Development work also encompasses realtime listeners for database change subscriptions, query performance analysis tools, and connectors that route realtime updates to queues or webhooks. Supabase is not yet production ready, storing only GitHub usernames and emails while running for free under Digital Ocean's startup program.
Paul CopplestoneZoom ·
Developer Platform updates Spring 20'
Zoom introduced several updates across its developer platform alongside new roadmap milestones for 2020. Marketplace enhancements include a Developer Dashboard tracking install numbers and API usage, as well as dynamic OAuth scope configurations that distinguish mandatory and optional permissions. For client development, the WebSDK adds Microsoft Edge Chromium compatibility with video and VOIP support, whereas the iOS SDK incorporates Xcode 11 support. Due to increased developer demand, Zoom updated its V2 REST API rate limits and delivered new chat webhooks tracking message, channel, and member activity. Upcoming platform initiatives include the permanent termination of all legacy V1 APIs on May 31, 2020, platform security enhancements, and an iframe-free JavaScript embedding framework for the WebSDK.
Michael PurnellZoom ·
Backwards Incompatible V2 API Changes
Following the removal of Zoom's Attention Tracking feature, backwards-incompatible modifications were introduced to the V2 REST API effective April 3, 2020. Under these deprecations, the attentiveness_score field in meeting and webinar participant reports returns an empty value, while the attention_tracking field across user, group, and account settings endpoints defaults to false. Both attention-tracking fields remain temporarily in API responses before full removal in a future release. Additionally, starting April 12, 2020, password-related configuration fields become immutable and permanently set to true for Free and single-host Pro K-12 Education accounts. These locking measures affect settings such as password requirements for scheduled, instant, or personal meetings, alongside PSTN password protection across account, group, and user endpoints.
Benjamin DeanGrab ·
Does Southeast Asia Run on Coffee?
GrabFood examined regional coffee ordering trends across major Southeast Asian cities over a nine-month period. Regional coffee orders expanded by 1,400%, with most countries recording their highest order volumes on Wednesdays before tapering off toward the weekend. Singapore and the Philippines deviated from this regional pattern, experiencing spikes in coffee orders on weekends and particularly on Sundays. Daily peak ordering times also differed across markets, peaking at 10:00 AM in Thailand, 2:00 PM in Indonesia, and 4:00 PM in Singapore. In addition to coffee, Green Tea Latte emerged as a top ten beverage item on the platform, accounting for over 25 million delivered cups.
Siu Sing LaiGrab ·
GrabChat Much? Talk Data to Me!
Grab launched GrabChat in September 2016 to facilitate in-app communication between passengers and driver-partners, subsequently adding features such as instant translation, image sharing, voice audio, and templates. An analysis of messaging data across Singapore, Malaysia, and Indonesia revealed distinct behavioral differences, with Indonesia averaging the highest volume at 5.5 chats per booking. Chat volume and the proportion of location-oriented messages peak between midnight and 4am across all three markets, driven by closed storefronts and obscure pickup spots. Image sharing sees higher utilization in remote areas, while pre-populated templates constitute nearly 50% of driver texts to reduce manual typing while driving. Furthermore, messaging responsiveness, directional guidance, and courtesy terms like saying thank you correlate directly with higher trip completion rates.
Jason Lee Jie ShienGrab ·
7 Fun Facts about Grab’s Driver-Partners in Singapore
Grab analyzed ride-hailing metrics from driver-partners operating in Singapore to identify platform usage trends and driving patterns. Findings indicate that drivers have a 1 in 400 chance of encountering a repeat passenger among the 5.4 million population, with Tampines recording the most pickups and Orchard and Marina Bay serving as top destinations in 2018. Driver behavior data shows that partners with over two years of platform experience routinely start shifts an hour earlier and leverage auto-accept features to minimize idle waiting time. Furthermore, drivers are twice as likely to receive back-to-back ride allocations during evening peak hours, resulting in roughly 50% higher hourly earnings. The dataset also highlights customer satisfaction metrics, showing that shared GrabShare rides achieved an average rating of 4.8 stars.
Lara PuReum YimGrab ·
Tackling UI Test Execution Time Imbalance for Xcode Parallel Testing
Parallel test execution in Xcode can suffer from test time imbalance when tasks finish at significantly different times across parallel simulator workers. Analysis of Xcode scheduling logs shows that the runner groups tests by test class and dispatches all tests from the same class to a single simulator. Attempts to customize the suite by swizzling XCTestSuite fail because made-up suites initialize only after tests are dispatched. To overcome this grouping constraint, unique tokens or test names are appended to the class name component in `-only-testing` command-line arguments. This trick forces Xcode to treat each test as an independent class, successfully distributing individual tests across separate workers.
Ngoc Thuyen Trinhhuggingface.co ·
How to generate text: using different decoding methods for language generation with Transformers
Auto-regressive language generation relies heavily on decoding methods to convert predicted next-token probability distributions into coherent text. Greedy search selects the single most probable token at each timestep, but it frequently leads to repetitive phrasing and misses high-probability sequences hidden behind low-probability intermediate words. Beam search alleviates this issue by maintaining multiple parallel hypotheses and can be paired with n-gram penalties to reduce immediate repetitions. Sampling approaches, including top-K and top-p sampling, offer alternative paths that can yield more fluent, varied open-ended generation. Using the Hugging Face transformers library, practitioners can configure these strategies via parameters on the model generate function.
Patrick von PlatenGrab ·
Returning 575 Terabytes of Storage Space to Our Users
Android Vitals data revealed that 15.7% of Grab users had less than 1GB of free device storage and uninstalled the app at 1.2 times the normal rate. To understand on-device storage consumption, the team instrumented session launches using the Android StorageManager API to collect binary size, cache folder size, and total footprint metrics. Analysis showed unusually large cache sizes driven by orphaned cache folders from discontinued third-party libraries, including an image library replacement of Picasso by Glide. An automated cleanup routine deployed in app updates purged legacy cache directories upon launch. This mechanism reclaimed 575 terabytes of junk data across more than 13 million devices, averaging 40MB per user.
Lucas Nelaupe
