Loading…
Latest reads
The engineering internet, summarised so you can actually read it.
Supabase ·
Epsilon3 Self-Host Supabase To Revolutionize Space Operations
Epsilon3 digitizes paper procedures and telemetry data for high-stakes space missions and complex operational testing. Operating in heavily regulated environments requires strict security, ITAR compliance, and flexible deployment options ranging from AWS GovCloud to on-premises installations. To meet these demands without sacrificing developer velocity, Epsilon3 deployed a self-hosted Supabase configuration running against an Amazon RDS Postgres instance on AWS. This architecture gives the team real-time data streaming capabilities over Postgres so operators can track live procedure changes as they happen. In addition to reducing DevOps complexity, self-hosting Supabase enables Epsilon3 to leverage Postgres Row Level Security while ensuring full compliance across diverse deployment environments.
Supabase ·
Supabase Community Day
Supabase announced a series of platform upgrades and open-source contributions centered around PostgreSQL and its broader tooling ecosystem. Every new project on the platform now deploys on PostgreSQL version 13.3, accompanied by marketplace images on Digital Ocean and AWS. PostgREST version 8.0 incorporates upstreamed performance enhancements, achieving up to 50% more throughput on GET requests by adopting prepared statements for SELECT queries and decreasing log verbosity. The release eliminates downtime during schema cache reloads and replaces the pg_listen requirement with standard NOTIFY commands for dynamic in-database configuration updates. Additionally, community-led efforts introduced beta releases of Flutter and Dart client libraries alongside a dedicated Discord server for developer discussions.
Steve ChavezSupabase ·
Supabase is now on Postgres 13.3
New Supabase projects now run on Supabase Postgres 13.3, transitioning the database platform forward from PostgreSQL 12.0. This release introduces features such as native UUID generation with gen_random_uuid, parallel index vacuuming, incremental sorting, smaller btree indexes, and extended statistics improvements. The underlying PostgreSQL binary was built from the ground up to enable Large System Extensions on AWS Graviton2 ARM instances, optimizing locking and synchronization performance. Alongside upgrading the underlying operating system to Ubuntu 20.04 for long-term support through 2025, the team integrated fail2ban to protect direct database connections on ports 5432 and 6543 against brute-force attacks. Finally, Supabase Postgres version numbering now mirrors upstream PostgreSQL major and minor versions to make tracking installed releases easier.
Angelico de los ReyesGrab ·
Protecting Personal Data in Grab's Imagery
Grab's KartaView platform collects geotagged street imagery across over 100 countries, requiring automated obfuscation of faces and licence plates to protect personal privacy. Because off-the-shelf solutions struggled with diverse global environments and equirectangular 360-degree camera formats, Grab built a custom machine learning pipeline. The system projects varied image formats into standardized planar views, applies a YOLOv4 object detection model to locate target regions, and transforms bounding coordinates back to the original imagery for blurring. Training the detector required iterative dataset updates to accommodate edge cases like face masks and mirror reflections, paired with offline view splitting and oversampling of scarce large bounding boxes. Assessments confirmed that obfuscating these regions had minimal negative impact on downstream map feature extraction services.
Adrian PopoviciSupabase ·
Supabase Launch Week II: The SQL
Supabase scheduled Launch Week II: The SQL for the week of Monday 26 July 2021, occurring four months after its initial launch week held in March. The five-day schedule opens by spotlighting the latest developments from the community alongside core ecosystem tools including Postgres and PostgREST. Tuesday introduces Storage Reloaded, representing four months of further engineering work following the Version 1 storage release. Wednesday focuses on authentication capabilities, specifically passwordless login features that build upon existing platform support for eight distinct OAuth providers. The week concludes with platform dashboard improvements alongside announcements regarding Supabase Functions updates, a community hackathon, and the official Supabase Swag Store.
Paul CopplestoneGrab ·
Processing ETL tasks with Ratchet
Grab's Lending team relies on multiple microservices to finance various user segments, requiring automated pipelines for operational tasks like loan rescheduling and merchant whitelisting. To execute these workflows, the team uses Ratchet, a Go library that organizes data pipelines into concurrent stages connected by Go channels and JSON payloads. In the Azkaban service, each ETL task is structured as a Job Processor that configures custom Data Processors for extraction, transformation, and loading. For merchant whitelisting, a custom processor pulls uploaded CSV files from Amazon S3, validates the rows to prevent manual errors, calls microservice APIs, and passes the output to an empty load stage. This pipeline architecture allows the team to isolate failures quickly, eliminate manual data validation, and automate routine operational tasks.
Amar Prakashhuggingface.co ·
Deep Learning over the Internet: Training Language Models Collaboratively
Pretraining state-of-the-art Transformer language models typically requires vast computing resources and expensive hardware clusters that are inaccessible to many researchers. Standard distributed deep learning struggles over public Internet infrastructure because consumer connections suffer from limited bandwidth, dynamic peer disconnections, and restrictive firewalls. To resolve these networking bottlenecks, Distributed Deep Learning in Open Collaborations (DeDLOC) introduces an adaptive training framework that accumulates massive batch sizes across heterogeneous volunteer devices before applying optimizer updates. The framework splits gradient vectors dynamically based on participant connection speeds and allows firewalled nodes to contribute data without performing aggregation. In a practical deployment with 40 volunteers, the 18-million-parameter sahajBERT model achieved Bengali downstream benchmark performance comparable to much larger multilingual models trained on dedicated supercomputers.
Max Ryabinin, Lucile SaulnierGrab ·
App Modularisation at Scale
Grab transitioned its monolithic mobile application into a modular architecture to resolve increasing code conflicts, slow releases, and difficult team collaboration. The team decomposed the single module by establishing base infrastructure modules, shared UI and utility libraries, discrete feature modules, and bridge kit modules for inter-module communication. Dependency injection using Dagger ties these components together in the main app module while preventing feature modules from directly depending on one another. The architecture spans over 1,000 modules across the app, with more than 200 modules in the Grab Financial Group payments domain where over 95% of modules build in under 15 seconds. This approach accelerated Gradle CI and local builds through parallel compilation and caching, though it increased Gradle sync times, IDE memory usage, and configuration maintenance overhead.
Amar Jainhuggingface.co ·
Welcome spaCy to the Hugging Face Hub
Hugging Face has integrated spaCy pipelines into the Hugging Face Hub to simplify the distribution, testing, and deployment of natural language processing models across the community. More than 60 canonical models from the spaCy 3.1 release are hosted in the spaCy organization, alongside discoverable community pipelines. Users can package and upload models directly through the command line using the spacy-huggingface-hub extension, which auto-generates required metadata and model cards. Hosted pipelines offer direct pip installation from live wheel URLs, interactive browser exploration, and production-ready HTTP endpoints through the Inference API. The platform provides immediate interactive widget support for named entity recognition components, with upcoming support planned for text classification and part-of-speech tagging pipelines.
Omar Sanseviero, Ines Montanihuggingface.co ·
Deploy Hugging Face models easily with Amazon SageMaker
Hugging Face and Amazon introduced an inference solution integrating Hugging Face Transformers with Amazon SageMaker. The SageMaker Hugging Face Inference Toolkit and specialized Deep Learning Containers enable deployment of trained models or publicly available Hub models to managed production endpoints. Users configure endpoints with minimal code using the SageMaker Python SDK, standard pipelines, or custom inference scripts that override default behaviors. The architecture supports models stored in Amazon S3 as well as direct references via HF_MODEL_ID and HF_TASK environment variables. Deployments benefit from native AWS infrastructure features, including built-in monitoring, Identity and Access Management permission controls, and Virtual Private Cloud connectivity.
Philipp SchmidGrab ·
Reshaping Chat Support for Our Users
Grab transitioned from voice hotlines and third-party tools to an in-house native chat support system integrated into their CRM. The team validated the platform through an MVP and user shadowing to address session disconnections, agent context switching, and routing bottlenecks. To optimize support operations at scale, they introduced dynamic queue limits based on Little's law, machine learning autocomplete suggestions for agents, and duration timers with visual nudges. These enhancements reduced chat waiting times by 30%, unresponsive users by 7%, and overall chat handling duration by 22%.
Elisa MonacchiSupabase ·
Protecting reserved roles with PostgreSQL Hooks
Supabase required granting database customers the CREATEROLE privilege to manage custom roles while preventing them from dropping or altering backend service roles such as supabase_storage_admin. Because stock PostgreSQL lacks a native mechanism to define custom reserved roles, the team created the SupaUtils extension using PostgreSQL hooks. Loaded via shared_preload_libraries, SupaUtils intercepts utility statements by overriding the global ProcessUtility_hook function pointer. The extension inspects incoming statements like ALTER ROLE and DROP ROLE against a configurable list defined in postgresql.conf using DefineCustomStringVariable. If a targeted role matches the configured reserved roles, the hook raises an error, preserving backend infrastructure roles while permitting broader role administration.
Steve ChavezGrab ·
Debugging High Latency Due to Context Leaks
Market-Store, Grab's feature store for real-time machine learning features, experienced latency spikes from under 200 milliseconds to 2 seconds as traffic grew. Metrics and logs showed no direct correlation to API issues, but heap profiling with PPROF revealed continuously increasing memory held by child contexts. Further analysis tracked the leak to an update in Grab's open-source Async Library, which switched background contexts to uncancelled task contexts for worker runners. Because parent contexts maintained references to these uncancelled child contexts, the garbage collector could not reclaim their memory. This progressive memory exhaustion directly degraded API latency.
Sourabh Sumanhuggingface.co ·
Sentence Transformers in the Hugging Face Hub
Sentence Transformers has collaborated with the Hugging Face Hub as part of its v2 release to make sentence, paragraph, and image embedding models readily accessible. The Hub now hosts over 90 pretrained models supporting more than 100 languages for use cases like semantic search and multilingual zero-shot classification. Alongside direct model loading in minimal code, the platform provides interactive Hub widgets for feature extraction and sentence similarity computation. Developers can execute inference programmatically through Inference API endpoints. Additionally, the library supports direct model uploading to the Hub via save_to_hub, which automatically generates a model card detailing layer architecture and dual usage with Sentence Transformers and Transformers.
Omar Sanseviero, Nils Reimershuggingface.co ·
Few-shot learning in practice: GPT-Neo and the 🤗 Accelerated Inference API
Limited labeled data often prevents machine learning models from achieving high performance during traditional fine-tuning. Few-shot learning addresses this hurdle in natural language processing by supplying a short task description and a few demonstration examples at inference time. EleutherAI developed GPT-Neo, an open transformer-based model trained on the Pile dataset, which can execute few-shot tasks via the Hugging Face Accelerated Inference API. Compared to larger models like GPT-3, the 2.7B parameter version of GPT-Neo requires three to four examples alongside tuned hyperparameters like temperature and stopping sequences to generate accurate completions. Because few-shot learning relies heavily on pre-trained associations, careful output monitoring and human feedback mechanisms remain necessary to prevent automated bias.
Philipp SchmidSupabase ·
Supabase Beta May 2021
Supabase published its May 2021 beta updates covering authentication, regional infrastructure, and tooling. Developers can now implement Apple and Twitter authentication via Supabase Auth, addressing App Store requirements and expanding third-party login options. The platform introduced a Policy Editor with templates to configure Row Level Security for Storage, alongside CSV data retrieval capabilities. Infrastructure expanded with a new Tokyo region to support developers in Japan and China, while Supabase Grid was publicly released as part of an effort to open-source the dashboard. Additionally, community contributors advanced client libraries, completing postgrest-go and progressing development on Swift libraries including gotrue-swift, realtime-swift, and storage-swift.
Paul CopplestoneSupabase ·
Supabase Beta June 2021
Supabase announced a series of platform updates, documentation additions, and tooling enhancements for June 2021 ahead of its scheduled Launch Week. Developers can now provision a Postgres database directly through a new Vercel integration and enable authentication using Discord logins. Supabase Storage added support for upsert operations alongside public buckets that allow unauthenticated file access. The web dashboard introduced a restart button for debugging and a table policy editor with templates to configure Row Level Security. Additional backend improvements include timestamps for PostgREST error logs, an OpenAPI mode patch ignoring anonymous privileges for Prisma users, and an XSS fix in the Auth0 Next.js library.
Paul Copplestonehuggingface.co ·
Using & Mixing Hugging Face Models with Gradio 2.0
The Hugging Face Model Hub hosts more than 10,000 user-submitted machine learning models spanning natural language processing, image classification, and audio processing. Machine learning developers increasingly need to demo models to interdisciplinary collaborators or non-programmers to uncover biases and failure points. The Gradio 2.0 library enables users to create graphical user interfaces and load almost any Hugging Face model using a single line of code. By default, these interfaces query Hugging Face's hosted Inference API without requiring an API key, though users can also run computations locally via the transformers library. Gradio interfaces can be customized through the Interface class and composed to load multiple models in parallel or chain them in series.
Abubakar AbidGrab ·
Building a Hyper Self-Service, Distributed Tracing and Feedback System for Rule & Machine Learning (ML) Predictions
Grab's Trust, Identity, Safety, and Security team processes billions of daily rule and machine learning decisions for fraud detection, safety, and identity checks. Earlier logging approaches using plain text Kibana logs and the ActionTrace library lacked structured formats, dynamic entity customization, and fine-grained access controls. To resolve these limitations, the team built Archivist, a centralized tracing, statistics, and feedback system. Archivist ingests events through an SDK into Kafka streams, buffers and routes data into Elasticsearch indices and Amazon S3, and provides a role-based user portal. The platform handles 80 million daily logs across roughly 50 business scenarios, reducing scenario onboarding times from days to minutes.
Warren ZhouGrab ·
Our Journey to Continuous Delivery at Grab (Part 2)
Conveyor, an in-house continuous delivery system at Grab, introduces hermetic deployments by tracking application code alongside static and dynamic configuration parameters. This hermeticity guarantees that production releases use combinations of versioned artifacts previously verified in staging, preventing rollback incompatibilities. Conveyor replaces single multi-environment pipelines with decoupled pipelines, while automating cluster locking, deployment slot scheduling, release note generation, and canary monitoring with automated rollbacks. These automations reduced production deployment failure rates from 1.5% to an average of 0.3% over a three-month period. Grab also doubled the volume of production changes between 2018 and 2020 while saving more than 5,000 man-days of engineering effort in 2020.
Sylvain Bougerel