Loading…
Topic
Open Source
563 posts about Open Source. Every summary links to the original.
- 563 posts
- 36 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Introducing Optimum: The Optimization Toolkit for Transformers at Scale
Running Transformer models fast and efficiently at production scale presents a significant challenge due to the complex compatibility between model architectures, acceleration techniques like quantization or sparsity, and specific silicon hardware features. Hugging Face introduced Optimum, an open-source library built to abstract hardware acceleration complexity and optimize training and inference on target devices through direct collaborations with hardware partners. Using quantization as a primary example, engineering teams often struggle with eager-mode model graph modifications, kernel compatibility checks, calibration parameter tuning, and acceptable accuracy loss trade-offs. Optimum addresses these challenges by integrating tools such as Intel Neural Compressor, which supports post-training quantization, quantization-aware training, and dynamic quantization driven by user-defined YAML configurations. Through these hardware-tailored integrations, the library aims to democratize production performance and reduce the aggregate energy consumed by machine learning workloads.
Supabase Beta August 2021
Supabase announced a thirty million dollar Series A funding round alongside multiple product updates and platform improvements during August 2021. The team opened an RFC for WALRUS, an initiative aimed at bringing Row Level Security to Postgres subscriptions via Write Ahead Log Realtime Unified Security. Auth updates included customizable SMS login messages, HCaptcha integration for passwordless authentication, and a revised email change flow that notifies both old and new addresses. Platform enhancements introduced a twelfth hosting region in Seoul, improved CSV importing for large files, and Dart reference documentation covering CRUD, Auth, and Realtime operations. Supabase reiterated an ongoing engineering focus on security, performance, and reliability to prepare the platform for production readiness across all workloads.
Open Source Hackathon Winners
Supabase announced the winners and runners-up for its inaugural open-source hackathon after concluding deliberations across a larger-than-expected pool of entries. Top entries included Work From, an employee schedule tracking application, and Fireplace, a video streaming platform that leverages Supabase Storage to synchronize playback events across multiple viewers. Additional recognized projects encompassed Feature.so for modular application component management, Flutter Add-ons for telemetry and crash reporting, and Peckish Peach, which uses machine learning to generate recipes from input ingredients. All submitted projects remain fully open source for community inspection, contribution, and forking. Gold and silver prize winners receive custom limited-edition Supabase apparel, while every hackathon participant receives a participation prize.
The Supabase Hackathon
Following the conclusion of Launch Week II: The SQL, Supabase announced its first week-long hackathon running from July 30 to August 6, 2021. Participants can enter individually or in teams of up to five to build any open-source tool, library, or application using Supabase. The judging criteria evaluate creativity, visual appeal, technical execution, and the integration of Supabase features across five distinct prize categories. All code and design elements must be created during the event and submitted via a repository link prior to the deadline. Winning teams receive limited edition Olympic gold and silver themed shirts while retaining full ownership of their intellectual property.
Supabase Swag Store
Supabase has officially launched the Supabase Swag Store to meet ongoing community demand for branded merchandise. The company built the online storefront using Shopify to go live quickly during launch week rather than dogfooding internal systems for e-commerce. Gifting remains the primary goal of the swag program, prioritizing contributors who create content, report bugs, answer community questions, or submit pull requests. All items are designed, packed, and dispatched directly by the internal team from Singapore in limited production batches of around one hundred items per run. Due to hands-on fulfillment and ongoing international shipping disruptions, inventory is scarce, and some small-batch designs will remain permanently limited editions.
Supabase Auth v2: Phone Auth now available
Supabase has released an update to its authentication server, a fork of Netlify's GoTrue, introducing phone-based authentication with SMS one-time passwords. Developers can now configure Twilio credentials in the Supabase Dashboard to support passwordless SMS logins or combine phone numbers with passwords for user authentication. The release also expands third-party login options by adding Twitch alongside existing integrations, bringing the total number of supported OAuth providers to ten. To assist teams needing dynamic or internationalized messaging, a new API endpoint exposed through supabase-js allows backend services using admin keys to generate confirmation, recovery, invite, and magic links directly. These features build on Supabase Auth's architecture, which stores credentials directly in PostgreSQL databases to enable granular Row Level Security policies across database tables and storage buckets.
Supabase Storage now in Beta
Supabase released Storage Beta with several functional and architectural improvements three months after its alpha launch. The storage service now supports the Content-Range HTTP header for streaming audio and video, alongside entirely public buckets configured with dashboard safeguards and standard authorization policies. Directory uploads are enabled via the File and Directory entries API, while self-hosted instances can use a new local filesystem backend rather than requiring AWS S3 configuration. To support Node.js without adding polyfills or bundling complexity, the storage server was updated to accept binary data and streams directly. Additionally, the dashboard storage explorer simulates filesystem directories using hidden placeholder files and implements list virtualization.
Epsilon3 Self-Host Supabase To Revolutionize Space Operations
Epsilon3 digitizes paper procedures and telemetry data for high-stakes space missions and complex operational testing. Operating in heavily regulated environments requires strict security, ITAR compliance, and flexible deployment options ranging from AWS GovCloud to on-premises installations. To meet these demands without sacrificing developer velocity, Epsilon3 deployed a self-hosted Supabase configuration running against an Amazon RDS Postgres instance on AWS. This architecture gives the team real-time data streaming capabilities over Postgres so operators can track live procedure changes as they happen. In addition to reducing DevOps complexity, self-hosting Supabase enables Epsilon3 to leverage Postgres Row Level Security while ensuring full compliance across diverse deployment environments.
Supabase Community Day
Supabase announced a series of platform upgrades and open-source contributions centered around PostgreSQL and its broader tooling ecosystem. Every new project on the platform now deploys on PostgreSQL version 13.3, accompanied by marketplace images on Digital Ocean and AWS. PostgREST version 8.0 incorporates upstreamed performance enhancements, achieving up to 50% more throughput on GET requests by adopting prepared statements for SELECT queries and decreasing log verbosity. The release eliminates downtime during schema cache reloads and replaces the pg_listen requirement with standard NOTIFY commands for dynamic in-database configuration updates. Additionally, community-led efforts introduced beta releases of Flutter and Dart client libraries alongside a dedicated Discord server for developer discussions.
Supabase Launch Week II: The SQL
Supabase scheduled Launch Week II: The SQL for the week of Monday 26 July 2021, occurring four months after its initial launch week held in March. The five-day schedule opens by spotlighting the latest developments from the community alongside core ecosystem tools including Postgres and PostgREST. Tuesday introduces Storage Reloaded, representing four months of further engineering work following the Version 1 storage release. Wednesday focuses on authentication capabilities, specifically passwordless login features that build upon existing platform support for eight distinct OAuth providers. The week concludes with platform dashboard improvements alongside announcements regarding Supabase Functions updates, a community hackathon, and the official Supabase Swag Store.
Deep Learning over the Internet: Training Language Models Collaboratively
Pretraining state-of-the-art Transformer language models typically requires vast computing resources and expensive hardware clusters that are inaccessible to many researchers. Standard distributed deep learning struggles over public Internet infrastructure because consumer connections suffer from limited bandwidth, dynamic peer disconnections, and restrictive firewalls. To resolve these networking bottlenecks, Distributed Deep Learning in Open Collaborations (DeDLOC) introduces an adaptive training framework that accumulates massive batch sizes across heterogeneous volunteer devices before applying optimizer updates. The framework splits gradient vectors dynamically based on participant connection speeds and allows firewalled nodes to contribute data without performing aggregation. In a practical deployment with 40 volunteers, the 18-million-parameter sahajBERT model achieved Bengali downstream benchmark performance comparable to much larger multilingual models trained on dedicated supercomputers.
Welcome spaCy to the Hugging Face Hub
Hugging Face has integrated spaCy pipelines into the Hugging Face Hub to simplify the distribution, testing, and deployment of natural language processing models across the community. More than 60 canonical models from the spaCy 3.1 release are hosted in the spaCy organization, alongside discoverable community pipelines. Users can package and upload models directly through the command line using the spacy-huggingface-hub extension, which auto-generates required metadata and model cards. Hosted pipelines offer direct pip installation from live wheel URLs, interactive browser exploration, and production-ready HTTP endpoints through the Inference API. The platform provides immediate interactive widget support for named entity recognition components, with upcoming support planned for text classification and part-of-speech tagging pipelines.
Protecting reserved roles with PostgreSQL Hooks
Supabase required granting database customers the CREATEROLE privilege to manage custom roles while preventing them from dropping or altering backend service roles such as supabase_storage_admin. Because stock PostgreSQL lacks a native mechanism to define custom reserved roles, the team created the SupaUtils extension using PostgreSQL hooks. Loaded via shared_preload_libraries, SupaUtils intercepts utility statements by overriding the global ProcessUtility_hook function pointer. The extension inspects incoming statements like ALTER ROLE and DROP ROLE against a configurable list defined in postgresql.conf using DefineCustomStringVariable. If a targeted role matches the configured reserved roles, the hook raises an error, preserving backend infrastructure roles while permitting broader role administration.
Sentence Transformers in the Hugging Face Hub
Sentence Transformers has collaborated with the Hugging Face Hub as part of its v2 release to make sentence, paragraph, and image embedding models readily accessible. The Hub now hosts over 90 pretrained models supporting more than 100 languages for use cases like semantic search and multilingual zero-shot classification. Alongside direct model loading in minimal code, the platform provides interactive Hub widgets for feature extraction and sentence similarity computation. Developers can execute inference programmatically through Inference API endpoints. Additionally, the library supports direct model uploading to the Hub via save_to_hub, which automatically generates a model card detailing layer architecture and dual usage with Sentence Transformers and Transformers.
Let’s Encrypt x Shopify: Securing the Web 4.5 Million Domains at a Time
Shopify describes how its SSL team moved merchants’ stores to HTTPS and ultimately secured more than 4.5 million domains with Let’s Encrypt. The effort began against certificate providers whose APIs were not designed for fully automated provisioning: errors often arrived as human-readable messages, and imposed throttling could have stretched initial issuance to about 100 days. Shopify tested the ACME protocol, valued its standardized specification and open-source implementation, and first added Let’s Encrypt as a backup certificate authority to improve emergency key or certificate-chain rotation. The team rolled out that integration in a few hours, compared with months for earlier providers, then made Let’s Encrypt its primary authority after finding it reliable. The account presents this change as evidence that ACME can make certificate automation more responsive and allow certificate authorities to be added without redesigning Shopify’s infrastructure.
Updating Illustrations at Scale
Shopify’s Polaris team needed to update an illustration style across a product whose assets had become fragmented across more than 6,000 repositories. Illustrators and developers could not reliably identify an image’s repository, code references, reuse, or file type, while GitHub API limits and search behavior prevented dependable large-scale discovery. An initial Node.js monorepo cloned repositories, used globby to find PNG, SVG, and WebP files, searched source lines, and served JSON results through a Preact interface, but rebuilding 24 repositories took more than 35 minutes and produced inaccurate data. The team replaced it with @shopify/get-repo-images, a Go executable that parallelizes file operations, bundles crawling and site generation, and exposes a searchable Next.js interface through NPM. Search and build time fell to eight minutes, with repository cloning as the remaining bottleneck, and the tool was open-sourced.
Few-shot learning in practice: GPT-Neo and the 🤗 Accelerated Inference API
Limited labeled data often prevents machine learning models from achieving high performance during traditional fine-tuning. Few-shot learning addresses this hurdle in natural language processing by supplying a short task description and a few demonstration examples at inference time. EleutherAI developed GPT-Neo, an open transformer-based model trained on the Pile dataset, which can execute few-shot tasks via the Hugging Face Accelerated Inference API. Compared to larger models like GPT-3, the 2.7B parameter version of GPT-Neo requires three to four examples alongside tuned hyperparameters like temperature and stopping sequences to generate accurate completions. Because few-shot learning relies heavily on pre-trained associations, careful output monitoring and human feedback mechanisms remain necessary to prevent automated bias.
Supabase Beta May 2021
Supabase published its May 2021 beta updates covering authentication, regional infrastructure, and tooling. Developers can now implement Apple and Twitter authentication via Supabase Auth, addressing App Store requirements and expanding third-party login options. The platform introduced a Policy Editor with templates to configure Row Level Security for Storage, alongside CSV data retrieval capabilities. Infrastructure expanded with a new Tokyo region to support developers in Japan and China, while Supabase Grid was publicly released as part of an effort to open-source the dashboard. Additionally, community contributors advanced client libraries, completing postgrest-go and progressing development on Swift libraries including gotrue-swift, realtime-swift, and storage-swift.
Using & Mixing Hugging Face Models with Gradio 2.0
The Hugging Face Model Hub hosts more than 10,000 user-submitted machine learning models spanning natural language processing, image classification, and audio processing. Machine learning developers increasingly need to demo models to interdisciplinary collaborators or non-programmers to uncover biases and failure points. The Gradio 2.0 library enables users to create graphical user interfaces and load almost any Hugging Face model using a single line of code. By default, these interfaces query Hugging Face's hosted Inference API without requiring an API key, though users can also run computations locally via the transformers library. Gradio interfaces can be customized through the Interface class and composed to load multiple models in parallel or chain them in series.
Supabase Beta April 2021
Supabase focused on platform stability, security enhancements, and community support during its April 2021 beta update cycle. To address visual accessibility needs for developers with astigmatisms or those operating in brightly lit environments, the team delivered Light Mode within the dashboard settings. The release also introduced OpenAPI-powered Storage API documentation, community translations for the main repository spanning nineteen distinct languages, and an experimental synchronization server connecting Postgres with Stripe. Dashboard components are systematically migrating into an open-source UI library, establishing necessary foundations for local development tooling and self-hosted environments. Additionally, the team reported substantial contributor growth across its realtime server and UI repositories while continuing engineering on a Workflows engine aimed at a July release.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.