---
title: "Open Source"
description: "568 posts about Open Source, summarised, each linking to the original."
---

# Open Source
> 568 posts about Open Source, summarised, each linking to the original.

## Articles

### [Updating Illustrations at Scale](https://yomu.fyi/post/updating-illustrations-at-scale.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jun 3, 2021
- Type: Problem & solution

Shopify’s Polaris team needed to update an illustration style across a product whose assets had become fragmented across more than 6,000 repositories. Illustrators and developers could not reliably identify an image’s repository, code references, reuse, or file type, while GitHub API limits and search behavior prevented dependable large-scale discovery. An initial Node.js monorepo cloned repositories, used globby to find PNG, SVG, and WebP files, searched source lines, and served JSON results through a Preact interface, but rebuilding 24 repositories took more than 35 minutes and produced inaccurate data. The team replaced it with @shopify/get-repo-images, a Go executable that parallelizes file operations, bundles crawling and site generation, and exposes a searchable Next.js interface through NPM. Search and build time fell to eight minutes, with repository cloning as the remaining bottleneck, and the tool was open-sourced.


### [Few-shot learning in practice: GPT-Neo and the 🤗 Accelerated Inference API](https://yomu.fyi/post/few-shot-learning-in-practice-gpt-neo-and-the-accelerated-inference-ap.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Philipp Schmid
- Published: Jun 3, 2021
- Type: Tutorial

Limited labeled data often prevents machine learning models from achieving high performance during traditional fine-tuning. Few-shot learning addresses this hurdle in natural language processing by supplying a short task description and a few demonstration examples at inference time. EleutherAI developed GPT-Neo, an open transformer-based model trained on the Pile dataset, which can execute few-shot tasks via the Hugging Face Accelerated Inference API. Compared to larger models like GPT-3, the 2.7B parameter version of GPT-Neo requires three to four examples alongside tuned hyperparameters like temperature and stopping sequences to generate accurate completions. Because few-shot learning relies heavily on pre-trained associations, careful output monitoring and human feedback mechanisms remain necessary to prevent automated bias.


### [Supabase Beta May 2021](https://yomu.fyi/post/supabase-beta-may-2021.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: Paul Copplestone
- Published: Jun 2, 2021
- Type: Announcement

Supabase published its May 2021 beta updates covering authentication, regional infrastructure, and tooling. Developers can now implement Apple and Twitter authentication via Supabase Auth, addressing App Store requirements and expanding third-party login options. The platform introduced a Policy Editor with templates to configure Row Level Security for Storage, alongside CSV data retrieval capabilities. Infrastructure expanded with a new Tokyo region to support developers in Japan and China, while Supabase Grid was publicly released as part of an effort to open-source the dashboard. Additionally, community contributors advanced client libraries, completing postgrest-go and progressing development on Swift libraries including gotrue-swift, realtime-swift, and storage-swift.


### [Using & Mixing Hugging Face Models with Gradio 2.0](https://yomu.fyi/post/using-mixing-hugging-face-models-with-gradio-2-0.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Abubakar Abid
- Published: May 25, 2021
- Type: Announcement

The Hugging Face Model Hub hosts more than 10,000 user-submitted machine learning models spanning natural language processing, image classification, and audio processing. Machine learning developers increasingly need to demo models to interdisciplinary collaborators or non-programmers to uncover biases and failure points. The Gradio 2.0 library enables users to create graphical user interfaces and load almost any Hugging Face model using a single line of code. By default, these interfaces query Hugging Face's hosted Inference API without requiring an API key, though users can also run computations locally via the transformers library. Gradio interfaces can be customized through the Interface class and composed to load multiple models in parallel or chain them in series.


### [Supabase Beta April 2021](https://yomu.fyi/post/supabase-beta-april-2021.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: Paul Copplestone
- Published: May 5, 2021
- Type: Announcement

Supabase focused on platform stability, security enhancements, and community support during its April 2021 beta update cycle. To address visual accessibility needs for developers with astigmatisms or those operating in brightly lit environments, the team delivered Light Mode within the dashboard settings. The release also introduced OpenAPI-powered Storage API documentation, community translations for the main repository spanning nineteen distinct languages, and an experimental synchronization server connecting Postgres with Stripe. Dashboard components are systematically migrating into an open-source UI library, establishing necessary foundations for local development tooling and self-hosted environments. Additionally, the team reported substantial contributor growth across its realtime server and UI repositories while continuing engineering on a Workflows engine aimed at a July release.


### [Introducing 🤗 Accelerate](https://yomu.fyi/post/introducing-accelerate.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Sylvain Gugger
- Published: Apr 16, 2021
- Type: Announcement

🤗 Accelerate enables PyTorch developers to execute raw training loops across CPUs, multi-GPU setups, and TPUs with minimal boilerplate modifications. Standard distributed training typically demands manual device assignments, DistributedSampler configurations, and divergent code paths that break single-device portability. By wrapping models, optimizers, and dataloaders through a unified prepare call, Accelerate automatically manages hardware placement, mixed precision operations, and sampler batch slicing without requiring custom distributed samplers. Distributed evaluation is streamlined using a gather utility that consolidates prediction tensors across active worker processes. Furthermore, a dedicated command-line interface provides an interactive questionnaire to store runtime configuration defaults and orchestrate local or AWS SageMaker runs.


### [Supabase Beta March 2021](https://yomu.fyi/post/supabase-beta-march-2021.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: Paul Copplestone
- Published: Apr 6, 2021
- Type: Announcement

Supabase announced a series of platform updates for March 2021, headlined by the launch of Supabase Storage and new local development capabilities. The new storage offering integrates an open-source Fastify and TypeScript API backed by S3, while direct database connections now bundle PgBouncer for connection pooling. Developers can run the platform locally via the Supabase CLI, utilize Docker tooling for self-hosting, and access the open-sourced Supabase UI component library. Community contributions brought OAuth scopes to Supabase Auth for requesting elevated third-party provider permissions, alongside newly published free and pro pricing tiers. In addition, development is underway on an Elixir-based workflow engine compatible with the Amazon States Language for automated event handling.


### [Supabase Dot Com](https://yomu.fyi/post/supabase-dot-com.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: Paul Copplestone
- Published: Apr 2, 2021
- Type: Announcement

Supabase announced a migration of its primary website from supabase.io to supabase.com over the upcoming week. User API endpoints will remain unchanged on supabase.co, requiring no configuration adjustments for active applications. The company originally selected supabase.io as a meme-inspired placeholder after releasing a Postgres Realtime engine on GitHub in 2019 and failing to find an available dot com name. Following advice from Y Combinator to secure a dot com address, the founders bypassed domain brokers charging percentage fees and negotiated directly with the domain owner, whose business address was located one hundred meters from the founder's home. The migration updates public website destinations while preserving existing developer infrastructure unaffected.


### [PgBouncer is now available in Supabase](https://yomu.fyi/post/pgbouncer-is-now-available-in-supabase.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: Angelico de los Reyes
- Published: Apr 2, 2021
- Type: Announcement

Serverless JavaScript frameworks and developer tools frequently establish separate database connections during bursty traffic, rapidly exhausting PostgreSQL connection limits in the absence of traditional middleware. To resolve connection surges, Supabase integrated the open-source connection pooler PgBouncer directly onto the PostgreSQL server across all newly created projects. Rather than increasing the total number of connections PostgreSQL can open, PgBouncer recycles open connections and queues excess requests until active connections become available. Developers can manage pooling through the dashboard across Session, Transaction, and Statement modes, with Transaction mode recommended for serverless functions despite disabling session features like prepared statements. The default pool size is initially configured to 15 connections.


### [Workflows are coming to Supabase](https://yomu.fyi/post/workflows-are-coming-to-supabase.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: Francesco Ceccon
- Published: Apr 2, 2021
- Type: Announcement

Supabase is developing Workflows, an Elixir-based orchestration engine designed to coordinate complex serverless tasks and event-driven logic. Standard serverless and database-triggered functions often struggle with delayed execution and queuing without external cron processes. To solve this with native Postgres integration, the new engine adopts the open-source Amazon States Language specification to orchestrate functions across platforms including AWS, GCP, Azure, OpenFaaS, and Postgres itself. Execution states, jobs, queues, and logs are deeply integrated with Postgres using the Oban job processing library, though transient in-memory workers are also supported. The engine responds directly to HTTP calls or database change events delivered through Supabase Realtime.


### [Supabase CLI](https://yomu.fyi/post/supabase-cli.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: Bobbie Soedirgo
- Published: Mar 31, 2021
- Type: Announcement

Supabase announces its CLI as the first step toward running Supabase locally, managing database migrations, generating types and API or validation schemas from a database, managing projects, and eventually pushing local changes to production. The local stack runs through Docker Compose, using tools described as 100% compatible with every Supabase project, and a React setup can initialize it with supabase init before supabase start. For schema changes, the post favors database diffing over accumulating manual SQL migrations and adds a CLI mode and Docker image for pgAdmin Schema Diff, which tracks view-owner changes needed for Row Level Security and handles domains. The supabase eject command outputs Docker Compose, Kong, and Postgres configuration for self-hosting. An update dated 15 August 2022 states that CLI V1 is generally available and that a Management API is in beta.


### [Understanding BigBird's Block Sparse Attention](https://yomu.fyi/post/understanding-bigbird-s-block-sparse-attention.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Vasudev Gupta
- Published: Mar 31, 2021
- Type: Explainer

Transformer-based models face severe computational bottlenecks due to quadratic time and memory complexity, making sequences longer than 512 tokens impractical. BigBird addresses this limitation by approximating full attention with block sparse attention, scaling context lengths up to 4096 tokens at a lower computational cost. Instead of attending to all tokens, BigBird combines sliding attention for local context, global tokens for long-range relationships, and random tokens to accelerate information transfer. The model is integrated into HuggingFace Transformers, enabling fine-tuning for tasks like long-document summarization and extractive question answering. When sequences are shorter than 1024 tokens, using original full attention remains recommended.


### [Angels of Supabase](https://yomu.fyi/post/angels-of-supabase.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: Paul Copplestone
- Published: Mar 25, 2021
- Type: Announcement

Supabase announced a 6 million dollar seed round led by Coatue, Mozilla, and Y Combinator, accompanied by over twenty technical angel investors. To guide its mission of providing an open-source alternative to Firebase, the company deliberately assembled a roster of angel backers comprising CTOs, developers, and developer advocates. The post features short interviews with several angel investors who share their career backgrounds, coding habits, and rationale for supporting the platform. Many investors highlighted the appeal of Supabase's developer ergonomics, its open-source foundation, and the advantages of utilizing Postgres directly alongside built-in authentication over proprietary NoSQL alternatives. Supabase also teased its upcoming Launch Week, during which the team plans to release one new feature daily for a week.


### [Launch week](https://yomu.fyi/post/launch-week.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: Ant Wilson
- Published: Mar 25, 2021
- Type: Announcement

Following its transition from Alpha to Beta, Supabase scheduled an intensive Launch Week to deliver multiple major features built over a three-month cycle. The event opened with a pricing model designed around predictable metrics like storage to prevent unexpected billing spikes from API requests. The team subsequently launched Supabase Storage, combining a Fastify Node.js API layer with Kong authentication and Postgres Row Level Security for policy evaluation. Additional launches delivered a CLI supporting database migrations and Docker self-hosting, alongside an open-source dashboard UI component library. Finally, Supabase added PgBouncer connection pooling for Postgres instances, previewed an Elixir workflow engine, and transitioned its primary domain to supabase.com.


### [The Partnership: Amazon SageMaker and Hugging Face](https://yomu.fyi/post/the-partnership-amazon-sagemaker-and-hugging-face.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Philipp Schmid
- Published: Mar 23, 2021
- Type: Announcement

Hugging Face and Amazon formed a strategic partnership establishing Amazon Web Services as Hugging Face's preferred cloud provider to simplify training and deploying Transformer models. To facilitate this collaboration, the teams introduced Hugging Face Deep Learning Containers optimized for PyTorch and TensorFlow in Amazon SageMaker. The integration provides an extension to the SageMaker Python SDK, enabling data science teams to configure distributed training jobs across single-node and multi-node GPU clusters. Workflows integrate directly with SageMaker Automatic Model Tuning, SageMaker Studio, and Amazon S3 for managed experiment tracking and artifact storage. The open-source containers allow trained models to be exported for inference via SageMaker or published directly to the Hugging Face Model Hub while utilizing EC2 Spot instances to reduce compute expenses.


### [Developers stay up to date with intheloop.dev](https://yomu.fyi/post/developers-stay-up-to-date-with-intheloop-dev.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: Rory Wilding
- Published: Mar 22, 2021
- Type: Problem & solution

Freelance engineer Kevin Grüneberg created intheloop.dev as an open-source aggregator to help developers track software updates, release notes, changelogs, tweets, and community discussions from a single platform. The initial release provides dedicated coverage for React, Kotlin, and Vite, with plans to introduce user accounts, notifications, and personalized digests. Although Grüneberg previously used Firebase for rapid prototyping, he found relational requirements like count queries, normalization, and foreign keys difficult or impossible to implement. He selected Supabase as the backend to combine fast integration with PostgreSQL capabilities, including SQL queries, constraints, views, and materialized views. This database architecture allows the application to handle structured data relationships efficiently while scaling future feature additions without restructuring core models.


### [Fine-Tune Wav2Vec2 for English ASR in Hugging Face with 🤗 Transformers](https://yomu.fyi/post/fine-tune-wav2vec2-for-english-asr-in-hugging-face-with-transformers.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Patrick von Platen
- Published: Mar 12, 2021
- Type: Tutorial

Wav2Vec2 is a pretrained Automatic Speech Recognition model that learns speech representations from over 50,000 hours of unlabeled audio using a contrastive objective and masked feature vectors. The model can be fine-tuned end-to-end on labeled speech datasets using Connectionist Temporal Classification without requiring an external language model. To demonstrate this process, a base-sized checkpoint is fine-tuned on the Timit dataset, which contains five hours of training data. Data preparation involves normalizing transcriptions to lowercase, removing special punctuation characters, and configuring both a specialized feature extractor and tokenizer. Evaluating the fine-tuned acoustic model without padding on the Timit test dataset yields a word error rate of 22.1%.


### [Hugging Face Reads, Feb. 2021 - Long-range Transformers](https://yomu.fyi/post/hugging-face-reads-feb-2021-long-range-transformers.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Victor Sanh
- Published: Mar 9, 2021
- Type: Explainer

Standard transformer models face severe memory and compute bottlenecks due to the quadratic scaling of self-attention with sequence length. To scale beyond short sequence limits, efficient architectures introduce varied mechanisms such as custom sparse attention patterns, compressed recurrence, low-rank projections, and kernel approximations. Longformer integrates local dilated windowed attention with task-specific global tokens, functioning as a drop-in replacement that enables standard pre-trained models to process long inputs without full retraining. Concurrently, Compressive Transformers extend temporal memory by compressing discarded activations, while models like Linformer and Performer reduce computational complexity through projection and kernel estimation. Progress in long-range modeling also reveals that incrementally training models from short to longer sequences accelerates convergence and improves downstream performance.


### [Supabase Beta February 2021](https://yomu.fyi/post/supabase-beta-february-2021.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: Paul Copplestone
- Published: Mar 2, 2021
- Type: Announcement

Supabase published its February 2021 beta update marking one year of development as an open-source Firebase alternative. The release introduces dashboard interface improvements across the Table view, Auth section, and SQL Editor, which now features autocomplete suggestions for tables and columns. Authentication updates allow user redirection to specific site routes upon invoking signIn and signUp, while a new database management interface lets users configure Postgres replication settings and events for the realtime API. Infrastructure additions include a new deployment region in South Africa and a Docker Compose configuration intended to power local development in the forthcoming CLI. Documentation expanded with resources for GoTrue and Google OAuth, alongside ongoing RFCs covering connection pooling and local workflows.


### [Hugging Face on PyTorch / XLA TPUs](https://yomu.fyi/post/hugging-face-on-pytorch-xla-tpus.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: ds, Lysandre
- Published: Feb 9, 2021
- Type: Announcement

PyTorch / XLA integrates Cloud TPU training into Hugging Face transformers while preserving the standard Trainer interface. The implementation adds an xla device type and uses lazy execution to trace intermediate representation graphs before lowering them to XLA Higher Level Operations for compilation. Parallel data loading through MpDeviceLoader overlaps CPU tracing with TPU execution, while xm.optimizer\_step coordinates gradient consolidation across data-parallel replicas. Checkpoints use xm.save to transfer tensors via CPU storage and restrict disk writes to the master process. Benchmarks on a Cloud TPU v3-8 system training bert-large-uncased on WikiText-103 demonstrate completion times of 178.4 minutes in FP32 precision and 106.4 minutes in BF16 precision.


[Newer posts](https://yomu.fyi/topic/open-source/page/26.md) · [Older posts](https://yomu.fyi/topic/open-source/page/28.md)