---
title: "Open Source"
description: "568 posts about Open Source, summarised, each linking to the original."
---

# Open Source
> 568 posts about Open Source, summarised, each linking to the original.

## Articles

### [Creating a React Library for Consistent Data Visualization](https://yomu.fyi/post/creating-a-react-library-for-consistent-data-visualization.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Mar 15, 2022
- Type: Problem & solution

Shopify found that teams across its more than 10,000 employees used different visualization tools, producing inconsistent chart styling and differing accessibility and print behavior. The Insights team built Polaris Viz, a React library intended to provide shared, accessible defaults while reducing repeated configuration, and focused this work on light and dark themes plus consumer-defined visual identities. The library replaces numerous visual-style props with a theme prop, stores themes through PolarisVizProvider, and uses createTheme to fill partial definitions with defaults and createThemes to preserve Default and Light themes. Themes can inherit from other themes, while a DataSeries colour can override theme series colors for dataset-specific gradients without creating additional themes. Polaris Viz was subsequently released as an open-source project.


### [Guiding Text Generation with Constrained Beam Search in 🤗 Transformers](https://yomu.fyi/post/guiding-text-generation-with-constrained-beam-search-in-transformers.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Chan Woo Kim
- Published: Mar 11, 2022
- Type: Explainer

Constrained beam search introduces direct control over generated text in Hugging Face Transformers. Standard beam search operates token-by-token without knowing the optimal step to force specific words or phrases, making it difficult to enforce mandatory vocabulary or choose between alternative expressions. To resolve this, the generate interface accepts constraints through arguments such as force\_words\_ids and a list of Constraint subclasses like PhrasalConstraint. This mechanism enables disjunctive constraints, where generation must include at least one phrase from a provided set, alongside strictly required sequences. Consequently, practitioners can inject prior knowledge or formatting requirements directly at generation time rather than filtering candidate outputs afterward.


### [After the Refactor: A Path to Faster Rendering with Liquid-C](https://yomu.fyi/post/after-the-refactor-a-path-to-faster-rendering-with-liquid-c.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Mar 10, 2022
- Type: Explainer

Liquid-C is an extension to the Liquid gem that parses, tokenizes, and renders templates, but some Liquid features, including If and For tags, still render in Liquid rather than Liquid-C. The work explains how Liquid-C compiles templates into bytecode for a customer VM and addresses the difficulty of moving instruction and constant pointers when control flow requires forward or backward jumps. It refactors instructions to store each constant’s index in the constants array, using a hash table to avoid duplicate constants and remove the constant pointer. The change reduced one example constants array from 157 objects to 104, but did not drastically improve theme rendering speed; it slightly increased memory usage and parsing time while preparing for GOTO or JMP operations and later If and For support.


### [Postgres Auditing in 150 lines of SQL](https://yomu.fyi/post/postgres-auditing-in-150-lines-of-sql.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: Oliver Rice
- Published: Mar 8, 2022
- Type: Tutorial

Traditional PostgreSQL auditing methods duplicate source table structures, creating maintenance overhead when schemas change. To solve this, a compact SQL implementation stores insert, update, and delete events across multiple tables in a single audit table using JSONB columns. Query performance is maintained by using a BRIN index on naturally ordered insertion timestamps, a B-tree index on internal table OIDs, and UUIDv5 identifiers hashed from primary keys. Row-level PL/pgSQL triggers automatically populate the audit log and can be enabled or disabled dynamically with dedicated tracking functions. While trigger-based auditing introduces minimal overhead at rates below 1,000 writes per second, write-heavy workloads may benefit from logging changes outside SQL with tools like pgAudit.


### [RubyConf 2021: The Talks You Might Have Missed](https://yomu.fyi/post/rubyconf-2021-the-talks-you-might-have-missed.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Mar 2, 2022
- Type: Announcement

RubyConf 2021: The Talks You Might Have Missed compiles talks presented by Shopify’s Ruby and Rails infrastructure engineers at the November conference in Denver, offering ways to revisit sessions for attendees who missed them. The program covers Ruby implementation topics including compiler history, Aaron Patterson’s pure-Ruby JIT compiler, YJIT’s incremental integration inside CRuby, parser generators, variable-width allocation, and Ruby memory layout. Other sessions address gradual Sorbet adoption, native extension compilation and security, pair programming, memoization performance, executable code as data, Ruby archaeology, fast CI, and Ractor-based parallel testing. The roundup reports early YJIT performance results, describes a CI environment with 2.8 million Rails monolith lines and 210,000 Ruby tests, and notes variable-width allocation passed Shopify’s Rails CI while serving more than 500 million requests in a week of limited production traffic.


### [BERT 101 - State Of The Art NLP Model Explained](https://yomu.fyi/post/bert-101-state-of-the-art-nlp-model-explained.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Britney Muller
- Published: Mar 2, 2022
- Type: Explainer

Developed in 2018 by Google AI Language, Bidirectional Encoder Representations from Transformers addresses the historical challenge of machines lacking contextual understanding of human language. The model relies on an encoder-only Transformer architecture pre-trained on a 3.3-billion-word corpus consisting of Wikipedia and Google BooksCorpus. Training simultaneously combines masked language modeling, which hides 15% of tokenized words to enforce bidirectional context learning, with next sentence prediction across balanced sentence pairs. Pre-trained on Cloud TPUs over four days, BERT unifies solutions for more than eleven common NLP tasks and can be fine-tuned on task-specific annotated data within minutes. Unmasking experiments demonstrate that the model can also inherit distinct societal and gender biases from its underlying training corpora when predicting professions.


### [How to Get an Engineering Internship at Shopify: A Complete Guide](https://yomu.fyi/post/how-to-get-an-engineering-internship-at-shopify-a-complete-guide.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Feb 18, 2022
- Type: Tutorial

Shopify’s guide describes its engineering internship program as a way for students, recent graduates, career switchers, and bootcamp graduates to gain hands-on experience through real projects and practical feedback. Internships last three to eight months and span front-end, back-end, infrastructure, data, and mobile development, with three application cycles each year. Applicants submit an online application, complete an Intern Challenge or specialization-specific skills assessment, and progress through a two-phase interview involving a Life Story conversation and a technical solution. The guide identifies collaboration, lifelong learning, curiosity, GitHub experience, remote-work experience, communication, and accountability as important skills, while recommending practice challenges, project portfolios, and company research; 98% of respondents in Shopify’s latest intern satisfaction survey said they would recommend the program.


### [Fine-Tune ViT for Image Classification with 🤗 Transformers](https://yomu.fyi/post/fine-tune-vit-for-image-classification-with-transformers.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Nate Raw
- Published: Feb 11, 2022
- Type: Tutorial

Vision Transformer models bring transformer architectures to computer vision by splitting images into grids of sub-image patches and projecting them into token sequences. To classify healthy and diseased leaves using the beans dataset, the Hugging Face datasets and transformers libraries enable streamlined data ingestion and model fine-tuning. Preprocessing relies on ViTImageProcessor paired with lazy on-the-fly dataset transforms to dynamically generate normalized pixel tensors. Training leverages ViTForImageClassification with the Trainer API, requiring remove\_unused\_columns set to False so raw image data is preserved for batch collation. Over four training epochs, fine-tuning the google/vit-base-patch16-224-in21k checkpoint achieves an evaluation accuracy of 98.5% alongside an evaluation loss of 0.0637.


### [Getting Started with Sentiment Analysis using Python](https://yomu.fyi/post/getting-started-with-sentiment-analysis-using-python.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Federico Pascual
- Published: Feb 2, 2022
- Type: Tutorial

Modern natural language processing tools allow developers to perform sentiment analysis without deep machine learning expertise. Sentiment analysis categorizes text polarity into positive, negative, or neutral labels to extract insights from large volumes of social posts, reviews, and support tickets. Using the Hugging Face Transformers library and the Hub, practitioners can quickly run pre-trained transformer pipelines or target specific multilingual and emotion-detection models with minimal Python code. For custom requirements, developers can fine-tune models like DistilBERT using the IMDB dataset via the Trainer API or train models automatically with AutoNLP. Applying these workflows to social media data enables automated sentiment distribution analysis and visualization using pandas, matplotlib, and word clouds.


### [Making automatic speech recognition work on large files with Wav2Vec2 in 🤗 Transformers](https://yomu.fyi/post/making-automatic-speech-recognition-work-on-large-files-with-wav2vec2.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Nicolas Patry
- Published: Feb 1, 2022
- Type: Problem & solution

Transformer-based automatic speech recognition models like Wav2Vec2 crash on long audio inputs because the O(n²) attention mechanism rapidly exhausts GPU memory. Simple audio chunking avoids out-of-memory errors but causes poor transcription accuracy at chunk boundaries due to a lack of surrounding context. To resolve this, Hugging Face Transformers leverages the Connectionist Temporal Classification architecture to process overlapping audio chunks with configurable strides. The pipeline discards low-quality logits at chunk edges and chains the remaining central predictions together to reconstruct a seamless transcript. This striding technique operates out of the box with language model-augmented pipelines and adapts directly to low-latency live audio streaming.


### [Fixing the Dependency Confusion Vulnerability in 600+ Ruby Apps - Shopify](https://yomu.fyi/post/fixing-the-dependency-confusion-vulnerability-in-600-ruby-apps-shopify.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jan 27, 2022
- Type: Problem & solution

Shopify identified a dependency confusion vulnerability through its bug bounty program; an attacker could exploit ambiguous package sources to install a malicious dependency and gain access to local, CI/CD, and production environments. To assess exposure, the Ruby Conventions team collected Bundler versions from repositories in a data lake and found around 600 susceptible Ruby applications. An experimental Bundler plugin converted Gemfile.lock files to the explicit-source format while restoring existing gem versions, and a CI-backed CLI later provided per-repository environments, parallel execution, pull requests, and easier failure tracking. After incremental migrations exposed configuration and tooling issues, Shopify updated supporting systems, set Bundler requirements, changed local and CI defaults, and worked with Bundler maintainers; the company reports migrating all its applications in less than a day and contributing to Bundler 2.2.21 and 2.2.22.


### [Welcome Stable-baselines3 to the Hugging Face Hub 🤗](https://yomu.fyi/post/welcome-stable-baselines3-to-the-hugging-face-hub.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Thomas Simonini
- Published: Jan 21, 2022
- Type: Announcement

Hugging Face announced an official integration with Stable-Baselines3, a popular PyTorch library for training and testing Deep Reinforcement Learning agents across diverse environments like Gym, Atari, MuJoco, and Procgen. The integration enables researchers and developers to host their saved reinforcement learning checkpoints on the Hugging Face Hub and download pre-trained community models. Interacting with the Hub requires installing the huggingface\_hub and huggingface\_sb3 packages, which supply helper methods for authentication, downloading, and uploading. Practitioners can retrieve checkpoint zip files using the load\_from\_hub function by providing the target repository identifier and filename before loading them into Stable-Baselines3 algorithms. Furthermore, users authenticated via CLI or notebook login can train policies such as PPO and publish their saved zip files to the Hub using push\_to\_hub.


### [Supabase Beta December 2021](https://yomu.fyi/post/supabase-beta-december-2021.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: Paul Copplestone
- Published: Jan 20, 2022
- Type: Announcement

Supabase announced a series of platform updates, educational content, and community milestones for December 2021. PostgreSQL instances now include the pg\_sodium extension, enabling users to perform encryption, decryption, hashing, and cryptographic signing directly within database queries and functions through the Supabase Dashboard. The release details educational video guides demonstrating remote procedure calls from JavaScript, invoking external HTTP endpoints from database functions, and setting up triggers to run SQL on table modifications. In addition to ecosystem updates featuring new Python library releases and integrations with Divjoy and n8n, the project surpassed 26,000 GitHub stars and opened fully remote job positions across engineering, marketing, and human resources.


### [Boosting Wav2Vec2 with n-grams in 🤗 Transformers](https://yomu.fyi/post/boosting-wav2vec2-with-n-grams-in-transformers.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Patrick von Platen
- Published: Jan 12, 2022
- Type: Tutorial

Wav2Vec2 models fine-tuned with Connectionist Temporal Classification transcribe speech without external language models, but decoding can still suffer from spelling inaccuracies. Hugging Face Transformers addressed this by integrating Kensho Technologies' pyctcdecode library to support decoding with n-gram language models. Instead of decoding using simple argmax operations over logits, the Wav2Vec2ProcessorWithLM class feeds full probability matrices into beam search guided by KenLM n-gram probabilities. KenLM's build\_binary utility compresses language model ARPA files into binary formats, reducing file size by more than half for faster loading and hub deployment. In Swedish xls-r-300m-sv benchmarks on Common Voice 7, this 5-gram boosted decoding setup achieved an 18.85% word error rate, delivering an approximate 30% relative performance gain.


### [Gradio is joining Hugging Face!](https://yomu.fyi/post/gradio-is-joining-hugging-face.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Abubakar Abid
- Published: Dec 21, 2021
- Type: Announcement

Hugging Face has acquired Gradio, the open-source machine learning library designed for building and sharing interactive model interfaces. Gradio originated in 2019 when its founder struggled to share a medical computer vision model with a physician collaborator who did not write Python. Co-founded alongside Ali Abdalla, Ali Abid, and Dawood Khan, the project expanded from computer vision into text, speech, and video modalities. Over 300,000 demos have been built using Gradio, allowing interdisciplinary industry teams and researchers to debug models internally and showcase them externally to non-technical users. The acquisition unites Gradio with Hugging Face to broaden browser-based machine learning accessibility and expand hiring efforts across the joint team.


### [Holiday Hackdays Winners 2021](https://yomu.fyi/post/holiday-hackdays-winners-2021.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: Thor Schaeff
- Published: Dec 17, 2021
- Type: Announcement

Supabase organized the Holiday Hackdays 2021 hackathon following their launch week and announced the winning and runner-up community submissions. Selected winners include Swappy.one by Zernonia, the realtime polling platform rtPoll by Emilio and Federico Schepis, Santa Banter for holiday jokes by Andy Keogh, and the wishlist app the get list by glowdexapp. Recognized runners-up include Chivel for YouTube channel landing pages, the end-to-end encrypted e2ee-chat, a Flutter-based Holiday Sweater voting app, and a greeting card platform built by high school hackers. Winners receive limited-edition gold medal shirts, while runners-up earn silver medal shirts. All project submissions were made available for viewing on madewithsupabase.com.


### [Supabase Beta November 2021: Launch Week Recap](https://yomu.fyi/post/supabase-beta-november-2021-launch-week-recap.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: Ant Wilson
- Published: Dec 15, 2021
- Type: Announcement

Supabase celebrated its third Launch Week and one year in beta by rolling out several major infrastructure and tooling updates. The company open-sourced its Dashboard for self-hosting and integrated Logflare to provide searchable database and API logs directly in the interface. For database querying and APIs, Supabase introduced a Postgres extension that resolves each GraphQL request with a single SQL statement to minimize network IO overhead, while upgrading its default hosted database to PostgreSQL 14 and deploying PostgREST 9.0. Security capabilities were expanded by enabling Row Level Security policies across the Realtime API to restrict streams and subscriptions on a per-user basis. Additional releases during the week included faster media file delivery, an open-sourced launch methodology, and a free course combining Next.js, Stripe, and Supabase.


### [Perceiver IO: a scalable, fully-attentional model that works on any modality](https://yomu.fyi/post/perceiver-io-a-scalable-fully-attentional-model-that-works-on-any-moda.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Niels Rogge
- Published: Dec 15, 2021
- Type: Announcement

Standard Transformer architectures scale poorly in compute and memory because pairwise dot-product self-attention depends quadratically on input size. Perceiver IO addresses this constraint by computing self-attention across a small set of latent variables rather than directly on high-dimensional inputs. Inputs and outputs interact with the model via cross-attention operations, decoupling compute and memory costs from input and output dimensions. Integrated into Hugging Face Transformers via the PerceiverModel class, the architecture supports diverse data types using optional preprocessors, decoders, and postprocessors. Experiments demonstrate competitive performance across text, multimodal video classification, 3D point cloud classification on ModelNet40, and StarCraft II reinforcement learning in AlphaStar.


### [Training CodeParrot 🦜 from Scratch](https://yomu.fyi/post/training-codeparrot-from-scratch.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Leandro von Werra
- Published: Dec 8, 2021
- Type: Tutorial

Training code generation models from scratch requires careful data filtering, tokenization, and compute management. To build CodeParrot, a 1.5-billion-parameter Python generation model, engineers extracted 20 million Python files from BigQuery and cleaned the resulting 180 GB corpus down to 50 GB after discovering extreme duplication. A custom GPT-2 tokenizer was trained over streamed samples, and the GPT-2 large architecture was initialized with layer-scaled, full-precision attention mechanisms. The training pipeline leveraged Hugging Face Accelerate alongside gradient checkpointing and a streaming iterable dataset designed to yield fixed-length concatenated token sequences. CodeParrot was trained on roughly 25 to 30 billion tokens and evaluated on coding tasks using the pass@k metric.


### [Five more things](https://yomu.fyi/post/five-more-things.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: Ant Wilson
- Published: Dec 3, 2021
- Type: Announcement

Supabase announced a series of platform updates, technical releases, and community initiatives to conclude its Launch Week. The team open-sourced pg\_graphql, an in-progress PostgreSQL extension that keeps schema generation, query parsing, and resolvers natively on the database server. Supabase also enabled a global CDN spanning over 250 cities, delivering edge caching with a default one-hour cache time for storage assets alongside TLS 1.3, HTTP/3 over QUIC, and Brotli compression. While Supabase Functions is not yet released, two public RFCs were opened to explore architecture options, leaning heavily toward container-based execution. Finally, the update introduced a free SaaS development course with egghead.io and launched the ten-day Holiday Hackdays hackathon.


[Newer posts](https://yomu.fyi/topic/open-source/page/23.md) · [Older posts](https://yomu.fyi/topic/open-source/page/25.md)