Loading…
Topic
Open Source
567 posts about Open Source. Every summary links to the original.
- 567 posts
- 36 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Realtime: Multiplayer Edition
Supabase announces the general availability of Realtime’s multiplayer features, Broadcast and Presence, alongside a new channel-based architecture and supabase-js support. Realtime originally used Postgres logical replication, rather than LISTEN/NOTIFY, to avoid the latter’s 8,000-byte payload limit and the latency of querying each change from the database. In the new design, Channels act as rooms that support ephemeral publisher/subscriber Broadcast messages, shared Presence state, and Postgres change events; Presence uses Phoenix Tracker, a delta-based CRDT, for eventually consistent synchronization and removes disconnected users automatically. Realtime now runs as a distributed Elixir and Phoenix cluster on Fly, with clients connecting to the nearest node and nodes communicating across a backbone network to improve connection speed, stability, and latency. Free-plan active projects and projects created from August 18 have access, while other projects are scheduled for migration over the following weeks.
Instant Performance Upgrade: From FlatList to FlashList
Shopify’s React Native Foundations team created FlashList as a faster, easier alternative to React Native’s FlatList, targeting frame drops, blank cells, and high memory use across a wide range of devices. FlatList creates new views during updates, while FlashList recycles already-allocated cells and moves some layout operations to native, using a 250-pixel default buffer to reduce memory footprint and load times. Its API is nearly the same as FlatList, so an existing list can be swapped by changing the component name and optionally adding estimatedItemSize, with features such as type-specific recycling, flexible column spans, and FPS and blank-cell metrics. Extensive Android testing found performance of 60 FPS or greater even on low-end devices, and Shopify Mobile uses FlashList as its default while Shop moved search to it.
Supabase Series B
Supabase announced an $80 million Series B in May, bringing total funding to $116 million, and reported growth to more than 150,000 hosted PostgreSQL databases and 110,000 developers. Community metrics also increased, with Discord membership rising from 4,000 to 8,000, Twitter followers from 18,000 to 37,000, and GitHub stars from 19,000 to 36,000. The round is led by Felicis, with Coatue, Lightspeed, and Square Peg Capital joining, while Supabase describes its open-source support through contributor payments, maintainer hiring, sponsorship, and community collaboration. The company is moving into its third phase, focused on cloud-native PostgreSQL capabilities including branching, scalable storage, distribution, ephemeral compute, snapshots, and time travel. It invested $500,000 in OrioleDB and hired a developer to work on it, while stating that OrioleDB is not currently used on the Supabase platform and that the company will not run a PostgreSQL fork.
Launch Week 5 Hackathon
Supabase’s Launch Week 5 Hackathon invites individuals and teams of up to five to build a new open-source project using Supabase during a 10-day event. It starts at 8:00am PT on Friday 12 August 2022, with submissions due by 11:59pm PT on Sunday 21 August; projects may use any language or framework and must be created during the event. The best overall project receives $1,500 in GitHub sponsorship paid as $500 per month for three months, while the listed category prizes include Gold Supabase Caps, with winner and runner-up prizes and caps for every winning team member. Supabase-js v2.0 and Flutter SDK v1.0 are announced during Launch Week, and using them earns extra credit; the supabase-js release candidate is installable with npm install @supabase/supabase-js@rc.
Supabase Beta July 2022
Supabase published its July 2022 beta update alongside the announcement of Launch Week 5 scheduled for August 15-19. Key feature releases include a developer preview of Flutter SDK 1.0 with an emphasis on developer experience and Auth Helpers supporting server-side rendering environments like SvelteKit. The team introduced pg_jsonschema, an experimental 10-line Postgres extension created with pgx to enforce structure on json and jsonb columns. Security and identity features were expanded with an hCaptcha integration in authentication settings to block bot attacks. Additionally, email one-time passwords now accommodate lengths between 6 and 10 digits, while the GenerateLink method returns explicit verification metadata.
Supabase Flutter SDK 1.0 Developer Preview
Supabase released the Developer Preview of version 1.0 for its Flutter SDK, prioritizing improvements to developer experience across Dart applications. The new version eliminates the need for boilerplate state classes, automating authentication state persistence and deep link processing directly after initialization. To better align with Dart conventions, the SDK now throws exceptions across auth, storage, and Postgrest instead of returning error objects, while also deprecating the .execute() method for database queries. Switching to the app_links library expands deep linking support to macOS and Windows alongside existing mobile and web platforms. Supabase also introduced a companion Auth UI library for prebuilt authentication interfaces and announced plans to integrate the Multiplayer Realtime engine in the stable release.
Revamped Auth Helpers for Supabase (with SvelteKit support)
Supabase has released an updated version of its framework-specific Auth Helpers libraries, introducing official support for SvelteKit alongside existing React and Next.js tooling to simplify server-side rendering authentication. The updated libraries transition from the @supabase/supabase-auth-helpers namespace to @supabase/auth-helpers within a dedicated monorepo managed by Turborepo. Structuring the codebase with Turborepo and changesets enables publishing isolated packages per framework through GitHub Actions, ensuring each helper contains only its own relevant dependencies. Versioning has also been reset to sub-0.x to reflect early framework lifecycles like pre-1.0 SvelteKit and manage expectations for future breaking changes. Meanwhile, the team has documented migration steps from previous packages and announced that Remix Auth Helpers are currently in active development.
Supabase Beta June 2022
Supabase released its Beta June 2022 updates covering developer tooling, authentication, and infrastructure changes. The team decoupled Supabase Auth Helpers into dedicated packages for Next.js and React, while introducing project pause and restore features to enable unlimited free projects. Developers can now initiate project creation directly through the Supabase CLI and toggle realtime updates from the table editor side panel. Authentication updates include exposing client IP addresses in audit logs, introducing an auth.jwt function, bubbling PostgreSQL errors from GoTrue, and deprecating older authentication functions. Additionally, infrastructure migrations are underway to support multiplayer features, accompanied by community-led tutorials, sponsorships, and platform growth reaching fifty thousand GitHub stars.
Shopify and Open Source: A Mutually Beneficial Relationship
Shopify describes its relationship with Ruby, Rails, and the open-source community as mutually beneficial, linking the company’s success to its Rails foundation and its engineering culture to the Rails Doctrine. It frames a 100-year company ambition around making Ruby and Rails viable for large teams, mature codebases, millions of lines of code, and billions of web requests through production performance, architecture guidance, upgrades, tooling, and selective typing. To support that goal, Shopify contributes projects and initiatives spanning developer tooling, gradual typing, VS Code support, automated upgrades, YJIT, Ruby performance work, TruffleRuby collaboration, $500,000 in research funding, Bundler and RubyGems, and RubyGems.org security. The post argues that company participation improves engineering skills, application longevity, and influence over foundational software, while describing contributors who range from occasional participants to full-time maintainers.
Visualizing Supabase Data using Metabase
Organizations seeking to analyze database records can bypass code-heavy Python visualization libraries by connecting a Supabase backend directly to Metabase. Deploying the open-source Metabase Docker container exposes an initial setup interface on default port 3000, allowing administrators to input PostgreSQL connection credentials retrieved from Supabase database settings. Once connected, Metabase automatically surfaces table insights and automated x-rays across the public schema, exposing metrics such as inventory count ranges, price distributions, and column-specific statistics without requiring manual queries. For specialized analytics, users can execute custom SQL joins across relational tables like Product and Vendor to render tailored visual dashboard formats, including configurable bar charts mapped to designated axis fields.
Liftoff! How to get started with your first ML project 🚀
Beginners in machine learning frequently face difficulties when selecting a framework and defining the scope of their initial hands-on project. Sentence Transformers provides an accessible starting point by computing dense vector representations for sentences, paragraphs, and images, enabling semantic search through cosine similarity. A structured four-step strategy guides practitioners through listing library capabilities, identifying compelling datasets, selecting a familiar secondary tool, and brainstorming project concepts. Applying this framework, a song lyrics dataset was paired with Gradio Blocks to build a prompt-based playlist generator. Developing this application required evaluating pre-trained models, hosting generated embeddings on Hugging Face Spaces, and utilizing multi-processor support to accelerate embedding generation.
Accelerate Large Model Training using DeepSpeed
Training large models on hardware with limited GPU memory frequently causes out-of-memory errors when using standard Distributed Data Parallel. To solve this bottleneck, Hugging Face Accelerate integrates DeepSpeed ZeRO data parallelism to shard optimizer states, gradients, and model parameters across workers. For a 900-million-parameter DeBERTa model, ZeRO Stage 2 increased the maximum per-device batch size from eight to forty while achieving a 3.5-fold training speedup over DDP without degrading accuracy or F1 score. Advanced configurations allow sequence-to-sequence chatbot finetuning and full ZeRO Stage 3 CPU offloading for models like the 1.5-billion-parameter GPT-XL, enabling batch size 16 training where DDP fails. Accelerate enables these memory optimizations through simple configuration files and the accelerate launch command with minimal or zero code modifications.
Announcing Evaluation on the Hub
Hugging Face introduced Evaluation on the Hub, a no-code tool designed to streamline the evaluation of machine learning models across diverse datasets. Traditional evaluation approaches often suffer from reproducibility issues, implementation inconsistencies, and cumbersome workflows across varied metrics and datasets. Powered by AutoTrain and Hugging Face Spaces, the service allows users to configure tasks, map dataset columns, select evaluation metrics, and run tests directly from dataset pages. Completed evaluations automatically open pull requests on the respective model cards to encode standardized verification metadata and update public dataset leaderboards. This system establishes consistent benchmarking pipelines across tasks like image classification, text summarization, and named entity recognition without requiring local code execution.
Getting Started With Embeddings
Embeddings represent unstructured information such as text and images as numerical vectors in a shared semantic space. To demonstrate their utility, a simple semantic search engine is built over US Social Security Medicare frequently asked questions. The system generates 384-dimensional vector representations for thirteen FAQ entries by dispatching POST requests to the Hugging Face Inference API using the sentence-transformers/all-MiniLM-L6-v2 model. Incoming user queries are converted into matching vector representations and evaluated against stored dataset vectors using the util.semantic_search function from the Sentence Transformers library. By calculating cosine similarity scores, the system retrieves and ranks the five most semantically relevant questions without requiring custom keyword rules or massive labeled training sets.
How We Built Hydrogen: A React Framework for Building Custom Storefronts
Hydrogen is a React framework for building custom Shopify storefronts, created to help merchants use the Storefront API without giving up capabilities associated with Liquid storefronts. Shopify built it around React Server Components and React 18’s streaming server-side rendering, using component-level data fetching, a preload cache to parallelize nested queries, and experimental loggers for query waterfalls and unused GraphQL properties. The framework evolved through more than 1,200 pull requests and 3,000 commits, with rigid components removed after feedback and nearly all exported GraphQL fragments eliminated to make queries more discoverable. The post reports that some demo store routes cut load time in half and that Hydrogen is used in production at Allbirds, Shopify Supply, Shopify Hardware, and Shopify, while future plans include a new router and incremental adoption from Liquid storefronts.
Python data loading with Supabase
Supabase offers a PostgreSQL-based backend-as-a-service platform alongside an open-source Python SDK for automating CRUD operations and data-intensive tasks. After spinning up a new project in the Supabase dashboard, developers can configure relational schemas in the Table Editor by establishing foreign key constraints between tables like Vendor and Product. Authentication for the SDK relies on passing project URL parameters and API keys, which can be securely stored and loaded through environment variables. Programmatic data population utilizes the supabase client library combined with the Faker package to generate realistic synthetic entries and push them directly to database tables using insert and execute calls. Populated records can subsequently be examined in real time through the dashboard Table Editor interface to verify schema integrity and data persistence.
Making Open Source Safer for Everyone with Shopify’s Bug Bounty Program
Shopify describes how a bug bounty report led to application fixes and Rails contributions aimed at making open-source software safer. The 2021 Shopify Experiments report showed that a leaked development-environment secret could be used to generate trusted-looking data and reach remote code execution in a local development environment. The root cause was ActiveSupport’s MessageEncryptor using Ruby’s unsafe Marshal serializer by default in Rails versions up to 7.0.X, allowing untrusted data to trigger deserialization risks. Shopify first configured JSON in its application, then helped change the default serializers for MessageEncryptor and MessageVerifier to JSON in Rails 7.1.X, with an upgrade path for existing applications. It also developed a patch that monitors Marshal load and dump calls, raises on unexpected gem-originated calls, and runs in local development and continuous integration to prevent new usage.
Intel and Hugging Face Partner to Democratize Machine Learning Hardware Acceleration
Intel has joined Hugging Face's Hardware Partner Program to accelerate Transformer training, fine-tuning, and inference on Intel platforms. Large Transformer models introduce latency bottlenecks in production workloads like search and chatbots, where hardware-level optimization typically requires tedious trial and error. To streamline model optimization, the collaboration introduces Optimum Intel, an open-source library integrating the Intel Neural Compressor for automated quantization, pruning, and distillation. A demonstration applies post-training dynamic quantization to a fine-tuned DistilBERT classification model using a CPU-only PyTorch setup. The quantized model converted 38 Linear and 2 Embedding operators to 8-bit integers, reducing evaluation duration by 1.34x while keeping the accuracy drop within a 5% threshold.
8 Data Conferences Shopify Data Thinks You Should Attend
Shopify’s data scientists and engineers curated eight upcoming data conferences for 2022, covering virtual, hybrid, and in-person formats. The selection supports learning and development by helping practitioners hear from peers about applications, techniques, use cases, and opportunities in the wider data community. Hybrid options include Data + AI Summit, Transform, RecSys, and ODSC West, with topics spanning analytics, machine learning, open source technologies, recommender systems, MLOps, natural language processing, and big data. In-person choices—PyData London, KDD, Re-Work Deep Learning Summit, and Crunch—feature tutorials, research, business applications, AI challenges, data-team practices, and machine learning at scale, with locations including London, Washington D.C., Toronto, and Budapest.
Building a Form with Polaris
Shopify’s Polaris design system provides design and content guidelines and reusable React components for Shopify admin and third-party apps. The tutorial builds a basic product form with Title and Description fields, a Save button, and top navigation, while introducing React state, events, and handlers. It uses Form, FormLayout, TextField, Page, and Card, then wraps the application in the required AppProvider after MissingAppProviderError appears; AppProvider shares global settings, including translations, across child components. The form is configured as controlled, with separate useState values and onChange handlers for each field, an onSubmit path connected to the Page component’s onAction prop, and clearing logic after submission.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.