Loading…
Topic
812 posts about Performance. Every summary links to the original.
See the idea
Bare metal gives engineers direct control over GPU, network, and virtualization layers, but it shifts elasticity and lifecycle work onto the team. This trade-off map shows when that bargain pays off.
Bun packs a runtime, package manager, bundler, and test runner into one binary. Here is how those four parts fit together and where the seams still show.
A five-step how-to for deciding when an AMD Instinct GPU should stay dedicated and when SR-IOV virtual functions, VMs, or sidecar isolation can safely share it.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Training large models on hardware with limited GPU memory frequently causes out-of-memory errors when using standard Distributed Data Parallel. To solve this bottleneck, Hugging Face Accelerate integrates DeepSpeed ZeRO data parallelism to shard optimizer states, gradients, and model parameters across workers. For a 900-million-parameter DeBERTa model, ZeRO Stage 2 increased the maximum per-device batch size from eight to forty while achieving a 3.5-fold training speedup over DDP without degrading accuracy or F1 score. Advanced configurations allow sequence-to-sequence chatbot finetuning and full ZeRO Stage 3 CPU offloading for models like the 1.5-billion-parameter GPT-XL, enabling batch size 16 training where DDP fails. Accelerate enables these memory optimizations through simple configuration files and the accelerate launch command with minimal or zero code modifications.
Hydrogen is a React framework for building custom Shopify storefronts, created to help merchants use the Storefront API without giving up capabilities associated with Liquid storefronts. Shopify built it around React Server Components and React 18’s streaming server-side rendering, using component-level data fetching, a preload cache to parallelize nested queries, and experimental loggers for query waterfalls and unused GraphQL properties. The framework evolved through more than 1,200 pull requests and 3,000 commits, with rigid components removed after feedback and nearly all exported GraphQL fragments eliminated to make queries more discoverable. The post reports that some demo store routes cut load time in half and that Hydrogen is used in production at Allbirds, Shopify Supply, Shopify Hardware, and Shopify, while future plans include a new router and incremental adoption from Liquid storefronts.
Intel has joined Hugging Face's Hardware Partner Program to accelerate Transformer training, fine-tuning, and inference on Intel platforms. Large Transformer models introduce latency bottlenecks in production workloads like search and chatbots, where hardware-level optimization typically requires tedious trial and error. To streamline model optimization, the collaboration introduces Optimum Intel, an open-source library integrating the Intel Neural Compressor for automated quantization, pruning, and distillation. A demonstration applies post-training dynamic quantization to a fine-tuned DistilBERT classification model using a CPU-only PyTorch setup. The quantized model converted 38 Linear and 2 Embedding operators to 8-bit integers, reducing evaluation duration by 1.34x while keeping the accuracy drop within a 5% threshold.
Shopify Party is a browser-based internal tool designed to make virtual hangouts more social and reduce video fatigue after Shopify moved to remote work. Its 3D spaces favor non-prescriptive play, letting participants explore and manipulate objects, join or leave at any point, and use a small control set rather than follow prescribed rules or win conditions. The initial version was built in three months with Unity 2020 and the Universal Render Pipeline, exported to WebGL, while Normcore synchronizes game state, provides VoIP and networked physics, and uses room time to keep autonomous objects aligned across clients. Performance work included third-party assets, GPU instancing for repeated models, simpler materials, and adjustable render resolution. The launch exceeded expectations, with hundreds of games played each week, and the tool is positioned as an option alongside video conferencing rather than a replacement.
Shopify’s development environment, the Ruby-based dev tool and xhyve-backed railgun VM, no longer handled the growing number of repositories, services, integrations, and resource demands around its monoliths. Developers needed more services, faced sluggish code-build-test loops, and encountered costly state changes when tophatting collaborators’ work. The Environments team concluded that development environments needed cloud-like elasticity, then tested an automatically configured local Kubernetes cluster and easy creation of Google Compute Engine VMs; the source also describes Isospin and Spin as the resulting direction. Local Kubernetes did not gain broad organic traction, while the current Isospin implementation is described as the team’s best iteration for developers working on majestic monoliths. The team plans to evolve it into an application-development library built on dev, add dependency inference, and address the tension between persistent environments and Kubernetes pod relocation.
Supabase announced several platform updates and performance improvements across its database, hosting, and developer tooling ecosystem for May 2022. Developers managing preview deployments on Jamstack platforms like Vercel and Netlify can now configure wildcard redirect domains such as *.mydomain.com/welcome. For serverless workflows, Edge Functions support a -no-verify-jwt CLI deployment flag to facilitate direct invocation via external webhooks without mandatory token validation. In addition, Supabase exposed a Prometheus-compatible metrics endpoint to all users for real-time project monitoring and alerting. Database security policies also gained a dedicated Target roles configuration, replacing older role check expressions and cutting query execution time from 46 seconds to 7 milliseconds in one customer deployment.
Threaded servers such as Puma and thread-based Sidekiq are popular Ruby deployment choices because threads can increase throughput without a proportional increase in memory. The post compares process and thread models using a memory model that separates static application memory from per-request processing memory, then refines it with Copy on Write (CoW), where forked processes share pages until writes force copies. It explains why RSS can double-count shared pages, recommends PSS and Linux’s /proc/$PID/smaps_rollup for more meaningful measurements, and gives a production example in which about two-thirds of static memory is shared. The conclusion is metric-dependent: preload applications to improve CoW, prefer fewer larger containers when appropriate, consider Unicorn for reliable request timeouts, and start web workloads at two threads unless substantial I/O wait justifies more; job processors may benefit from higher counts.
Graphcore and Hugging Face expanded the range of machine learning modalities and tasks available in Hugging Face Optimum. Developers can now access ten transformer models optimized for Graphcore IPUs across natural language processing, speech, and computer vision. The available architectures include BERT, ViT, GPT-2, RoBERTa, DeBERTa, BART, LXMERT, T5, HuBERT, and Wav2Vec2, complete with IPU configuration files and ready-to-use pre-trained or fine-tuned weights. The integration supports the Bow IPU processor, which uses 3D Wafer-on-Wafer stacking to achieve up to 350 teraFLOPS of AI compute. Optimum also integrates with the Poplar SDK 2.5, enabling compatibility with frameworks such as PyTorch, TensorFlow, Docker, and Kubernetes.
Shopify describes operating Apache Airflow 2.2 on Kubernetes with the Celery executor and MySQL 8 for data extraction, machine-learning training, Iceberg maintenance, and DBT modeling. In its largest environment, Shopify runs over 10,000 DAGs, averages over 400 tasks running at a given moment, and executes over 150,000 runs per day, while facing slow file access, metadata growth, ownership tracking, DAG-author control, traffic bursts, and resource contention. The authors describe replacing per-pod GCSFuse access with an in-cluster NFS volume synchronized to GCS, retaining GCS as the user-facing source of truth; they also use 28-day metadata retention, a YAML manifest, DAG policies, and scheduling controls including pools, priority weights, and Celery queues. These measures are presented as ways to improve file performance, limit database growth, identify owners, enforce constraints, and reduce workload interference, though the retention policy does not support durable history-dependent features such as long-running backfills.
Shopify says it has invested nearly half a million dollars over the past year in academic research directed toward Ruby and the needs of its community. The work is intended to make programming-language developments account for Ruby’s metaprogramming, loose C extension API, and large-scale workloads with frequent redeployments, rather than assuming long virtual-machine warmup periods. Collaborators include Laurence Tratt, who is investigating hardware meta-tracing and basic-block stitching to generate a just-in-time compiler from the Ruby interpreter; Steve Blackburn, whose group is putting the MMTk garbage-collection toolkit into Ruby; and Stefan Marr, who is studying interpreter speed and startup and warmup time. The program also connects these researchers with Shopify’s Ruby engineers and complements internal work on YJIT and TruffleRuby, with more research and academic papers planned.
Gradio 3.0 introduces a ground-up redesign of the Gradio library to improve machine learning demo development and browser performance. The frontend has been rebuilt using modern technologies such as Svelte, resulting in smaller payloads, faster page load times, and cleaner visual designs suitable for embedding in blog posts. Component updates include revamping the Dataframe component to support CSV drag-and-drop, introducing new UI elements like Gallery, and adding the TabbedInterface class to group demos under distinct tabs. Developers also gain access to Gradio Blocks, a low-level Python API enabling custom layouts, multi-step interfaces, dynamic component property modifications, and complex data flows. A competition called the Gradio Blocks Party has been organized through the end of May to encourage community adoption.
Transformer-based models are moving into production for large-scale workloads, but default implementations remain slow and resource-intensive compared to traditional machine learning algorithms. Hugging Face Optimum addresses this issue by integrating performance optimization tools and accelerated runtimes like ONNX Runtime directly into standard Transformers pipelines. Developers replace standard model classes with Optimum equivalents to convert models to ONNX, run graph optimizations such as operator fusion, and apply quantization. In an evaluation on an AWS EC2 m5.xlarge CPU instance, optimizing and quantizing a RoBERTa question-answering model cut average latency from 117.61 ms to 64.94 ms while retaining 99.61 percent accuracy. Known limitations include a 2GB remote model size limit, lack of seq2seq support, omitted past key values in causal language models, and no local caching for optimized ONNX models.
Increasing parameter counts in modern machine learning models make loading and training them prohibitive on standard hardware. While Distributed Data Parallel replicates the entire model across GPUs, PyTorch Fully Sharded Data Parallel shards optimizer states, gradients, and parameters across data-parallel workers. Benchmarks on causal language modeling using GPT-2 Large and XL across Titan RTX GPUs demonstrate that FSDP increases allowable batch sizes and prevents out-of-memory errors encountered in standard DDP. Integrating FSDP through Hugging Face Accelerate enables these capabilities alongside CPU offloading to run models exceeding GPU memory limits. However, Accelerate requires preparing models before creating optimizers when using multiple models and notes that mixed precision support with FSDP currently has transformer compatibility issues.
Habana Labs and Hugging Face partnered to accelerate Transformer model training using Habana Gaudi accelerators, which offer up to 40% better price performance than recent GPU-based Amazon EC2 instances. Setting up the environment requires launching an Amazon EC2 dl1.24xlarge instance using the Habana Deep Learning Base AMI on Ubuntu 20.04. Inside an interactive PyTorch Docker container configured for Habana runtimes, users install the Optimum Habana package from source along with necessary dependencies. Fine-tuning the bert-large-uncased-whole-word-masking model on the GLUE MRPC task completes in 2 minutes and 12 seconds with an F1 score of 0.9181. Utilizing EC2 Spot Instances reduces the hourly compute cost by 70%, from $13.11 down to $3.93.
Ruby's string model supports mutable and immutable operations, per-object or per-file freezing, and more than 100 encodings, while permitting byte sequences invalid for their attached encoding. This flexibility helps legacy and unusual platforms but makes encoding checks, validity scans, and character-boundary handling part of many operations. To reduce repeated O(n) validation scans, Ruby implementations cache each string's state in a four-valued code range: unknown, 7bit, valid, or broken. The post explains how these values map to public methods, how MRI stores them in two header bits, and how JRuby and TruffleRuby represent them differently. It also describes code ranges as inputs to fast and slow paths and as useful but fallible cache information, especially for native extensions and string-heavy metaprogramming.
In this interview with Britney Muller, Hugging Face machine learning engineer Lewis Tunstall discusses his background, his book Natural Language Processing with Transformers, and tooling optimizations for deploying transformer models. Tunstall notes that large model parameter sizes often lead to high latency, making off-the-shelf transformers impractical for responsive applications like chatbots. To resolve this issue, he developed single-line export tooling within the transformers library to convert models into the ONNX format. This conversion enables engineers to run PyTorch models in TensorFlow or on specialized hardware, achieving faster latency and higher throughput. Tunstall also recounts co-authoring the O'Reilly book with Leandro von Werra and Thom Wolf, collaborating on the Hugging Face Course, and training a GPT-2 model named code parrot on Python code.
Training transformer models across computer vision, speech, and natural language processing tasks at scale often demands heavy compute resources, incurring significant time and financial expense. To address these bottlenecks, Habana Labs and Hugging Face partnered to integrate the SynapseAI software suite into the Hugging Face Optimum open-source library. This integration allows machine learning practitioners to accelerate transformer training workflows on Habana Gaudi processors using minimal code adjustments. Habana Gaudi hardware, featured in Amazon EC2 DL1 instances and Supermicro X12 servers, incorporates ten 100 Gigabit Ethernet ports per processor to scale from single units to thousands of chips. The combined hardware and software architecture supports TensorFlow and PyTorch while delivering price and performance metrics up to 40% lower than comparable training alternatives.
Grab developed a batch job service within its Trident automation engine to execute multi-step marketing campaigns for millions of users simultaneously. The system replaces sequential, single-server execution with a distributed architecture powered by Apache Kafka, which distributes batches of 100 users across server clusters using hashed partition keys. To reduce network overhead and queries per second, downstream reward and messaging services introduced batch endpoints backed by bulk database queries, decreasing API latency by up to 85%. Grab further optimized performance by sharding Kafka topics by country and action type to prevent long-running reward tasks from blocking time-sensitive messaging workloads. Additionally, making terminal messaging calls asynchronous allows subsequent batch processing to proceed without waiting for message delivery confirmations.
Hydrogen, Shopify’s framework for custom React storefronts, chooses Vite to improve developer experience while targeting performance for end users and evergreen browsers. Traditional bundling with tools such as Webpack can become complex and slow during development because each change may require processing more project files; Vite instead serves user code through native ES Modules, while pre-bundling and caching third-party dependencies. Vite combines an HMR development server with sensible defaults, ESBuild compilation, Rollup’s plugin system, CommonJS-to-ESM conversion, and SSR support, including transformations that differ between browser and server environments. Hydrogen integrates Vite through four plugins covering Hydrogen configuration, React Server Components, SSR and RSC middleware, and React fast refresh, enabling server components, streaming requests, and caching; the text concludes that Vite is a suitable choice, while acknowledging Webpack remains appropriate for some advanced use cases.
Supabase announced the general availability of GraphQL support on its platform via pg_graphql, an open-source PostgreSQL extension that reflects SQL schemas into GraphQL types, fields, and relationships. Accessible through SQL functions or HTTP endpoints, the extension enforces PostgreSQL row-level security and filters introspection schemas according to the SQL roles assigned to request JWTs. To optimize performance, pg_graphql transpiles every GraphQL query into a single SQL query that aggregates results into JSON, avoiding the N+1 query problem and eliminating redundant join data payloads. Transpiled queries are cached as prepared statements to bypass planning overhead, and all generated mutations and queries support bulk operations. On Supabase Free Plan hardware, these optimizations achieve mean throughputs of 377.4 requests per second over the API and 656.2 queries per second over SQL connections.
The best engineering reads, once a week.
