Loading…
Browse by type
Benchmark reads
Benchmark posts on Yomu, summarised newest first.
- 167 posts
- 20 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Benchmark posts
1.5x faster MoE training with custom MXFP8 kernels
An upgrade from Hopper H100s to Blackwell B200s exposed the Mixture-of-Experts (MoE) layer, implemented with MegaBlocks, as a major training bottleneck: it consumed nearly 53% of forward-pass time and 27% of backward-pass time. The team rewrote the layer at the GPU-kernel level using CUDA, PTX, and some ThunderKittens, while moving from BF16 to microscaled FP8 (MXFP8) with nearly zero loss in training quality. The implementation addresses Blackwell-specific tensor memory constraints and quantization overhead, including custom MXFP8 quantization that sustains more than 6.2 TB/s and emits scale matrices directly compatible with tcgen05.mma. On an internal model, MoE execution became 3.5x faster in both passes, producing a 1.5x end-to-end speedup on Blackwell and 2x versus the original Hopper setup, measured in tokens per second per GPU.
How we reduced initialisation time of Product Configuration Management SDK
GrabX, Grab’s central product configuration management platform, provides an SDK whose eventually consistent updates take about a minute to reach clients, but initialisation took roughly four minutes for services loading around 400 MB. The SDK’s sequential workflow downloaded subscribed-service and common data from AWS S3 to disk and memory before initialising dependent modules, creating cold-start pressure and possible throttling when new instances were launched. Phase 1 made the two data loads concurrent, reducing initialisation time by approximately 80%, while Phase 2 concurrently downloaded and loaded large configurations within a subscribed service, adding a further approximate 6% reduction. Phase 3 removed disk persistence because its stale-data fallback conflicted with most clients’ need for current data, allowing failed connections to trigger client action such as retrying. Benchmarks across services with varied data sizes indicated reductions of up to 90%.
Intercom vs Zendesk: Two AI agents put to the test
The post compares Intercom’s Fin AI Agent with Zendesk’s AI agent, focusing on answer quality, complexity, and resolution performance. In a test using 48 fictional bed-and-breakfast help articles and 200 questions, both systems answered the same dataset, while four AI models judged responses and an Elo rating system selected winners. It reports that Fin produced better answers in 80% of cases, answered 96% of multi-source questions versus Zendesk’s 78%, and delivered responses averaging 120 words versus 50. The analysis also found no statistical difference in hallucination levels and estimated that Fin was 66% more likely to provide a complete resolution when both agents answered. The post notes limitations including simulated data, AI judging, and differing product features, then describes Fin’s integrations and 99¢-per-resolution pricing.
LLM-assisted vector similarity search
Vector similarity search can struggle with queries containing negation, constraints, or conceptual requirements, even though it efficiently retrieves semantically similar records. The proposed method uses FAISS and OpenAI text-embedding-ada-002 embeddings to shortlist 15 candidates, then asks GPT-4o to rank the candidates against the original query and return the top three. On two GPT-4o-curated synthetic datasets—100 food dishes and 100 Asian tourist spots—the assisted method matched raw search on straightforward queries but produced more relevant results for complex ones, including excluding fish or shrimp and identifying wildlife-related places. The authors note limits from the small datasets and query set, report similar relevance improvements in an internal deployment involving more than 4,500 relational-database rows, and identify shortlist size and added LLM latency as areas for further evaluation.
Profile-guided optimisation (PGO) on Grab services
Profile-guided optimisation (PGO) improves Go application performance by feeding production CPU profile data into subsequent compiler builds. Teams evaluated PGO across multiple Grab services, including TalariaDB and an orchestrator within the Catwalk service, using Go 1.20 and Go 1.21 builds. TalariaDB saw substantial resource savings after compiling Docker images with a 360-second pprof profile, resulting in at least a 10% drop in CPU usage, a 30% drop in memory consumption, and a 38% reduction in persistent volume usage for event ingestion queues. Conversely, testing on Catwalk showed only a 5% performance gain after adjusting the profiling duration from 59 seconds to 6 minutes, an outcome deemed insufficient to justify pipeline modifications. Broader adoption across monorepo services remains constrained until the central build process supports PGO arguments and automated profile retrieval.
pgvector: Fewer dimensions are better
Embeddings support semantic search, retrieval augmented generation, and clustering in pgvector, but high-dimensional vectors increase storage, memory, and distance-computation costs as datasets grow. The post explains how IVF indexes reduce the records examined and compares embedding models using Massive Text Embedding Benchmark results, noting little correlation between dimension count and performance. In a 224,482-vector Wikipedia dataset, all-MiniLM-L6-v2 with 384 dimensions delivered 78% higher queries per second than OpenAI text-embedding-ada-002 with 1536 dimensions at accuracy@10 of 0.99, while using 4GB rather than 7.5GB of RAM. The 384-dimensional gte-small model produced more than a 200% queries-per-second improvement over all-MiniLM-L6-v2 at the same accuracy, though model choice still depends on task performance, sequence length, language support, and operational costs.
Inference characteristics of Llama
Llama-2-70B is evaluated against GPT-3.5-turbo on serving cost and latency, with the comparison deliberately excluding GPT-4 and noting fine-tuning as a separate reason to choose Llama. The evaluation serves Llama on two 80-GB A100 GPUs in 16-bit precision, then uses transformer FLOP, memory-bandwidth, and KV-cache analysis to explain the difference between prompt processing and token generation. The measurements indicate that Llama-2 has worse completion-token pricing than GPT-3.5 on two A100s, while its prompt-token pricing is more than three times cheaper; eight A100s might improve pricing but introduce unacceptable latency. The resulting guidance favors GPT-3.5 for most workloads and open-source Llama models for prompt-heavy tasks such as classification or reranking, with batch processing and workloads lacking prompt tokens identified as additional cases where Llama may make sense.
pgvector 0.4.0 performance
The post benchmarks pgvector 0.4.0 for approximate nearest-neighbor search and assesses whether PostgreSQL can support production-scale vector workloads. Using the ANN Benchmarks methodology, pre-warmed RAM, the dbpedia-entities-openai-1M dataset of 1 million 1,536-dimensional OpenAI embeddings, and Supabase ARM instances, it varies probes, lists, concurrency, and compute size. On an 8-core, 32 GB instance, pgvector reached 0.91 accuracy@10 at 380 QPS with 10 probes and 0.98 accuracy at 140 QPS with 40 probes; a 64-core, 256 GB server reached about 1,800 QPS at 0.91 accuracy. The results indicate predictable scaling, while higher lists can improve query speed at the cost of index-build time, and the roadmap identifies HNSW, product quantization, and parallel index builds as planned improvements.
Performance Testing At Scale—for BFCM and Beyond
Shopify treats Black Friday Cyber Monday (BFCM) as its most critical period, and uses full-scale BFCM Scale Testing to assess whether the platform can handle peak traffic. Preparation combines capacity planning, resiliency testing, single-application performance testing, and repeated exercises that scale core services to their BFCM profiles before generating load. The hybrid architecture centers on Shopify Core, a monolithic Ruby application supported by services for storefront rendering, analytics, fraud analysis, payments, webhooks, and more. Five load flows cover browsing and buying, admin activity, flash sales, Storefront APIs, and Hydrogen and Oxygen, with Cronograma layering requests gradually and providing an abort switch. The exercises surfaced connection-count pressures, a Kubernetes ingress-nginx endpoint limit, and uneven load-balancer distribution; mitigations included refactoring, scaling dependent services, increasing pod capacity, and dedicated nginx node pools.
Postgres Full Text Search vs the rest
The post compares PostgreSQL Full Text Search with dedicated search engines including MeiliSearch, OpenSearch, SQLite FTS, and Typesense, asking whether a separate system is justified for common application search. It presents a PostgreSQL setup using a generated stored column built with to_tsvector('english', ...) and a GIN index, queried with websearch_to_tsquery. The benchmark loads a movie dataset and focuses on query speed after data ingestion, using comparable scripts and default configurations with limited tuning. PostgreSQL was close in performance to several alternatives; in-memory SQLite was fastest, while Typesense and MeiliSearch handled misspellings correctly and OpenSearch ingested data quickly. The conclusion is that PostgreSQL can provide good-enough search quality and speed for basic cases without adding another system, although dedicated engines offer capabilities PostgreSQL does not.
Implementing "seen by" functionality with Postgres
Tracking unique post views within Postgres presents trade-offs between counter accuracy, row bloat, and concurrent write performance. To evaluate options under an architecture constraint preventing external dependencies, a benchmark suite generated synthetic users, skewed post distributions, and replayed view actions. The benchmark tested four implementations: a naive counter column, an hstore key-value approach, an association table, and HyperLogLog (HLL). Benchmark results demonstrated that simple-hstore achieved the lowest average latency among deduplicating approaches at 2.15 milliseconds, closely followed by HLL at 2.16 milliseconds. Despite hstore's raw performance in prototypes, HLL is recommended for production because it circumvents row bloat as view counts grow while avoiding expensive row counts.
Accelerate Large Model Training using PyTorch Fully Sharded Data Parallel
Increasing parameter counts in modern machine learning models make loading and training them prohibitive on standard hardware. While Distributed Data Parallel replicates the entire model across GPUs, PyTorch Fully Sharded Data Parallel shards optimizer states, gradients, and parameters across data-parallel workers. Benchmarks on causal language modeling using GPT-2 Large and XL across Titan RTX GPUs demonstrate that FSDP increases allowable batch sizes and prevents out-of-memory errors encountered in standard DDP. Integrating FSDP through Hugging Face Accelerate enables these capabilities alongside CPU offloading to run models exceeding GPU memory limits. However, Accelerate requires preparing models before creating optimizers when using multiple models and notes that mixed precision support with FSDP currently has transformer compatibility issues.
Test Budget: Time Constrained CI Feedback
Shopify’s core monolith contains more than 170,000 tests, and variable CI feedback times create delays and context switches as the test pool grows. The Test Infrastructure team evaluated a Test Budget approach that builds prioritized orders on top of deterministic test selection, then stops execution at a predetermined time to increase the chance of finding failures early while accepting some risk. Six criteria were compared—failure_rate, avg_duration, churn, coverage, complexity, and default/random—using Time to First Failure, APFD, and Convergence Index; historical results were stored and queried with Redis Sorted Sets. The analysis found failure_rate had the best detection rate, while churn and complexity were inappropriate; running 60% of the suite detected 80% of failures in the mean case, and 70% of the test-selection suite detected 50% at the fifth percentile.
Adding the V8 CPU Profiler to v8go
v8go adds support for V8’s CPU Profiler, allowing Go users to measure JavaScript execution in a V8 isolate and inspect sampled top-down call trees with function and engine details. The API work aimed to stay idiomatic for Go, remain close to V8’s API, support future profiler features, and minimize Cgo overhead. The first design loaded profile data lazily, making separate Cgo calls for profile and node properties; the alternative built the complete graph in C++ before returning it to Go. Benchmarks showed lazy loading was faster for a small tree, averaging about 20 microseconds versus 25 for eager loading, while a larger tree reversed the result at roughly 90 versus 60 microseconds. The comparison demonstrates that Cgo costs depend on workload size and that performance assumptions require benchmarks using representative inputs.
Case Study: Millisecond Latency using Hugging Face Infinity and modern CPUs
Deploying large Transformer models at scale often faces severe prediction latency bottlenecks, driving up infrastructure costs and limiting real-time production use cases. To address this challenge, Hugging Face evaluated Infinity, a containerized hardware-optimized inference solution paired with the Infinity Multiverse model optimization service. Testing covered 192 configurations on Amazon EC2 C6i instances powered by 3rd generation Intel Xeon Scalable processors across varying CPU cores, sequence lengths, and batch sizes. The benchmarks demonstrated that an Ice Lake-optimized DistilBERT container achieved up to 800% higher throughput than vanilla Transformers and delivered 1 to 4 millisecond end-to-end latencies for sequence lengths up to 64 tokens. Although Infinity was later discontinued in favor of Inference Endpoints and Optimum libraries, the results demonstrated substantial efficiency gains on modern CPU hardware.
Using Propensity Score Matching to Uncover Shopify Capital’s Effect on Business Growth
Shopify’s data team sought to estimate whether accepting Shopify Capital for the first time increased merchants’ subsequent sales, without randomly denying funding in an A/B test. They compared US first-time adopters in January 2019 with Canadian shops that later pre-qualified for Capital, using propensity scores built from Shopify sales and performance characteristics and matching them with a caliper-based greedy nearest-neighbor algorithm. After matching about 600 treated shops against roughly 8,000 controls, they evaluated log-transformed cumulative GMV from February through July 2019 with a binary regression and checked balance using standardized mean differences, visual comparisons, and variance ratios. The estimated geometric-average GMV was 36% higher for US merchants, with a 95% confidence interval of 13% to 65%; robustness checks supported a positive treatment effect.
Winning AI4TSP: Solving the Travelling Salesperson Problem with Self-programming Machines
The 2021 AI4TSP Competition addressed the Travelling Salesperson Problem when travel times are sampled from probability distributions and locations carry rewards and time windows. With n locations, the number of possible tours is n!, making exhaustive enumeration impractical, while the competition also required decisions under uncertain travel outcomes. One track required a fixed tour optimized for expected performance across scenarios; the other allowed an adaptive policy to select the next location as time elapsed, with results evaluated across many instances and realizations. The Shopify-led team used dialectic search for the fixed-tour track and deep reinforcement learning for the on-the-fly track, and won first prize in both.
Scaling-up BERT Inference on CPU (Part 1)
Scaling BERT-like model inference on CPUs presents practical deployment challenges as transformer architectures proliferate in production environments. Using a reproducible Hydra-based benchmarking framework, inference workloads were evaluated on an Amazon Web Services c5.metal instance featuring an Intel Xeon Platinum 8275 processor with AVX512 and VNNI support. Out-of-the-box evaluations showed PyTorch 1.8.1 delivering lower latency than Google TensorFlow 2.4.0, likely due to PyTorch relying on OpenMP and Intel MKL/oneDNN rather than Eigen. Implementing multiple inference streams by pinning parallel model instances to dedicated CPU cores yielded near-linear throughput scaling. Matching instance core allocations to specific problem sizes also demonstrated that smaller workloads can achieve optimal latency on fewer cores, enabling up to a sixfold reduction in cloud compute costs.
Fit More and Train Faster With ZeRO via DeepSpeed and FairScale
Machine learning model sizes frequently outpace consumer GPU memory, preventing practitioners from loading or fine-tuning large architectures on standard hardware. To address these resource constraints, the Hugging Face Trainer introduced experimental support for Zero Redundancy Optimizer features from DeepSpeed and FairScale starting in transformers v4.2.0. In dual-GPU benchmarks with a t5-large model, FairScale and DeepSpeed integrations expanded allowable batch sizes and shortened training durations compared to baseline DistributedDataParallel runs. Furthermore, DeepSpeed permitted single-GPU fine-tuning of a t5-3b model on a 24GB RTX-3090 card at batch size 20, whereas the baseline execution crashed immediately with an out-of-memory error. These optimizations manage memory allocation internally to curb memory fragmentation while requiring changes only to training scripts rather than model code.
Unpopular Opinions: Reddit vs Twitter Data
An analysis of 1.6 million social-media mentions from January 1 to June 30, 2020 examines consumers’ “unpopular opinions” and compares Reddit with Twitter. Overall mentions rose 34% from March through June versus the preceding four months, while Reddit’s r/unpopularopinions posts increased 105% during that period. Across online sources, pop-culture characters generated 63,000 mentions, followed by TV shows at 55,000, fandoms at 45,000, and books at 44,000; Friends was criticised for aspects considered unacceptable today. Reddit contributed 958,000 mentions, including 19,000 about subreddits, while Twitter contributed 554,000, led by 15,000 movie and TV opinions. The platform breakdown also surfaces differences such as Reddit’s meta discussions and Twitter’s opinions about money, chocolate, and ice.
Top topics
- Architecture830
- Performance812
- LLMs648
- Machine Learning617
- Open Source561
- AI Agents542
- AI454
- Developer Experience446
Weekly digest
The best engineering reads, once a week.