---
title: "Performance"
description: "817 posts about Performance, summarised, each linking to the original."
---

# Performance
> 817 posts about Performance, summarised, each linking to the original.

## Articles

### [Scaling-up BERT Inference on CPU (Part 1)](https://yomu.fyi/post/scaling-up-bert-inference-on-cpu-part-1.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Morgan Funtowicz
- Published: Apr 20, 2021
- Type: Benchmark

Scaling BERT-like model inference on CPUs presents practical deployment challenges as transformer architectures proliferate in production environments. Using a reproducible Hydra-based benchmarking framework, inference workloads were evaluated on an Amazon Web Services c5.metal instance featuring an Intel Xeon Platinum 8275 processor with AVX512 and VNNI support. Out-of-the-box evaluations showed PyTorch 1.8.1 delivering lower latency than Google TensorFlow 2.4.0, likely due to PyTorch relying on OpenMP and Intel MKL/oneDNN rather than Eigen. Implementing multiple inference streams by pinning parallel model instances to dedicated CPU cores yielded near-linear throughput scaling. Matching instance core allocations to specific problem sizes also demonstrated that smaller workloads can achieve optimal latency on fewer cores, enabling up to a sixfold reduction in cloud compute costs.


### [How We Improved Agent Chat Efficiency with Machine Learning](https://yomu.fyi/post/how-we-improved-agent-chat-efficiency-with-machine-learning.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Suman Anand
- Published: Apr 19, 2021
- Type: Problem & solution

Agent typing time represented a large portion of Grab's chat support journey, and 85% of messages were still free typed because agents customized static templates to fit their personal style. To accelerate typing across multilingual markets without robotic templates, Grab built SmartChat, a machine learning feature that provides contextual sentence completion. The team opted for a lightweight seq2seq architecture using single-layered GRU encoder-decoders in TensorFlow instead of bulky attention models to keep model latency under 100ms. The user interface was implemented in React using a content-editable div with inline typeahead suggestions activated via keyboard shortcuts.


### [Understanding BigBird's Block Sparse Attention](https://yomu.fyi/post/understanding-bigbird-s-block-sparse-attention.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Vasudev Gupta
- Published: Mar 31, 2021
- Type: Explainer

Transformer-based models face severe computational bottlenecks due to quadratic time and memory complexity, making sequences longer than 512 tokens impractical. BigBird addresses this limitation by approximating full attention with block sparse attention, scaling context lengths up to 4096 tokens at a lower computational cost. Instead of attending to all tokens, BigBird combines sliding attention for local context, global tokens for long-range relationships, and random tokens to accelerate information transfer. The model is integrated into HuggingFace Transformers, enabling fine-tuning for tasks like long-document summarization and extractive question answering. When sequences are shorter than 1024 tokens, using original full attention remains recommended.


### [How Grab Leveraged Performance Marketing Automation to Improve Conversion Rates by 30%](https://yomu.fyi/post/how-grab-leveraged-performance-marketing-automation-to-improve-convers.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Sc Ng
- Published: Mar 22, 2021
- Type: Problem & solution

Grab faced operational bottlenecks managing direct-response Google Ads campaigns across thousands of ad groups due to its hyperlocal marketing across Southeast Asian markets. To eliminate the manual burden of tracking and updating ad creatives, the team built CARA, an in-house automation tool deployed on AWS serverless compute. CARA utilizes standardized file naming conventions to map assets to specific campaigns and connects with Google Ads and YouTube APIs to detect and replace low-performing assets. During an experimental rollout across more than 8,000 active ad groups, CARA replaced nearly 2,000 underperforming creatives. The automated asset replacement workflow produced an 18% to 30% increase in clickthrough and conversion rates.


### [My Journey to a serverless transformers pipeline on Google Cloud](https://yomu.fyi/post/my-journey-to-a-serverless-transformers-pipeline-on-google-cloud.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Dominici
- Published: Mar 18, 2021
- Type: Tutorial

The post documents Maxence Dominici’s effort to deploy a Transformers sentiment-analysis pipeline on Google Cloud for classifying customer reviews from Discord. After considering AI Platform Prediction and App Engine, he chose Cloud Run with a Docker image because custom checkpoint handling, dependencies, and resource requirements made the other options unsuitable. The implementation uses a PyTorch DistilBERT model fine-tuned on SST-2, Flask, Gunicorn, and a GET request protected by an optional API key; the shown handler retrieves a review but passes the hard-coded text “that was great” to the classifier. Cloud Run is configured with 4 GB of memory, one worker, and one thread to limit memory pressure and billing. Requests take under five seconds including model loading, while cold starts may add about ten seconds, and the author reports that the pretrained model’s accuracy was already impressive.


### [Hugging Face Reads, Feb. 2021 - Long-range Transformers](https://yomu.fyi/post/hugging-face-reads-feb-2021-long-range-transformers.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Victor Sanh
- Published: Mar 9, 2021
- Type: Explainer

Standard transformer models face severe memory and compute bottlenecks due to the quadratic scaling of self-attention with sequence length. To scale beyond short sequence limits, efficient architectures introduce varied mechanisms such as custom sparse attention patterns, compressed recurrence, low-rank projections, and kernel approximations. Longformer integrates local dilated windowed attention with task-specific global tokens, functioning as a drop-in replacement that enables standard pre-trained models to process long inputs without full retraining. Concurrently, Compressive Transformers extend temporal memory by compressing discarded activations, while models like Linformer and Performer reduce computational complexity through projection and kernel estimation. Progress in long-range modeling also reveals that incrementally training models from short to longer sequences accelerates convergence and improves downstream performance.


### [Cracking PostgreSQL Interview Questions](https://yomu.fyi/post/cracking-postgresql-interview-questions.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: Ant Wilson
- Published: Feb 27, 2021
- Type: Tutorial

The guide prepares candidates for PostgreSQL interviews by concentrating on transactional SQL rather than mixing it with analytical topics such as window functions, ranking, and aggregates. It covers relational modeling with foreign keys and join tables, PostgreSQL’s richer types, schemas, indexing strategies, and EXPLAIN ANALYZE for comparing estimated and actual query costs. It also addresses views, automatically updatable views and their limitations, database permissions, roles, PUBLIC grants, row-level security policies, and business logic implemented with PL/pgSQL functions and triggers. The guide recommends thinking in sets with SQL operations such as JOIN and UNION, using common table expressions for readable or recursive queries, and practicing these topics to build a strong foundation for PostgreSQL interview answers.


### [Keeping Developers Happy with a Fast CI](https://yomu.fyi/post/keeping-developers-happy-with-a-fast-ci.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Feb 24, 2021
- Type: Problem & solution

Shopify’s Test Infrastructure team tackled a 45-minute 95th-percentile CI time for its Core monolith, which had more than 170,000 tests and ran on every git push. They instrumented Buildkite commands and prioritized agent preparation, dependency building, and test execution using command frequency and duration. Disk I/O from writing large caches slowed Docker startup, so the team increased disk capacity and write speed, mounted most caches read-only, skipped unchanged database and asset work through hashes, parallelized setup, and expanded changed-file test mappings. These changes reduced container-start p95 from 90 to 25 seconds, raised builds avoiding the full suite from 45% to over 60%, improved test stability from 88% to 97%, and lowered overall CI p95 to 18 minutes.


### [Read Consistency with Database Replicas](https://yomu.fyi/post/read-consistency-with-database-replicas.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Feb 22, 2021
- Type: Explainer

Database replicas can improve read-heavy application performance and reduce pressure on primary servers, but differing replication lag can make multi-query reads produce incoherent datasets. Shopify’s Database Connection Management team compared tight consistency and GTID-based causal consistency before choosing opt-in monotonic read consistency, in which related reads are routed to one replica and therefore follow a consistent timeline rather than necessarily seeing the latest data. Their ProxySQL fork accepts a UUID in the consistent\_read\_id query comment, hashes it into a pseudorandom index over a weighted hostgroup server list, and disables later rebalancing that could override the selection. The implementation indexes configured servers before handling outages, though a selected server failure can still break consistency. The approach offers low overhead but tolerates occasional inconsistencies.


### [Retrieval Augmented Generation with Huggingface Transformers and Ray](https://yomu.fyi/post/retrieval-augmented-generation-with-huggingface-transformers-and-ray.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: system
- Published: Feb 10, 2021
- Type: Problem & solution

Huggingface Transformers added the Retrieval Augmented Generation architecture to combine standard sequence-to-sequence models with external document retrieval for knowledge-intensive tasks. Scaling RAG fine-tuning across multiple GPUs previously created bottlenecks because the torch.distributed implementation forced the single rank 0 worker to handle all retrieval index lookups. To resolve this synchronization bottleneck, Ray was integrated into the contextual document retrieval mechanism using stateful actor abstractions. Dedicated Ray processes separate from the training workers now load the large index and process retrieval requests concurrently. This architectural separation delivers up to a 2x speedup per retrieval call and removes the strict dependency on PyTorch for training.


### [Hugging Face on PyTorch / XLA TPUs](https://yomu.fyi/post/hugging-face-on-pytorch-xla-tpus.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: ds, Lysandre
- Published: Feb 9, 2021
- Type: Announcement

PyTorch / XLA integrates Cloud TPU training into Hugging Face transformers while preserving the standard Trainer interface. The implementation adds an xla device type and uses lazy execution to trace intermediate representation graphs before lowering them to XLA Higher Level Operations for compilation. Parallel data loading through MpDeviceLoader overlaps CPU tracing with TPU execution, while xm.optimizer\_step coordinates gradient consolidation across data-parallel replicas. Checkpoints use xm.save to transfer tensors via CPU storage and restrict disk writes to the master process. Benchmarks on a Cloud TPU v3-8 system training bert-large-uncased on WikiText-103 demonstrate completion times of 178.4 minutes in FP32 precision and 106.4 minutes in BF16 precision.


### [Supabase Beta January 2021](https://yomu.fyi/post/supabase-beta-january-2021.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: Paul Copplestone
- Published: Feb 2, 2021
- Type: Announcement

Supabase released a collection of platform updates during January 2021 covering database tools, authentication integrations, regional availability, and infrastructure performance. Client libraries added support for PostgREST exact, planned, and estimated count queries to eliminate counting friction commonly experienced in platforms like Firebase. Authentication capabilities expanded to include Facebook and Azure login providers, an in-dashboard GoTrue audit trail, a helper function for user emails, and a prebuilt React Auth Widget. The platform also enabled new database hosting regions in London and Sydney while transitioning storage to GP3 for improved throughput. Furthermore, migrating API subdomains to Route53 alongside custom Let's Encrypt certificates yielded read performance benchmark gains of up to twelve percent.


### [Serving Driver-partners Data at Scale Using Mirror Cache](https://yomu.fyi/post/serving-driver-partners-data-at-scale-using-mirror-cache.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Indrajit Sarkar
- Published: Jan 26, 2021
- Type: Problem & solution

Grab's Drivers Data service handles up to 10,000 requests per second during peak hours to supply driver information across backend microservices. The original setup used MySQL with Redis and standalone in-memory local caches, but yielded a low 25% local cache hit rate due to traffic patterns characterized by high burst frequency for individual drivers alongside redundant database calls across nodes. To solve this, the team developed Mirror Cache, an in-memory caching system that pairs Dgraph's Ristretto library with an asynchronous gRPC replication layer to mirror updates across cluster nodes. The replicator batches updates within the same AWS availability zone and forwards data to single nodes across zones to minimize transfer overhead. Production deployment increased the in-memory cache hit rate to approximately 75% and reduced direct MySQL queries by 5%.


### [Faster TensorFlow models in Hugging Face Transformers](https://yomu.fyi/post/faster-tensorflow-models-in-hugging-face-transformers.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Julien Plu
- Published: Jan 26, 2021
- Type: Tutorial

Hugging Face updated Transformers to improve the computational performance and robustness of its TensorFlow models, including BERT, RoBERTa, ELECTRA, and MPNet. In version 4.2.0, the optimized BERT implementation runs up to approximately 10 percent faster than Google's official implementation and twice as fast as Transformers release 4.1.1 when tested on a V100 GPU. The release also improves TensorFlow SavedModel generation by allowing flexible sequence lengths across runs, exposing all model inputs, and grouping hidden states or attention outputs into single tensors. In addition, users can customize serving signatures by subclassing models and decorating custom methods with explicit input specifications. Models exported to the SavedModel format can be packaged into Docker containers and served via TensorFlow Serving using either HTTP REST or gRPC APIs for production inference.


### [Using GraphQL for High-Performing Mobile Applications](https://yomu.fyi/post/using-graphql-for-high-performing-mobile-applications.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jan 22, 2021
- Type: Explainer

GraphQL is presented as a way to address mobile developers’ REST API pain points, especially repeated JSON-to-model mappings and failures when server fields or types change. It provides a strongly typed, self-documented schema contract, while introspection enables tooling to generate client code and surface deprecations at compile time. The post describes mobile practices including using reusable UI-oriented fragments, aliases for readable data models, and directives that conditionally include or skip fields; Shopify uses the open-source Syrup tool to generate strongly typed Swift and Kotlin code from application queries. This flexibility shifts responsibility toward query design: Shopify calculates field costs, limits calls through a replenishing per-user cost bucket, and requires clients to consider query complexity for responsiveness and server resources. GraphQL is positioned as language-independent and suited to bandwidth, latency, and UX-sensitive clients.


### [Fit More and Train Faster With ZeRO via DeepSpeed and FairScale](https://yomu.fyi/post/fit-more-and-train-faster-with-zero-via-deepspeed-and-fairscale.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Stas Bekman
- Published: Jan 19, 2021
- Type: Benchmark

Machine learning model sizes frequently outpace consumer GPU memory, preventing practitioners from loading or fine-tuning large architectures on standard hardware. To address these resource constraints, the Hugging Face Trainer introduced experimental support for Zero Redundancy Optimizer features from DeepSpeed and FairScale starting in transformers v4.2.0. In dual-GPU benchmarks with a t5-large model, FairScale and DeepSpeed integrations expanded allowable batch sizes and shortened training durations compared to baseline DistributedDataParallel runs. Furthermore, DeepSpeed permitted single-GPU fine-tuning of a t5-3b model on a 24GB RTX-3090 card at batch size 20, whereas the baseline execution crashed immediately with an out-of-memory error. These optimizations manage memory allocation internally to curb memory fragmentation while requiring changes only to training scripts rather than model code.


### [How we sped up transformer inference 100x for 🤗 API customers](https://yomu.fyi/post/how-we-sped-up-transformer-inference-100x-for-api-customers.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Nicolas Patry
- Published: Jan 18, 2021
- Type: Problem & solution

Deploying large transformer models in production poses severe latency and scaling hurdles for machine learning engineers. Hugging Face achieved a cumulative 100x inference acceleration on its Accelerated Inference API by pairing library-level algorithmic adjustments with low-level hardware compilation. The initial 10x improvement reduces forward-pass computations, such as restricting attention calculations to the latest token in GPT generation tasks, while utilizing Rust-based tokenizers and caching mechanisms. The subsequent 10x speedup requires custom static graph compilation targeted to selected CPU or GPU hardware profiles, applying unused flow removal, instruction-specific layer fusion, and precision quantization via ONNX Runtime. These coordinated techniques enable real-time prediction latency across varied model sizes and demand profiles without sacrificing task accuracy.


### [Trident - Real-time Event Processing at Scale](https://yomu.fyi/post/trident-real-time-event-processing-at-scale.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Jie Zhang
- Published: Jan 13, 2021
- Type: Problem & solution

Trident serves as Grab's internal real-time event-processing and workflow automation engine, driving user campaigns, rewards, and notifications across multiple business lines. To handle peak loads exceeding 2,000 events per second without duplicate execution, the system consumes decoupled Kafka streams and enforces exactly-once semantics using Redis and MySQL deduplication checks. Processing efficiency relies on server autoscaling aligned with Kafka partition counts, combined with dynamic goroutine allocation per consumer. To minimize rule evaluation overhead, Trident indexes active campaigns into an in-memory hash map by event type, cutting processing time by at least 90%. Furthermore, condition evaluation is optimized through lazy loading and a weighted sorting algorithm that checks low-cost in-memory data prior to executing expensive database queries or external service calls.


### [Supabase Beta December 2020](https://yomu.fyi/post/supabase-beta-december-2020.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: Paul Copplestone
- Published: Jan 2, 2021
- Type: Announcement

Supabase transitioned to Beta following ten months of development concentrated on performance, security, and reliability. The platform introduced inline table documentation updates, storing descriptions directly as PostgreSQL comments and exposing them through OpenAPI specifications. Developers can now utilize a table view featuring immediate realtime update reflection alongside pagination for handling large datasets. Performance measurements show a 26 percent increase to 1460 requests per second in regions supporting AWS Graviton instances. Furthermore, the update introduces a new region in Sao Paulo, editing support for native Postgres arrays in the grid editor and side panel, and broader compatibility for custom database types.


### [How Shopify Uses WebAssembly Outside of the Browser](https://yomu.fyi/post/how-shopify-uses-webassembly-outside-of-the-browser.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Dec 18, 2020
- Type: Explainer

Shopify describes using WebAssembly (Wasm) outside the browser to run untrusted Partner code on its own infrastructure, reducing partners’ need to operate scalable web services and avoiding network latency for time-sensitive customizations. Wasm provides a sandboxed, stack-based environment with explicit host imports, protected call stacks, and runtime type checking, while targeting near-native performance and multiple source languages. Shopify wraps Fastly’s open-source Lucet runtime and compiler in a Rust web service called the Wasm Engine, which manages module I/O and storage and applies results during runtime processes such as web requests. In a test executing 100,000 modules per minute for roughly five minutes, Lucet execution hovered around 100 μs and total p99 execution time was about 4 ms. Shopify selected AssemblyScript, with TypeScript-like syntax, for first-class developer support and is extending its tooling, SDKs, monitoring, and language support.


[Newer posts](https://yomu.fyi/topic/performance/page/37.md) · [Older posts](https://yomu.fyi/topic/performance/page/39.md)