---
title: "Performance"
description: "817 posts about Performance, summarised, each linking to the original."
---

# Performance
> 817 posts about Performance, summarised, each linking to the original.

## Articles

### [Iterating Towards a More Scalable Ingress](https://yomu.fyi/post/iterating-towards-a-more-scalable-ingress.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: Francisco Mejia
- Published: Aug 31, 2018
- Type: Benchmark

Shopify’s infrastructure uses Kubernetes to serve millions of requests per minute, and its growth exposed ingress performance problems during frequent application deployments. After replacing Google Cloud Load Balancer Controller (glbc) with endpoint-aware ingress-nginx, the Edgescale team focused on reducing the NGINX reloads that caused latency increases. They added dynamic configuration: the controller POSTs endpoint lists to an NGINX HTTP endpoint, where lua-nginx-module updates an in-memory shared dictionary instead of regenerating the configuration file and reloading NGINX. Reload avoidance preserves keepalive connections and load-balancing state while reducing the resource effects of replacing worker processes. Tests found ingress-nginx outperformed glbc by multiple orders of magnitude at the 99.99th latency percentile or higher, while dynamic configuration produced its largest gains during deploys, especially beyond the 80th percentile.


### [Grab Senior Data Scientist Liuqin Yang Wins Beale-Orchard-Hays Prize](https://yomu.fyi/post/grab-senior-data-scientist-liuqin-yang-wins-beale-orchard-hays-prize.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Yang Liuqin
- Published: Jul 20, 2018
- Type: Announcement

Grab Senior Data Scientist Dr. Liuqin Yang, Professor Defeng Sun, and Professor Kim-Chuan Toh received the 2018 Beale-Orchard-Hays Prize for their research paper introducing SDPNAL+. The software employs a majorised semismooth Newton-CG augmented Lagrangian method to solve large-scale semidefinite programming problems with nonnegative constraints. While traditional methods struggled beyond matrix dimensions of 2,000 and 5,000 constraints, SDPNAL+ successfully scales to matrix dimensions of 9,261 and over 12 million constraints. In benchmark testing, the software solved a problem on a desktop PC in 1.5 hours that required 122 hours on a 56-core CPU and 128-GPU cluster using a traditional solver. Grab implements these optimisation techniques to accelerate its passenger-driver allocation algorithms by hundreds of times.


### [MJIT: A Method Based Just-in-time Compiler for Ruby](https://yomu.fyi/post/mjit-a-method-based-just-in-time-compiler-for-ruby.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: Jonan Scheffler
- Published: Apr 12, 2018
- Type: Explainer

Ruby programs are compiled into YARV instructions and executed by a stack-oriented virtual machine, while register-oriented representations can expose more opportunities for compiler optimization. The post explains how YARV represents a simple expression, contrasts stack and register architectures, and introduces Vladimir Makarov’s Ruby RTL, an intermediate representation distinct from GCC’s RTL. MJIT uses RTL generated by MRI to make speculative, operand-specific optimizations, then compiles methods to machine code rather than relying only on interpretation. The interview discusses deoptimization: when assumptions change, MJIT generates a new machine-code version and can switch between JITed execution and RTL interpretation. The participants say this overhead should be tiny and should not degrade performance overall, while acknowledging edge cases where MJIT is not faster and that benchmarking JITs is difficult.


### [Future Proofing Our Cloud Storage Usage](https://yomu.fyi/post/future-proofing-our-cloud-storage-usage.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: Andrew Louis
- Published: Feb 26, 2018
- Type: Problem & solution

Shopify’s growth from 375,000 to over 600,000 merchants coincided with about 450,000 S3 operations per second and rising error rates plus read and write latency. Merchant-uploaded images and theme installations exposed a partitioning problem: S3 groups keys by shortest common asset prefixes, and abrupt throughput spikes could trigger SlowDown failures across a range of shops. Black-box observations suggested affected keyspaces were unlikely to recur, while newer shops were more often involved, though the latter was explicitly described as a shaky inference. Shopify prepended a hash-generated digest to new shops’ asset keys, distributing writes across more partitions instead of concentrating them under shop-specific prefixes. After the first post-change SlowDown event, exceptions nearly disappeared, while median latency for digested assets fell about 60% and the 95th percentile about 25%.


### [Deep Dive into Database Timeouts in Rails](https://yomu.fyi/post/deep-dive-into-database-timeouts-in-rails.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Jia Hao Goh
- Published: Jan 29, 2018
- Type: Explainer

Following a production outage where a database failover caused a Ruby on Rails application to exhaust its Puma server threads, an investigation was conducted to understand how ActiveRecord and MySQL timeout settings behave. A reproduction environment using Docker, Puma, and Toxiproxy replicated how hanging requests to a failing database consume all available server threads, ultimately starving unrelated endpoints. The analysis breaks down ActiveRecord connection pooling mechanics alongside underlying mysql2 and libmysqlclient settings, specifically checkout\_timeout, connect\_timeout, and read\_timeout. Testing confirmed how existing and new TCP connections transition through socket states during network interruptions while waiting on configured timeout intervals.


### [Dealing with the Meltdown Patch at Grab](https://yomu.fyi/post/dealing-with-the-meltdown-patch-at-grab.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Althaf Hameez
- Published: Jan 7, 2018
- Type: Incident / postmortem

AWS infrastructure maintenance related to Meltdown patches led to severe CPU utilization spikes across Grab's ElastiCache Redis instances. Because Redis is single-threaded, spikes past 50% CPU on two-vCPU instances threatened service capacity, and initial Multi-AZ failovers only provided temporary relief until the new master nodes received rolling patches. To handle the increased overhead before their peak traffic window, the engineering team horizontally scaled both clustered and non-clustered Redis fleets. For Redis 3.2.4 clusters lacking live re-sharding support, they provisioned larger clusters, warmed caches, and redirected traffic. Non-clustered workloads were resolved by provisioning extra nodes, migrating compatible services to Redis Cluster, or updating application code to shard data across multiple instances.


### [Namespacing Actions for Redux](https://yomu.fyi/post/namespacing-actions-for-redux.md)
- Company: [Kickstarter](https://yomu.fyi/company/kickstarter.md)
- Author: Sarah Groff Hennigh-Palermo
- Published: Nov 20, 2017
- Type: Explainer

Kickstarter’s front-end team encountered a Redux action-namespacing problem when multiple instances of the same functional React component responded to actions intended for only one instance. They rejected local state because the application used functional components and redux-loop, which routes state updates and side effects through reducers, and they sought programmatic namespacing compatible with combineReducers and a low-surface-area component API. The team compared nested reducers, higher-order reducers, and a closure-based module pattern; the latter met the stated constraints with namespaced actions, wrapper components, reducers, and a namespace returned by a factory that defaults to a UUID. Namespaces must be preserved when reducers trigger further actions, including redux-loop results, and complex composition can lose the required binding. The approach has worked for most reusable and compound components, while possible performance costs from many reducers remain a future concern.


### [Upgrading Kickstarter to Rails 5](https://yomu.fyi/post/upgrading-kickstarter-to-rails-5.md)
- Company: [Kickstarter](https://yomu.fyi/company/kickstarter.md)
- Author: Logan McDonald
- Published: Nov 6, 2017
- Type: Problem & solution

Kickstarter upgraded its main Rails application to Rails 5, alongside payments app Rosie, to support site security and performance while moving a nine-year-old legacy codebase forward. The team planned a three-month project, upgraded Ruby 2.2.5 to 2.4.x, classified release-note changes, resolved dependency conflicts, used Rails’ upgrade script, and moved through twelve documented steps. Work included merging Rails 4.2-compatible fixes, repairing nearly 4,000 CI test failures from a roughly 70 percent coverage baseline, removing deprecations, smoke-testing major flows, backporting more than thirty pull requests, and preparing reviewers and deployment contingencies. The upgrade met its timeline without rollbacks or major downtime, although untested bugs increased Zendesk tickets and exposed issues involving ActionController::Parameters, tracking events, and mobile applications. A retrospective emphasized planning, documentation, cross-team communication, and broader native-team participation for future upgrades.


### [How We Scaled Our Cache and Got a Good Night's Sleep](https://yomu.fyi/post/how-we-scaled-our-cache-and-got-a-good-night-s-sleep.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Gao Chao
- Published: Jun 19, 2017
- Type: Problem & solution

Growing business load on the Common Data Service (CDS) created potential bottlenecks for its single-threaded Redis cache on ElastiCache, necessitating horizontal scaling for greater capacity and throughput. After ruling out master-slave replication and intermediate Twemproxy setups due to memory constraints and proxy I/O bottlenecks, the team implemented client-side sharding. Using an internal Go package for consistent hashing, CDS instances hash cache keys locally to determine the target shard. The implementation encapsulates hashing inside a thin \`ShardedCache\` wrapper sharing the original cache interface while supporting Ketama and custom hash functions. Deploying via double-writing cron jobs during off-peak hours reduced database read pressure and improved P99 latency.


### [Functional Swift Conference 2017 — Brooklyn](https://yomu.fyi/post/functional-swift-conference-2017-brooklyn.md)
- Company: [Kickstarter](https://yomu.fyi/company/kickstarter.md)
- Author: Brandon Williams
- Published: Apr 28, 2017
- Type: Announcement

Kickstarter hosted the fourth Functional Swift Conference on April 15th, marking the third time the event had been held at its office. The all-day gathering brought together 10 speakers, including four Kickstarter engineers, and 90 attendees to discuss what functional programming can contribute to better Swift code. Sessions covered recursive enums and KD-trees, testing side-effects, domain-specific languages with GraphQL, algebraic laws, monoids, predicates, sorting functions, applicative functors, parsing, and type inference. Examples included using value types and recursive data structures to describe complicated objects and live-coding an applicative parsing API that generates its own documentation. The closing type-system talk examined why inference is hard and described the Swift compiler team’s use of graph theory to address inference problems practically.


### [How We Sped up SNI TLS Handshakes by 5x](https://yomu.fyi/post/how-we-sped-up-sni-tls-handshakes-by-5x.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: Fred Hebert
- Published: Dec 22, 2016
- Type: Incident / postmortem

Heroku’s SNI TLS termination work exposed severe handshake performance problems under benchmarks designed to avoid session resumption, caching, and keep-alive reuse. Initial tests made nodes slow and unresponsive 7–8x earlier than comparable HTTP tests, while perf top showed Erlang garbage collection and ETS hash operations consuming unexpectedly large amounts of CPU. Tracing linked repeated ets.select\_delete calls to the centralized ssl\_manager process, where ssl\_pkix\_db’s certificate cache path was still used even for DER certificates supplied directly in memory. Engineers patched the SSL library to bypass that cache wholesale. Each instance then handled three times the benchmark capacity, while SNI latencies fell from 100–120 milliseconds to 20–25 milliseconds for the reported app subset; the patch was later shipped in Erlang/OTP-19.1.


### [Apache Kafka 0.10 Evaluating Performance in Distributed Systems](https://yomu.fyi/post/apache-kafka-0-10-evaluating-performance-in-distributed-systems.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: Tom Crayford
- Published: May 27, 2016
- Type: Benchmark

Testing Apache Kafka 0.10 revealed an apparent 33% throughput decline versus Kafka 0.9, prompting an investigation into whether the regression affected different plans, connection types, message sizes, and workload scales. Heroku’s benchmark system used multiple producer and consumer dynos, while the team applied Brendan Gregg’s USE method to examine resource utilization, saturation, and errors. The diagnosis was network-card saturation: Kafka 0.10 adds an 8-byte timestamp to every message, which at 1.5 million messages per second and replication factor 3 produced 288 Mbps of additional cluster traffic, causing packet drops and retransmits. Reproducing the effect by adding 8 bytes to Kafka 0.9 confirmed the explanation, and the post says most production workloads are unlikely to be affected, though clusters near network limits may be.


### [Speeding Up Sprockets for Faster Asset Pipelines](https://yomu.fyi/post/speeding-up-sprockets-for-faster-asset-pipelines.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: Richard Schneeman
- Published: Feb 22, 2016
- Type: Explainer

The Rails asset pipeline can take over 20 times longer than dependency installation because Sprockets consumes memory, disk I/O, and CPU while reading, fingerprinting, compressing, and resolving dependent assets. Reusing its disk cache failed across Capistrano releases and Heroku builds because cache keys and dependency data embedded absolute paths, so identical files appeared different when compiled from changing directories. Sprockets 3.3+ addresses this with UnloadedAsset, normalized relative cache keys, and URITar, which compresses paths and URIs before caching and expands them when loading; Windows portability was fixed in 3.3.4. In an example app, the initial compile remained about 38 seconds, while the second took 3 seconds, a roughly 1,200% speed increase when compiled assets were reused.


### [How We Migrated to Active Record 4](https://yomu.fyi/post/how-we-migrated-to-active-record-4.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: Owen Ou
- Published: Nov 4, 2015
- Type: Problem & solution

Heroku Platform API had remained on Active Record 2.3.18, released in March 2013, creating maintenance and security burdens as support ended and custom query infrastructure replaced newer features. The team moved first to Active Record 3.0.20, preserving 2.x APIs, then directly to 4.2.4, using a feature branch, console-driven fixes, automated and acceptance tests, private deployments, staging, and a partial production rollout. The first step took one engineer about two weeks, while the second took two engineers about two weeks; deployment had no downtime but briefly required a rollback for an acts\_as\_paranoid bug. Performance improved 18% from 2.3.18 to 3.0.20 but worsened 13% from 3.0.20 to 4.2.4, challenging micro-benchmark expectations. The upgrade ultimately left performance at the same overall level, removed patches, restored supported status, and made future updates and training easier.


### [We Patched Ruby On Rails for a 10% Performance Bump](https://yomu.fyi/post/we-patched-ruby-on-rails-for-a-10-performance-bump.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: Richard Schneeman
- Published: Aug 6, 2015
- Type: Benchmark

Rails response time improved by more than 10% in a patch that also reduced allocated objects by 29%, motivated by memory and performance problems observed in production Heroku applications. The work began with customer reports of R14 out-of-memory errors, swap usage, and suspected memory leaks, then used a customer app and reusable benchmarks to reproduce issues. The derailed benchmarks perf:objects task and memory\_profiler exposed allocation hot spots, including arrays joined into strings and duplicated hashes; refactoring reduced unnecessary objects, while routing changes produced a 31% speed increase for constrained URL generation. The resulting savings required no application changes and were planned for the next Rails version, with Rails 5 pre-release testing mentioned.


### [Incremental Garbage Collection in Ruby 2.2](https://yomu.fyi/post/incremental-garbage-collection-in-ruby-2-2.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: Koichi Sasada
- Published: Feb 4, 2015
- Type: Explainer

Ruby 2.2 introduces RincGC, an incremental garbage collector intended to reduce the long pause times that Ruby 2.1’s generational collector could produce during major GC. The approach splits marking into fine-grained steps interleaved with Ruby execution, while lazy sweeping remains incremental; because CRuby lacks sufficient write barriers, unprotected objects are rescanned before white objects are reclaimed. The article explains the white, grey, and black object model, the write-barrier safety problem, and uses GC::Tracer events with a Rails benchmark simulating 30,000 requests to measure pauses. In the reported measurement, generational GC produced seven large pauses with a maximum of about 15 ms, while incremental GC reduced the maximum to around 2 ms, without reducing total GC time or improving throughput.


### [Benchmarking Rack Middleware and Improved Speed](https://yomu.fyi/post/benchmarking-rack-middleware-and-improved-speed.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: Richard Schneeman
- Published: Nov 3, 2014
- Type: Benchmark

ActionDispatch::Static, enabled by config.serve\_static\_assets = true, checks incoming Rails requests to decide whether to serve a disk file or pass the request onward. To measure the middleware in isolation, the author loads it directly, wraps it around a no-op Proc, and uses Rack::MockRequest with benchmark-ips to compare requests against the no-op baseline. The original middleware achieved 15,891.2 iterations per second, versus 28,660.9 for the no-op, while the modified version reached 24,490.9 iterations per second. That change made non-asset middleware requests roughly 54% faster, bringing the gap to the no-op case to about 10%, although asset requests became slightly slower and the difference was not measurable. Across the full Rails application stack, the reported overall speedup was approximately 2.6%.


[Newer posts](https://yomu.fyi/topic/performance/page/40.md)