Loading…
Topic
812 posts about Performance. Every summary links to the original.
See the idea
Bare metal gives engineers direct control over GPU, network, and virtualization layers, but it shifts elasticity and lifecycle work onto the team. This trade-off map shows when that bargain pays off.
Bun packs a runtime, package manager, bundler, and test runner into one binary. Here is how those four parts fit together and where the seams still show.
A five-step how-to for deciding when an AMD Instinct GPU should stay dedicated and when SR-IOV virtual functions, VMs, or sidecar isolation can safely share it.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Grab experienced severe system strain when sudden localized spikes in ride demand, triggered by events like heavy rain or concert dismissals, coincided with driver shortages. These localized bursts overloaded the platform and degraded the experience for users outside the affected areas. To mitigate this, engineers created the Spampede filter, a circuit-breaker mechanism placed at the start of the booking pipeline. The filter converts pickup locations into Geohash Integer buckets and partitions time using Unix timestamps, tracking unfulfilled requests in Redis with atomic increments and time-to-live expirations. When unallocated requests exceed configured thresholds within a specific bucket, the system immediately short-circuits new incoming bookings to protect overall platform stability.
Grab undertook an initiative to reduce startup time and improve time to interactive (TTI) on its passenger mobile app. Because local benchmarks failed to simulate real device and network conditions, the team instrumented code in production across 8–9 million daily users to capture p50 and p95 metrics. Initial gains came from caching service tiles between sessions and removing a startup animation, saving four seconds. Architectural changes followed, including converting iOS dynamic frameworks to static linking and merging others, while Android initialisation was refactored with Kotlin coroutines. Replacing a heavy third-party analytics library with an internal experimentation platform yielded further startup reductions.
Slow database queries can stem from schema choices as well as inefficient SQL. The example uses PostgreSQL with a json column holding large payloads, then selects a single url value for an app_uuid and orders the results. EXPLAIN ANALYZE with BUFFERS showed 7,106 shared buffer hits, representing about 60MB of JSON data loaded and discarded, while execution reached 1,141.296 milliseconds in the reported run. The solution split the JSON into separate text fields for data and url, bringing query time down from about 1,200 milliseconds to 10 milliseconds. The broader conclusion is that schemas should be revisited as application usage becomes better understood, but changes are not necessary when the design isn't causing a problem.
Designing digital products for Southeast Asia requires tailoring user experiences to unique regional constraints and consumer behaviors across diverse populations. Users in the region often operate low-end mobile hardware on congested networks while carefully rationing prepaid mobile data. Grab addresses these challenges by designing comprehensively for non-ideal UI stacks, implementing loading skeletons, and replacing heavy video tutorials with lightweight SVG animations to minimize bandwidth consumption. Furthermore, product teams adapt to mobile-only environments by prioritizing phone number and one-time-password registrations while avoiding legacy desktop-era iconography. Visual accessibility is validated by testing UI readability on dimmed, low-resolution screens under bright ambient sunlight.
Sam Saffron discusses Discourse’s approach to Ruby performance and monitoring, arguing that production memory constraints matter more than isolated microbenchmarks for many deployments. He favors clear code and selective optimization, sometimes bypassing ActiveRecord with MiniSql on performance-sensitive paths rather than broadly sacrificing Ruby’s readability. Discourse keeps performance under continuous observation, and Saffron describes budgets for dependencies, boot time, and high-profile pages, including alerts for query-count regressions. He identifies memory leaks as especially difficult to diagnose, describes bisecting an application to isolate a V8-Ruby interop leak, and says MRI remains the only feasible runtime for Discourse while memory profiling and analysis tooling remain substantially behind Java and .NET.
Grab's Trust/Identity/Safety team built Griffin, an in-house anti-fraud risk rule engine designed to process billions of daily predictions across multiple business verticals. Initially, Grab managed fraud rules directly within backend service code, but escalating rule complexity, tight rule interdependencies, and translation gaps between data scientists and developers caused deployment delays and misfiring errors. To overcome the limitations and steep learning curves of third-party engines like Drools, the team separated the workflow into data orchestration and rule-based prediction. Griffin enables analysts and data scientists to author Python-based rules directly via a web portal and reload updated logic into memory without manual developer intervention. By eliminating I/O during rule evaluation and leveraging Gunicorn multi-processing, the engine handles over 100,000 queries per second at peak on six EC2 instances with single-prediction latencies under six milliseconds.
Grab's Trust Platform team built the Counter service to detect fraud across business verticals like transportation, food, and payments. The platform replaces manual, multi-week engineering cycles with a self-service UI where data analysts can define and experiment with counters independently. Operating on an asynchronous ingestion and synchronous transaction model, the architecture evaluates incoming stream data, enriches it via internal services, and persists aggregated signals to ScyllaDB through Grab-Stats. A multi-bucket strategy partitions queries into fifteen-minute, hourly, and daily granularities to maintain low-latency query aggregations across wide time ranges under strict SLAs.
Shopify’s guide presents performance advice for Ruby on Rails across Active Record, Rails, and Ruby, while treating speed as a feature rather than the first optimization priority. It recommends understanding Active Record’s lazy query execution, selecting fewer columns, avoiding unindexed queries, using safe indexing approaches for large tables, and treating query cache as short-lived rather than dependable. For Rails applications, it covers caching, throttling expensive or abusive operations, moving long-running work into Active Job-backed queues, and reducing dependency growth to limit boot time and memory use. Ruby-specific guidance includes limiting metaprogramming and indirection, choosing O(1) hash lookups over O(n) array searches when appropriate, and reducing allocations while avoiding harmful global mutation; benchmark figures show method-definition and invocation costs can differ.
Shopify’s team examined UI performance in the Marketing section of the Shopify admin, aiming to improve load time, perceived load time, and interactions after navigation. They tested the experience on a low-end Moto G3, profiled browser activity with Chrome Developer Tools and React DevTools, and used merchant workflows to reorder priorities. The work reduced unnecessary initial work by memoizing repeated date-formatting tasks, deferring the nonessential ImagePicker bundle, and prefetching that component and its data when a merchant hovered over its activator; memoization cut one component’s render time from about 64.7 ms to 0.5 ms. For loading states, the team recommends stable skeletons with static headings and immediately useful controls, such as the Create campaign button, so merchants can act before unrelated data arrives.
Incremental page-number pagination becomes increasingly expensive for large offsets, because the database scans and discards earlier records before returning a page. Shopify describes relative cursor pagination, which remembers the last record seen and continues from that point, using an indexed id to keep query time nearly constant as pages increase. In tests on a shop with 14 million products, a 100,000 offset took 2,221.60 ms, compared with 5.24 ms using the last id, while offsets of 1,000,000 and above consistently timed out. Sorting by another field requires retaining both its last value and a unique secondary key such as id, with a matching compound index to avoid skipped records and maintain performance. Shopify introduced Link-header URLs for next and previous pages and began migrating REST endpoints from page numbers, reporting early /admin/products.json usage as about 11 times faster, though adoption remained low and full results were pending.
Grab needed a scalable logging platform to replace slow, fragmented systems that hindered debugging across their growing service fleet. Generating 25TB of daily logs, the team built a horizontally scalable Elasticsearch cluster configured via Ansible and monitored with Datadog. Although the initial proof of concept assigned all node roles (ingest, coordinator, master, and data) to every machine, operating at scale introduced major challenges with JVM heap exhaustion and cluster stability. The team resolved memory pressure and performance bottlenecks by tuning circuit breakers, lowering field data cache limits, adjusting shard allocations based on segment memory, and disabling translog compression during shard transfers.
Transport demand spikes during regular commuting hours often outpace driver availability, resulting in passenger wait times and fare surges. To mitigate these imbalances, Grab created the Travel Trends Widget for its mobile feed to redistribute ride requests toward off-peak windows. The widget uses machine learning forecasting to present historical supply-demand patterns alongside pricing trends for the upcoming two hours. To handle anticipated high query rates across millions of database entries, engineers periodically load precomputed trend data into an in-memory data structure rather than querying the database per request. The feature rolled out to feeds in Singapore and Jakarta within four weeks of initial development.
Controlled online experimentation across diverse product verticals requires tracking interactions across systems to prevent local optimizations from causing global degradation. Grab built a batch data pipeline to capture, ingest, and process petabytes of event data to support its experimentation platform and analytics stakeholders. The architecture loads ingested event data from Amazon S3, transforms and sorts it, and writes partitioned output back to S3 with metadata registered in Apache Hive. Using Apache Spark on AWS Elastic MapReduce with Apache Airflow for orchestration, the system handles roughly 400,000 incoming events per second. The data is partitioned by event type and ingestion time and stored in Apache ORC format to streamline query workloads and reduce retrieval overhead.
Ruby 2.6 introduces an initial Just-In-Time (JIT) compiler, making the feature available through the --jit option and on Heroku. The implementation follows Takashi Kokubun’s YARV-MJIT approach, retaining Ruby’s existing YARV instructions instead of replacing them with the proposed RTL intermediate representation; it starts a compiler process at startup and supports GCC, Clang, and Microsoft Visual C++. On Heroku, applications can select Ruby 2.6 in the Gemfile and enable JIT through a Procfile flag or RUBYOPT, although setting RUBYOPT with heroku config is not yet supported. The optcarrot benchmark showed a reported 1.8x speed increase over Ruby 2.5, while Rails and Sidekiq were slower because compilation costs and roughly 2MB per compiled method create overhead. JIT compaction reduces memory use near the 1,000-method cache limit, but the source says further work is needed before JIT becomes the default.
While preparing for Black Friday Cyber Monday (BFCM), Shopify’s Products team found that historical reporting records were causing MySQL to scan many InnoDB pages to return a single record. An intern built a Rails background job to delete the records safely, using Shopify’s job-iteration library to process small batches and an internal throttling enumerator to pause when database health or replication lag became a concern. Indexed fetch queries and cursor-based batching avoided repeated scans and timeouts, while the iterator deleted each batch. Run about two weeks before BFCM, the job deleted more than 5 billion records, peaked at approximately six million records per minute, and released three terabytes of storage. Afterward, the database scanned around three times fewer pages per returned record, improving resilience to flash-sale traffic and producing large cost savings.
Shopify’s infrastructure uses Kubernetes to serve millions of requests per minute, and its growth exposed ingress performance problems during frequent application deployments. After replacing Google Cloud Load Balancer Controller (glbc) with endpoint-aware ingress-nginx, the Edgescale team focused on reducing the NGINX reloads that caused latency increases. They added dynamic configuration: the controller POSTs endpoint lists to an NGINX HTTP endpoint, where lua-nginx-module updates an in-memory shared dictionary instead of regenerating the configuration file and reloading NGINX. Reload avoidance preserves keepalive connections and load-balancing state while reducing the resource effects of replacing worker processes. Tests found ingress-nginx outperformed glbc by multiple orders of magnitude at the 99.99th latency percentile or higher, while dynamic configuration produced its largest gains during deploys, especially beyond the 80th percentile.
Grab Senior Data Scientist Dr. Liuqin Yang, Professor Defeng Sun, and Professor Kim-Chuan Toh received the 2018 Beale-Orchard-Hays Prize for their research paper introducing SDPNAL+. The software employs a majorised semismooth Newton-CG augmented Lagrangian method to solve large-scale semidefinite programming problems with nonnegative constraints. While traditional methods struggled beyond matrix dimensions of 2,000 and 5,000 constraints, SDPNAL+ successfully scales to matrix dimensions of 9,261 and over 12 million constraints. In benchmark testing, the software solved a problem on a desktop PC in 1.5 hours that required 122 hours on a 56-core CPU and 128-GPU cluster using a traditional solver. Grab implements these optimisation techniques to accelerate its passenger-driver allocation algorithms by hundreds of times.
Ruby programs are compiled into YARV instructions and executed by a stack-oriented virtual machine, while register-oriented representations can expose more opportunities for compiler optimization. The post explains how YARV represents a simple expression, contrasts stack and register architectures, and introduces Vladimir Makarov’s Ruby RTL, an intermediate representation distinct from GCC’s RTL. MJIT uses RTL generated by MRI to make speculative, operand-specific optimizations, then compiles methods to machine code rather than relying only on interpretation. The interview discusses deoptimization: when assumptions change, MJIT generates a new machine-code version and can switch between JITed execution and RTL interpretation. The participants say this overhead should be tiny and should not degrade performance overall, while acknowledging edge cases where MJIT is not faster and that benchmarking JITs is difficult.
Shopify’s growth from 375,000 to over 600,000 merchants coincided with about 450,000 S3 operations per second and rising error rates plus read and write latency. Merchant-uploaded images and theme installations exposed a partitioning problem: S3 groups keys by shortest common asset prefixes, and abrupt throughput spikes could trigger SlowDown failures across a range of shops. Black-box observations suggested affected keyspaces were unlikely to recur, while newer shops were more often involved, though the latter was explicitly described as a shaky inference. Shopify prepended a hash-generated digest to new shops’ asset keys, distributing writes across more partitions instead of concentrating them under shop-specific prefixes. After the first post-change SlowDown event, exceptions nearly disappeared, while median latency for digested assets fell about 60% and the 95th percentile about 25%.
Following a production outage where a database failover caused a Ruby on Rails application to exhaust its Puma server threads, an investigation was conducted to understand how ActiveRecord and MySQL timeout settings behave. A reproduction environment using Docker, Puma, and Toxiproxy replicated how hanging requests to a failing database consume all available server threads, ultimately starving unrelated endpoints. The analysis breaks down ActiveRecord connection pooling mechanics alongside underlying mysql2 and libmysqlclient settings, specifically checkout_timeout, connect_timeout, and read_timeout. Testing confirmed how existing and new TCP connections transition through socket states during network interruptions while waiting on configured timeout intervals.
The best engineering reads, once a week.
