Loading…
Latest reads
Every post, summarised
Grounded summaries of engineering blogs, newest first. Each links to the original.
- 4,570 posts
- 20 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Latest posts
Handling Very Large Tables in Postgres Using Partitioning
Very large Postgres tables containing application events or log-like data can degrade query performance, slow index updates, and make vacuum disproportionately long. The article explains table partitioning through inheritance, using a master table, child partitions, CHECK constraints, and a trigger that routes inserts according to created_at; for an events workload with 10 million daily inserts and short-lived data, it uses daily partitions. It then introduces pg_partman, whose create_parent command configures time-based partitioning and whose run_maintenance function creates new partitions and removes old ones according to retention settings. Partitioning can improve performance and simplify deletion of expired data, but it should follow indexing and usage analysis, especially when queries cover a small time range and older data has little value.
Why Test the Design with Only 5 Users
Stakeholders often question whether qualitative testing with only five participants yields reliable design findings. The chosen research problem dictates the methodology, with qualitative usability evaluations requiring smaller cohorts than broad quantitative surveys. Jakob Nielsen's 1993 analysis demonstrated that testing with five participants uncovers 75 to 80 percent of usability issues, after which additional users yield diminishing returns. In agile environments, teams maximize efficiency by iterating designs across successive rounds of five participants rather than testing large cohorts simultaneously. This five-user threshold applies strictly to qualitative usability testing for specific feature flows rather than opinion gathering, A/B testing, or multi-segment user analysis.
Programmers Beware - UX is Not Just for Designers
Software engineers frequently overlook user experience when designing APIs, SDKs, and code-level functions. Usability issues arise across various technical interfaces, from mobile apps forced to coordinate multiple round-trip network calls to ambiguous function signatures that obscure boolean arguments. To counter this, engineers can apply a five-question discovery framework to identify user identity, core objectives, user capabilities, ways to reduce user burden, and familiar paradigms. Practical remedies include sacrificing strict RESTful separation to merge mobile endpoints, placing validation logic inside internal RPC servers, and replacing boolean arguments with explicitly named helper functions. Ultimately, treating calling systems, end users, and fellow programmers as users shifts implementation complexity from consumers to servers.
Apache Kafka 0.10 Evaluating Performance in Distributed Systems
Testing Apache Kafka 0.10 revealed an apparent 33% throughput decline versus Kafka 0.9, prompting an investigation into whether the regression affected different plans, connection types, message sizes, and workload scales. Heroku’s benchmark system used multiple producer and consumer dynos, while the team applied Brendan Gregg’s USE method to examine resource utilization, saturation, and errors. The diagnosis was network-card saturation: Kafka 0.10 adds an 8-byte timestamp to every message, which at 1.5 million messages per second and replication factor 3 produced 288 Mbps of additional cluster traffic, causing packet drops and retransmits. Reproducing the effect by adding 8 bytes to Kafka 0.9 confirmed the explanation, and the post says most production workloads are unlikely to be affected, though clusters near network limits may be.
Heroku Metrics
Heroku’s original Dashboard Metrics system stored 10-minute rollups in a 21-shard InfluxDB cluster, but new requirements for one-minute data, richer events, extensibility, and failure tolerance exposed infrastructure limits. To replace it, the team built a Kafka-based pipeline on Private Spaces: an ingress app parses Logplex frames into Protocol Buffers messages, Go consumers aggregate measurements into one-minute summaries, and a sink writes rolled-up data to seven sharded, partitioned Heroku Postgres databases. The custom Yahtzee framework uses accumulator maps for counts and streaming min, max, and sum calculations with deterministic partition ownership; late data and aggregation-process restarts were implementation hurdles. The design reduces reliance on expensive InfluxDB continuous queries and makes adding another datastore straightforward, although Postgres performance remains merely acceptable and measurements can still be lost before Kafka because the logging pipeline is lossy by design.
Incident Preparation: Simulating Third-Party Downtime
Heroku’s help application depends on the Platform API for authentication, authorization, and user data, so it must remain available to agents and customers during platform incidents. The open-sourced api-maintenance-sim app tests this requirement by imitating an outage in which every request receives an HTTP 503 Service Unavailable response. After deploying the simulator, teams point the application under test to its hostname through PLATFORM_API_URL, passed to platform-api’s connect_oauth method; when unset, the gem uses the production URI. A heroku config:set command enables the alternate endpoint, making regular simulations possible and helping reveal what failure means for the application or service when the Platform API is unavailable, supporting preparation for incidents and communication during outages.
Speeding Up Sprockets for Faster Asset Pipelines
The Rails asset pipeline can take over 20 times longer than dependency installation because Sprockets consumes memory, disk I/O, and CPU while reading, fingerprinting, compressing, and resolving dependent assets. Reusing its disk cache failed across Capistrano releases and Heroku builds because cache keys and dependency data embedded absolute paths, so identical files appeared different when compiled from changing directories. Sprockets 3.3+ addresses this with UnloadedAsset, normalized relative cache keys, and URITar, which compresses paths and URIs before caching and expands them when loading; Windows portability was fixed in 3.3.4. In an example app, the initial compile remained about 38 seconds, while the second took 3 seconds, a roughly 1,200% speed increase when compiled assets were reused.
Grab You Some Post-Mortem Reports
Grab describes how its service-oriented architecture makes cross-team production debugging difficult because engineers may lack familiarity with other teams’ code and architecture. Post-mortem reports are presented as the glue for shared understanding, but a sample incident entry lacks service context, bug detail, impact, chronology, and investigation steps. The proposed standard has four requirements—Chronology, Context, Empowerment, and Solutions—covering timelines, service internals, blameless educational diagnosis, and remedies across People, Product, and Process. A template includes initial symptoms, a timestamped timeline, log-based investigation with code evidence, temporary and permanent fixes, and improvement actions, while peer review by another team is required before finalization.
The Curious Case of the Phantom Instance
Datadog dashboards for the grab_attention cluster displayed periodic 1.5X step increases in Elastic Load Balancer (ELB) health check requests and ElastiCache Redis connections, creating the illusion of an untracked instance running outside Auto Scaling Group records. Inspecting instance hostname tags revealed that the existing two instances were simply receiving elevated ping counts from the load balancers. AWS Support clarified that ELB scaling events provision new nodes while keeping old nodes running for roughly 90 minutes to handle cached DNS clients. Investigation also revealed that two separate ELBs were attached to the cluster, altering expected request baselines. Furthermore, the application's health check endpoint initiated a non-pooled Redis connection on every request, directly translating load balancer pings into database connection spikes.
Introducing Heroku’s React Refetch Operations Console
Heroku built an operations console for Private Spaces to provide a bird’s-eye view of the platform and detailed investigation of individual spaces. It is a single-page React application backed by a reverse proxy, but its constituent applications originally used jQuery state, mixins, or classic Flux. The team considered Redux, then rejected a global store for server-rendered data because application state was already represented in the URL. React Refetch instead maps component props to remote requests, injects immutable PromiseState props, and recalculates requests when URL parameters change. Its declarative model removes stores, callbacks, actions, reducers, and switch statements while supporting composed and chained requests, periodic refresh, custom headers and methods, lazy loading, and server writes; the team presents it as a simpler alternative when client-side state need not persist.
How We Migrated to Active Record 4
Heroku Platform API had remained on Active Record 2.3.18, released in March 2013, creating maintenance and security burdens as support ended and custom query infrastructure replaced newer features. The team moved first to Active Record 3.0.20, preserving 2.x APIs, then directly to 4.2.4, using a feature branch, console-driven fixes, automated and acceptance tests, private deployments, staging, and a partial production rollout. The first step took one engineer about two weeks, while the second took two engineers about two weeks; deployment had no downtime but briefly required a rollback for an acts_as_paranoid bug. Performance improved 18% from 2.3.18 to 3.0.20 but worsened 13% from 3.0.20 to 4.2.4, challenging micro-benchmark expectations. The upgrade ultimately left performance at the same overall level, removed patches, restored supported status, and made future updates and training easier.
Heroku Proxying becomes Free Software
Heroku’s HTTP routing stack has three layers: state synchronization, request routing, and HTTP proxying. The platform team open-sourced the proxying layer as Vegur, an Erlang library that validates, normalizes, and forwards requests between clients and dynos. Unlike a conventional nginx reverse proxy, Vegur’s hosts and configuration are fully programmable, allowing state from hardcoded values, files, databases, or direct updates, while routing algorithms and debugging headers can also remain dynamic. Its delegated capabilities include domain lookup, backend selection, interface choice, resource checkout and checkin, customizable failure pages, per-application or per-backend controls for tunneling 100-continue requests, and upstream or downstream header insertion. Vegur had run in production on Heroku for over a year and powered the Heroku Private Spaces router.
We Patched Ruby On Rails for a 10% Performance Bump
Rails response time improved by more than 10% in a patch that also reduced allocated objects by 29%, motivated by memory and performance problems observed in production Heroku applications. The work began with customer reports of R14 out-of-memory errors, swap usage, and suspected memory leaks, then used a customer app and reusable benchmarks to reproduce issues. The derailed benchmarks perf:objects task and memory_profiler exposed allocation hot spots, including arrays joined into strings and duplicated hashes; refactoring reduced unnecessary objects, while routing changes produced a 31% speed increase for constrained URL generation. The resulting savings required no application changes and were planned for the next Rails version, with Rails 5 pre-release testing mentioned.
Preparing Our Response to the OpenSSL Security Flaw
Heroku prepared for an announced high-severity OpenSSL vulnerability whose details were embargoed until patch release. The team began by mapping exposure and system criticality, considering SSL clients, possible man-in-the-middle attacks, encrypted database connections, and worst-case remote code execution, then created stack-image update actions and priorities. Engineers were assigned responsibilities under the existing incident-response process, with communication rules, escalation criteria, paging, and availability planned in advance. When the OpenSSL update and vulnerability details arrived on July 9 at about 8am Central, triage showed it was neither an RCE nor server-side issue, and Ubuntu reported that neither LTS version was affected; because Heroku production used LTS, no remediation or all-hands response was needed. The preparation still exercised exposure review and response coordination for future incidents.
Improved production stability with circuit breakers
Heroku’s API consumes more service endpoints than it serves, making its availability dependent on public, internal, and third-party services that can produce tail latency, 500 errors, broken sockets, or bad deploys. To contain these failures, the team wrapped external calls with Redis-backed circuit breakers that count errors over a time window, trip at an error-rate threshold, and support fallback responses or user-friendly exceptions. CB2 provides this implementation, including individually identified breakers, while percentage-based thresholds avoid the scaling problem of fixed error counts. Deployment starts with logging-only breakers so parameters and unexpected internal-API behavior can be reviewed before traffic is affected, followed by monitoring state changes with safeguards against duplicate notifications caused by race conditions. The post also recommends explicit, low network-level timeouts, arguing that slow services can damage stability more than fast failures.
Incremental Garbage Collection in Ruby 2.2
Ruby 2.2 introduces RincGC, an incremental garbage collector intended to reduce the long pause times that Ruby 2.1’s generational collector could produce during major GC. The approach splits marking into fine-grained steps interleaved with Ruby execution, while lazy sweeping remains incremental; because CRuby lacks sufficient write barriers, unprotected objects are rescanned before white objects are reclaimed. The article explains the white, grey, and black object model, the write-barrier safety problem, and uses GC::Tracer events with a Rails benchmark simulating 30,000 requests to measure pauses. In the reported measurement, generational GC produced seven large pauses with a maximum of about 15 ms, while incremental GC reduced the maximum to around 2 ms, without reducing total GC time or improving throughput.
Debugging Super Methods with Ruby 2.2
Ruby exposes method metadata and caller-stack information, but before Ruby 2.2 it was difficult to identify the method ultimately reached by super, especially in metaprogrammed code. The example defines Dog#bark and SchneemsDog#bark, then extends an instance with DoubleBark, whose bark method calls super twice; inspecting the class hierarchy incorrectly reports Dog#bark because the module is not an ancestor of the instance's class. Ruby 2.2 adds Method#super_method, a feature proposed in request #9781 and accepted into the Ruby trunk, which returns the method Ruby will invoke for super, including the correct SchneemsDog#bark in this case. Calling source_location on that returned Method provides the location of the actual implementation and avoids the custom metaprogramming previously needed to trace it.
Short Time Outs and Asynchronous Requests on Heroku
Heroku’s short request timeout and H12 errors reflect a scaling problem: standard web servers process queued requests serially, so one long operation can block others. The post separates slow server-side actions, such as video processing or PDF generation, from file uploads, and recommends background processes for the former and direct browser uploads to Amazon S3 or Google Cloud Storage for the latter. When a client must wait for completion, it describes polling or a WebSocket, noting that WebSocket connections close after 55 seconds of inactivity and ordinary pages still need to respond within 30 seconds. Avoiding lengthy web-process work improves performance and scalability; rack-timeout for Ruby or expressjs’ timeout for Node can terminate requests after a few seconds, with a recommended limit no higher than 10 seconds.
Benchmarking Rack Middleware and Improved Speed
ActionDispatch::Static, enabled by config.serve_static_assets = true, checks incoming Rails requests to decide whether to serve a disk file or pass the request onward. To measure the middleware in isolation, the author loads it directly, wraps it around a no-op Proc, and uses Rack::MockRequest with benchmark-ips to compare requests against the no-op baseline. The original middleware achieved 15,891.2 iterations per second, versus 28,660.9 for the no-op, while the modified version reached 24,490.9 iterations per second. That change made non-asset middleware requests roughly 54% faster, bringing the gap to the no-op case to about 10%, although asset requests became slightly slower and the difference was not measurable. Across the full Rails application stack, the reported overall speedup was approximately 2.6%.
Django and Node together on Heroku
Django-based Heroku Connect needed realtime dashboard support through socket.io and Node.js, but Heroku apps could route traffic through only one dyno type. The team chose to run both stacks in one dyno using a runit buildpack and its sub-Procfile format: Node binds to PORT, while node-http-proxy forwards normal web requests to Gunicorn-hosted Django on 127.0.0.1:$DJANGO_PORT. This arrangement increases contention because processes share memory, CPU, and a filesystem, and it makes dyno logs less directly attributable; the team also flattened Procfiles into Procfile.local for Foreman-based local development. After running the architecture in production for a few months without issue, the team reported minimal proxy overhead and retained single-git-push deployment while establishing a path for websocket communication.
Top topics
- Architecture830
- Performance812
- LLMs655
- Machine Learning619
- Open Source562
- AI Agents546
- AI458
- Developer Experience447
Weekly digest
The best engineering reads, once a week.