Loading…
Latest reads
Every post, summarised
Grounded summaries of engineering blogs, newest first. Each links to the original.
- 4,543 posts
- 20 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Latest posts
Latency Incident Report: Socket Bind Before Connect
An August 11, 2016 incident caused increased routing latency in Heroku’s EU Common Runtime, even though CPU, packet, memory, request, and socket metrics remained normal. The routing layer received EADDRINUSE while connecting to dynos, leading engineers to investigate Linux TCP port allocation, TIME_WAIT, and the Erlang VM’s system calls with strace. They found that the VM called bind() with 0.0.0.0:0 before connect(), forcing Linux to allocate an ephemeral source port without knowing the destination; this made the ephemeral port range an effective per-instance traffic limit and could fail before 65,536 connections. Testing reproduced the failures and showed that setting SO_REUSEADDR before bind(), exposed in Erlang as {reuseaddr, true}, removed the bind errors; the corresponding Vegur patch was deployed a few days later.
A Key Expired in Redis, You Won't Believe What Happened Next
Grab experienced an issue where its Unicorn API served stale data for up to 45 to 60 minutes despite expected cache invalidation times totaling around 11 minutes. The setup utilized ElastiCache Redis 2.x configured with a single master node for writes and two read-only slaves handling reads. Investigation revealed that in Redis 2.x, slave nodes do not expire keys on their own and only delete them upon receiving an explicit DEL command from the master. Because the master only actively checks and deletes 200 random keys per second, clearing expired keys across roughly 5.6 million cached items mathematically required over 110 hours, resulting in slaves serving expired data.
Why you should co-locate your Xcode tests
Xcode projects with unit tests commonly separate implementation and test files into two directories, often mirroring the source structure in the tests directory. The post argues that this arrangement creates maintenance overhead, slows navigation, obscures how well-tested a codebase is when browsed or reviewed, and can leave tests with lower coding standards. It proposes co-locating each unit-test file beside its implementation file while retaining the test target association, accepting more files per directory in exchange for clearer relationships and easier movement between code and tests. The same SwiftLint configuration is used for implementation and test code, treating tests as first-class code. Tests without a single-file correspondence, including integration or UI tests, should remain in separate directories and may warrant separate targets; the authors say their codebase has adopted the style.
Always Be Learning
Kickstarter’s Data team describes how pair learning fits into its broader “always be learning” culture, alongside monthly reading groups open to all employees. Two team members worked through Hadley Wickham’s Advanced R by reading one chapter every two weeks, then meeting for an hour and a half to discuss the material and complete exercises. They found that a steady pace, applying concepts to Kickstarter data, explaining ideas aloud, and documenting lessons in an internal R guide supported sustained learning; the guide includes a demonstration of R’s lazy evaluation. The pair considered the format easier to schedule and harder to neglect than larger groups, while acknowledging fewer perspectives and inclusion challenges, and concluded that it strengthened job-related skills and intellectual curiosity.
How Grab Hires Engineers in Singapore
Grab maintains a selective engineering recruitment process in Singapore, with only three to five percent of candidates ultimately receiving an offer. The evaluation pipeline typically spans three to four weeks and includes an initial HR phone screen, an online coding round, and consecutive technical interviews. Recruiters assess candidate capability via public GitHub projects, presentations, and technical blogs rather than traditional CV qualifications alone. The company prioritizes experience in low-latency distributed systems and microservices design alongside strong cultural alignment with peer collaboration.
How We Found and Fixed a Filesystem Corruption Bug
An aborted Cedar-14 stack-image upgrade led container tools to truncate and overwrite a disk image while it was mounted as the read-only / filesystem for running dynos. Because direct writes bypassed filesystem caching and consistency mechanisms, reads alternated between cached pages from the old image and missing data from the truncated file, partially corrupting newly loaded programs. Applications consequently experienced elevated segmentation faults and other hard crashes for about five hours; a virtual-machine reproduction using an ext4 loop-mounted image and sleep reliably produced the same failure. Engineers migrated affected dynos to fresh hosts, then changed host-side permissions so installed filesystem images are read-only, and retested the conditions that triggered the incident.
Battling with Tech Giants for the World's Best Talent
Grab maintains research and development centres across Singapore, Seattle, and Beijing to develop mobility and transaction platforms for Southeast Asia. Engineer Brandon Gao turned down an offer from a major Seattle-based technology corporation to remain at the company due to rapid growth and high engineering impact. During backend infrastructure changes starting in May 2015, Grab transitioned core services from Node.js and Ruby to Golang. A weekend Golang prototype created by Gao evolved into the primary data service linking backend servers to 580,000 drivers across the region. Additionally, his initial work on the User Trust team established a machine-learning risk and fraud detection system supporting large-scale mobile transaction volumes.
Pulling the Thread on Kafka’s Compacted Topics
Apache Kafka’s log cleaner for compacted topics could crash silently when its offset map lacked enough space to hold a full log segment. Because the broker continued running without that internal thread, superseded data accumulated, threatening node and potentially cluster stability. The post explains compaction’s two phases: building an offset map for newer messages, then rewriting segments while retaining the latest offset for each key, with the latest segment excluded from the map. Kafka was changed to remove the crashing assertion and track partial segment loads so cleaning can recover without fitting an entire segment in memory. The roughly 100-line patch shipped in Kafka 0.10.1.1, which the post identifies as Heroku’s default version, and prevents the cleaner from dying silently.
How We Sped up SNI TLS Handshakes by 5x
Heroku’s SNI TLS termination work exposed severe handshake performance problems under benchmarks designed to avoid session resumption, caching, and keep-alive reuse. Initial tests made nodes slow and unresponsive 7–8x earlier than comparable HTTP tests, while perf top showed Erlang garbage collection and ETS hash operations consuming unexpectedly large amounts of CPU. Tracing linked repeated ets.select_delete calls to the centralized ssl_manager process, where ssl_pkix_db’s certificate cache path was still used even for DER certificates supplied directly in memory. Engineers patched the SSL library to bypass that cache wholesale. Each instance then handled three times the benchmark capacity, while SNI latencies fell from 100–120 milliseconds to 20–25 milliseconds for the reported app subset; the patch was later shipped in Erlang/OTP-19.1.
This Rocket Ain't Stopping - Achieving Zero Downtime for Rails to Golang API Migration
Grab transitioned its public passenger app APIs from a legacy Rails application to a Golang service-oriented architecture to consolidate its codebase and engineering teams. Initial attempts to proxy traffic through a cloned Rails server via gRPC were abandoned after encountering TCP load imbalances during autoscaling events and memory leaks in the gRPC Ruby gem. The team pivoted to direct logic migration, porting Ruby logic directly into Go while decomposing modules into standalone services. Verification relied on log-based load testing and live shadow testing, where write operations were safely validated using mock data access layers that evaluated expected database outcomes. Production rollout progressed endpoint-by-endpoint using requests-per-second traffic throttling and prewarmed AWS Elastic Load Balancers before executing the final DNS switch.
Grab Vietnam Careers Week
Grab announced its first Grab Vietnam Careers Week, held in Ho Chi Minh City from 22 to 26 October 2016, aimed at hiring engineers to improve regional transportation and reduce congestion. At the time, Grab operated on 23 million mobile devices supported by 460,000 drivers across Southeast Asia. The announcement features an interview with Singapore-based iOS engineer Hai Pham and Android engineer Son Nguyen discussing team culture, regular hackathons termed "Grabathons", and relocation to Singapore. Technical candidates undergo a Codility test and are evaluated on clean mobile architecture, testing, and performance maintenance. The engineers also offer advice on apartment hunting through online community forums and adapting to life in Singapore.
GrabPay Wins Best Fraud Prevention Innovation at the Florin Awards
Grab won the Best Fraud Prevention Innovation (Community Votes) Award at the 2016 Florin Awards for its GrabPay platform. To support secure cashless transactions across Southeast Asia, Grab deployed a dedicated risk and fraud detection engine when launching GrabPay earlier in the year. The system utilizes machine learning algorithms that continually evolve by analyzing driver, passenger, and travel pattern data across massive transaction volumes. Furthermore, Grab provides full financial protection by covering any unauthorized fraudulent transactions for both passengers and drivers. At the time of the announcement, the platform supported up to 1.5 million daily bookings and exceeded 23 million app downloads.
Round-robin in Distributed Systems
Building client-side load balancing for Grab's Common Data Service prompted a move from AWS Elastic Load Balancers to DNS discovery due to persistent connection issues and unpredictable scaling events. After patching an open-source library that failed to rotate IP sequences properly, the author evaluated different Go patterns for round-robin routing. A mutex-protected array counter provides the simplest model for basic retrieval, though adding mutations requires careful lock coordination. Alternatively, a dedicated balancer goroutine receiving requests over nested channels enables explicit operation timeouts and centralized event handling at the cost of higher code complexity and channel creation overhead. The author recommends the mutex approach for resource fetching and the goroutine-based design for workload balancing.
Handling Very Large Tables in Postgres Using Partitioning
Very large Postgres tables containing application events or log-like data can degrade query performance, slow index updates, and make vacuum disproportionately long. The article explains table partitioning through inheritance, using a master table, child partitions, CHECK constraints, and a trigger that routes inserts according to created_at; for an events workload with 10 million daily inserts and short-lived data, it uses daily partitions. It then introduces pg_partman, whose create_parent command configures time-based partitioning and whose run_maintenance function creates new partitions and removes old ones according to retention settings. Partitioning can improve performance and simplify deletion of expired data, but it should follow indexing and usage analysis, especially when queries cover a small time range and older data has little value.
Why Test the Design with Only 5 Users
Stakeholders often question whether qualitative testing with only five participants yields reliable design findings. The chosen research problem dictates the methodology, with qualitative usability evaluations requiring smaller cohorts than broad quantitative surveys. Jakob Nielsen's 1993 analysis demonstrated that testing with five participants uncovers 75 to 80 percent of usability issues, after which additional users yield diminishing returns. In agile environments, teams maximize efficiency by iterating designs across successive rounds of five participants rather than testing large cohorts simultaneously. This five-user threshold applies strictly to qualitative usability testing for specific feature flows rather than opinion gathering, A/B testing, or multi-segment user analysis.
Programmers Beware - UX is Not Just for Designers
Software engineers frequently overlook user experience when designing APIs, SDKs, and code-level functions. Usability issues arise across various technical interfaces, from mobile apps forced to coordinate multiple round-trip network calls to ambiguous function signatures that obscure boolean arguments. To counter this, engineers can apply a five-question discovery framework to identify user identity, core objectives, user capabilities, ways to reduce user burden, and familiar paradigms. Practical remedies include sacrificing strict RESTful separation to merge mobile endpoints, placing validation logic inside internal RPC servers, and replacing boolean arguments with explicitly named helper functions. Ultimately, treating calling systems, end users, and fellow programmers as users shifts implementation complexity from consumers to servers.
Apache Kafka 0.10 Evaluating Performance in Distributed Systems
Testing Apache Kafka 0.10 revealed an apparent 33% throughput decline versus Kafka 0.9, prompting an investigation into whether the regression affected different plans, connection types, message sizes, and workload scales. Heroku’s benchmark system used multiple producer and consumer dynos, while the team applied Brendan Gregg’s USE method to examine resource utilization, saturation, and errors. The diagnosis was network-card saturation: Kafka 0.10 adds an 8-byte timestamp to every message, which at 1.5 million messages per second and replication factor 3 produced 288 Mbps of additional cluster traffic, causing packet drops and retransmits. Reproducing the effect by adding 8 bytes to Kafka 0.9 confirmed the explanation, and the post says most production workloads are unlikely to be affected, though clusters near network limits may be.
Heroku Metrics
Heroku’s original Dashboard Metrics system stored 10-minute rollups in a 21-shard InfluxDB cluster, but new requirements for one-minute data, richer events, extensibility, and failure tolerance exposed infrastructure limits. To replace it, the team built a Kafka-based pipeline on Private Spaces: an ingress app parses Logplex frames into Protocol Buffers messages, Go consumers aggregate measurements into one-minute summaries, and a sink writes rolled-up data to seven sharded, partitioned Heroku Postgres databases. The custom Yahtzee framework uses accumulator maps for counts and streaming min, max, and sum calculations with deterministic partition ownership; late data and aggregation-process restarts were implementation hurdles. The design reduces reliance on expensive InfluxDB continuous queries and makes adding another datastore straightforward, although Postgres performance remains merely acceptable and measurements can still be lost before Kafka because the logging pipeline is lossy by design.
Incident Preparation: Simulating Third-Party Downtime
Heroku’s help application depends on the Platform API for authentication, authorization, and user data, so it must remain available to agents and customers during platform incidents. The open-sourced api-maintenance-sim app tests this requirement by imitating an outage in which every request receives an HTTP 503 Service Unavailable response. After deploying the simulator, teams point the application under test to its hostname through PLATFORM_API_URL, passed to platform-api’s connect_oauth method; when unset, the gem uses the production URI. A heroku config:set command enables the alternate endpoint, making regular simulations possible and helping reveal what failure means for the application or service when the Platform API is unavailable, supporting preparation for incidents and communication during outages.
Speeding Up Sprockets for Faster Asset Pipelines
The Rails asset pipeline can take over 20 times longer than dependency installation because Sprockets consumes memory, disk I/O, and CPU while reading, fingerprinting, compressing, and resolving dependent assets. Reusing its disk cache failed across Capistrano releases and Heroku builds because cache keys and dependency data embedded absolute paths, so identical files appeared different when compiled from changing directories. Sprockets 3.3+ addresses this with UnloadedAsset, normalized relative cache keys, and URITar, which compresses paths and URIs before caching and expands them when loading; Windows portability was fixed in 3.3.4. In an example app, the initial compile remained about 38 seconds, while the second took 3 seconds, a roughly 1,200% speed increase when compiled assets were reused.
Top topics
- Architecture830
- Performance812
- LLMs649
- Machine Learning617
- Open Source561
- AI Agents543
- AI456
- Developer Experience447
Weekly digest
The best engineering reads, once a week.