Loading…
Latest reads
Every post, summarised
Grounded summaries of engineering blogs, newest first. Each links to the original.
- 4,637 posts
- 20 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Latest posts
Migrating Existing Datastores
Grab's Identity team faced imminent memory exhaustion on a single Redis node used to cache mobile authentication tokens under rapid user growth. Because read traffic outweighed write traffic by roughly 200 times, the team opted for an AWS ElastiCache cluster with three shards and two read replicas per shard. They executed a zero-downtime, six-phase migration plan while handling a peak load of 20,000 queries per second. The migration transitioned through initial one-time data replication, asynchronous shadow writes, synchronous dual writes, asynchronous read validation, switching primary reads, and final write cleanup. Controlled by feature flags and monitored with metrics at every stage, the migration completed without invalidating tokens or causing service disruptions.
So You Need to Hire Good Engineers
Hiring strategies in fast-growing technology startups dictate the quality and scalability of expanding engineering teams. Insights gathered from conversations with over one hundred engineering leaders identify crucial evaluation criteria across technical competence, cultural alignment, and problem-solving capability. Candidates rarely present a perfect technical fit, requiring hiring managers to weigh passion, willingness to learn, and risk appetite against baseline technical requirements. Rather than focusing solely on static factual questions, interviewers gain deeper insights by probing the rationale behind candidates' architectural and implementation choices. High-caliber engineers ultimately elevate team capability by simplifying complex challenges, writing cleaner code, and learning quickly.
Come and #hackallthethings at Grab
Grab has officially launched a public bug bounty program in partnership with HackerOne to strengthen the security of its platform. This rollout follows a private bounty initiative operated over the previous year, during which the organization worked with over 350 security researchers and resolved nearly 200 awarded bug reports. The new public program invites external researchers to scrutinize Grab's code for critical flaws, including remote code execution, SQL injections, and exportable cross-site scripting vulnerabilities. To support ethical and responsible disclosure, Grab offers payouts reaching up to $10,000 per valid vulnerability report based on severity and impact.
How We Scaled Our Cache and Got a Good Night's Sleep
Growing business load on the Common Data Service (CDS) created potential bottlenecks for its single-threaded Redis cache on ElastiCache, necessitating horizontal scaling for greater capacity and throughput. After ruling out master-slave replication and intermediate Twemproxy setups due to memory constraints and proxy I/O bottlenecks, the team implemented client-side sharding. Using an internal Go package for consistent hashing, CDS instances hash cache keys locally to determine the target shard. The implementation encapsulates hashing inside a thin `ShardedCache` wrapper sharing the original cache interface while supporting Ketama and custom hash functions. Deploying via double-writing cron jobs during off-peak hours reduced database read pressure and improved P99 latency.
Grab's Front End Study Guide
Rapid hiring and engineering expansion created a steep learning curve for newcomers and backend engineers unfamiliar with modern web architecture. Grab created an opinionated front-end study guide to help engineers navigate JavaScript tooling, state management, and modern application practices. The guide frames client-side single-page applications as a scalable alternative to traditional server-side rendering, enabling clear client-server separation for multiple client apps hitting the same backend API. It establishes core competencies in ECMAScript 2015 via Babel and explains how component-driven interfaces in React improve maintainability and performance through virtual DOM reconciliation. Ultimately, the resource aims to ramp up developer onboarding and accelerate shipping speeds across web platforms.
DNS Resolution in Go and Cgo
Go applications experiencing load balancing issues across AWS Elastic Load Balancer (ELB) nodes trace uneven traffic distribution to IP address sorting defined in RFC 6724. Comparing Go's native DNS resolver with Cgo and glibc's getaddrinfo shows that both initially sort destination addresses using Rule 9 longest matching prefix rules. Disabling IPv6 on the network interface causes C and Cgo resolvers to return IP addresses in randomized order, while the native Go resolver continues deterministic sorting. Examination of net/addrselect.go reveals that Go's native resolver implements only a subset of the RFC rules and omits dynamic source address selection. Achieving permanent parity requires modifying the Go source code directly.
Driving Southeast Asia Forward with AWS
Grab transitioned its transportation platform from a single Ruby on Rails monolith on Amazon EC2 and Amazon RDS MySQL to a microservices architecture hosted on Amazon Web Services. The platform processes multi-petabyte real-time data flows and hundreds of millions of GPS data points to match drivers with passengers and push proactive demand heat maps. Operational efficiency is sustained with fewer than ten full-time infrastructure engineers, supported by AWS managed services. The analytics backend also transitioned from MySQL to Amazon Redshift, eventually moving to an Amazon S3 data lake using Amazon EMR and Presto. These data-driven matching systems improved driver-passenger allocation rates by up to 30%.
How to Go from a Quick Idea to an Essential Feature in Four Steps
Grab engineered an in-app messaging platform, GrabChat, to help drivers and passengers coordinate pickups across Southeast Asian markets characterized by weak 2G connectivity and high packet loss. The team developed an in-house TCP messaging architecture consisting of a TCP gateway named Gundam and a message dispatcher named Hermes connected to internal backend services over HTTPS. To protect backend server resources from resend loops during poor connection states, the communication protocol adopts a "server only push once" model that delegates retry handling to the client. Data science evaluations using a pre-trained cancellation prediction model confirmed that GrabChat adoption correlated with reduced booking cancellations. Following early usage feedback, the team further iterated on the feature by introducing pre-written message templates to reduce driver distraction on the road.
Troubleshooting Unusual AWS ELB 5XX Error
Grab experienced intermittent HTTP 5XX alerts when its Gothena service sent driver location updates to the Astrolabe service through an AWS Elastic Load Balancer (ELB). CloudWatch metrics revealed that requests were failing to reach healthy backend instances because of an uneven load distribution favoring a single ELB node in one Availability Zone. The team verified that Route 53 was properly using Alias records and ruled out OS-level DNS caching since Linux does not cache DNS queries by default. Connection inspection with netstat across multiple Go services confirmed a heavily skewed distribution of connections toward specific ELB IP addresses. Comparative tests with cURL, tcpdump, Go, Python, and Ruby in an isolated environment demonstrated that Go reused connections across requests while other runtimes opened new connections per request.
Hello RedBeat: A New Celery Beat Scheduler
Heroku Connect relied on Celery Beat to trigger frequent periodic tasks while synchronizing Salesforce and Heroku Postgres data, but the default file-based scheduler developed intermittent pauses and hangs that required manual intervention. Existing SQL-backed scheduling introduced blocked transactions and pauses from periodically saving and reloading the full schedule, while an existing Redis-backed package retained similar reload behavior. RedBeat moves the active schedule and runtime metadata into Redis, allowing updates through Redis keys that Beat reads on its next tick without pausing or reloading. It also uses a lock to prevent concurrent Beat daemons, supports schedule updates from other languages, and has run in production for more than a year without recurring scheduler-related incidents.
Scaling Like a Boss with Presto
Grab experienced severe performance degradation, long queue times, and connection timeouts on its Amazon Redshift analytics cluster as user concurrency and reporting workloads expanded. Although an initial Amazon S3 data lake decoupled storage from compute, business users required standard SQL interfaces rather than Spark data pipelines. The team deployed Presto clusters on AWS EMR, switching their storage format from AVRO to Parquet to support ANSI SQL querying directly against S3. Utilizing a shared Hive metastore on Amazon RDS allowed Grab to adopt a shared-data multi-cluster architecture that isolated distinct workloads across dedicated compute clusters. This setup enabled rapid cluster scaling, streamlined failover, and matched Redshift performance on partitioned time-range queries.
Functional Swift Conference 2017 — Brooklyn
Kickstarter hosted the fourth Functional Swift Conference on April 15th, marking the third time the event had been held at its office. The all-day gathering brought together 10 speakers, including four Kickstarter engineers, and 90 attendees to discuss what functional programming can contribute to better Swift code. Sessions covered recursive enums and KD-trees, testing side-effects, domain-specific languages with GraphQL, algebraic laws, monoids, predicates, sorting functions, applicative functors, parsing, and type inference. Examples included using value types and recursive data structures to describe complicated objects and live-coding an applicative parsing API that generates its own documentation. The closing type-system talk examined why inference is hard and described the Swift compiler team’s use of graph theory to address inference problems practically.
Deep Dive into iOS Automation at Grab - Continuous Delivery
Grab manages continuous delivery for its iOS applications using four build configurations per target: Adhoc QA, Hot Dogfood, Dogfood, and Testflight. The engineering team moved away from Fastlane in favor of custom bash scripts under 100 lines that execute clean builds and archives via xcodebuild. To eliminate manual logins for build execution, Grab developed a server-side Swift application integrating SlackKit and a bot named Iris. Incoming Slack commands are parsed and scheduled onto a serial DispatchQueue to trigger the appropriate build scripts. Deployment and rollback of updates to Iris and its build scripts are handled through Capistrano.
Deep Dive into iOS Automation at Grab - Integration Testing
Grab replaced third-party cloud CI services with an in-house Xcode Server pipeline to run automated iOS integration tests across an expanding engineering team. Instead of performing fresh repository clones on shared cloud instances, Xcode Server pulls code incrementally and mirrors local developer configurations. The team automated testing bots through Ruby scripts and Arcanist hooks tied to Phabricator code reviews, dynamically provisioning bots on diffs and deleting them after merges. This architecture preserves step-by-step UI test execution logs and screenshots for easier failure diagnostics. To maintain stability on on-premises hardware, pre-integration scripts clear application cache states using simctl commands.
Latency Incident Report: Socket Bind Before Connect
An August 11, 2016 incident caused increased routing latency in Heroku’s EU Common Runtime, even though CPU, packet, memory, request, and socket metrics remained normal. The routing layer received EADDRINUSE while connecting to dynos, leading engineers to investigate Linux TCP port allocation, TIME_WAIT, and the Erlang VM’s system calls with strace. They found that the VM called bind() with 0.0.0.0:0 before connect(), forcing Linux to allocate an ephemeral source port without knowing the destination; this made the ephemeral port range an effective per-instance traffic limit and could fail before 65,536 connections. Testing reproduced the failures and showed that setting SO_REUSEADDR before bind(), exposed in Erlang as {reuseaddr, true}, removed the bind errors; the corresponding Vegur patch was deployed a few days later.
A Key Expired in Redis, You Won't Believe What Happened Next
Grab experienced an issue where its Unicorn API served stale data for up to 45 to 60 minutes despite expected cache invalidation times totaling around 11 minutes. The setup utilized ElastiCache Redis 2.x configured with a single master node for writes and two read-only slaves handling reads. Investigation revealed that in Redis 2.x, slave nodes do not expire keys on their own and only delete them upon receiving an explicit DEL command from the master. Because the master only actively checks and deletes 200 random keys per second, clearing expired keys across roughly 5.6 million cached items mathematically required over 110 hours, resulting in slaves serving expired data.
Why you should co-locate your Xcode tests
Xcode projects with unit tests commonly separate implementation and test files into two directories, often mirroring the source structure in the tests directory. The post argues that this arrangement creates maintenance overhead, slows navigation, obscures how well-tested a codebase is when browsed or reviewed, and can leave tests with lower coding standards. It proposes co-locating each unit-test file beside its implementation file while retaining the test target association, accepting more files per directory in exchange for clearer relationships and easier movement between code and tests. The same SwiftLint configuration is used for implementation and test code, treating tests as first-class code. Tests without a single-file correspondence, including integration or UI tests, should remain in separate directories and may warrant separate targets; the authors say their codebase has adopted the style.
Always Be Learning
Kickstarter’s Data team describes how pair learning fits into its broader “always be learning” culture, alongside monthly reading groups open to all employees. Two team members worked through Hadley Wickham’s Advanced R by reading one chapter every two weeks, then meeting for an hour and a half to discuss the material and complete exercises. They found that a steady pace, applying concepts to Kickstarter data, explaining ideas aloud, and documenting lessons in an internal R guide supported sustained learning; the guide includes a demonstration of R’s lazy evaluation. The pair considered the format easier to schedule and harder to neglect than larger groups, while acknowledging fewer perspectives and inclusion challenges, and concluded that it strengthened job-related skills and intellectual curiosity.
How Grab Hires Engineers in Singapore
Grab maintains a selective engineering recruitment process in Singapore, with only three to five percent of candidates ultimately receiving an offer. The evaluation pipeline typically spans three to four weeks and includes an initial HR phone screen, an online coding round, and consecutive technical interviews. Recruiters assess candidate capability via public GitHub projects, presentations, and technical blogs rather than traditional CV qualifications alone. The company prioritizes experience in low-latency distributed systems and microservices design alongside strong cultural alignment with peer collaboration.
How We Found and Fixed a Filesystem Corruption Bug
An aborted Cedar-14 stack-image upgrade led container tools to truncate and overwrite a disk image while it was mounted as the read-only / filesystem for running dynos. Because direct writes bypassed filesystem caching and consistency mechanisms, reads alternated between cached pages from the old image and missing data from the truncated file, partially corrupting newly loaded programs. Applications consequently experienced elevated segmentation faults and other hard crashes for about five hours; a virtual-machine reproduction using an ext4 loop-mounted image and sleep reliably produced the same failure. Engineers migrated affected dynos to fresh hosts, then changed host-side permissions so installed filesystem images are read-only, and retested the conditions that triggered the incident.
Top topics
- Architecture832
- Performance814
- LLMs658
- Machine Learning620
- Open Source563
- AI Agents549
- AI465
- Developer Experience448
Weekly digest
The best engineering reads, once a week.