Loading…
Latest reads
Every post, summarised
Grounded summaries of engineering blogs, newest first. Each links to the original.
- 4,543 posts
- 20 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Latest posts
Tell Your Stories: The Benefits of Strategic Engineering Communications
Shopify created engineering communications in early 2016 as rapid growth and expansion from four to five offices made informal knowledge sharing insufficient. The program serves internal and external audiences through weekly technical talks, an internal newsletter, engineering-blog posts, conference presentations, and an editorial calendar, with Shopifolk developers treated as a primary audience. Communications staff hunt for stories by building relationships with project leads, then coach speakers and work alongside developers while preserving their voices; coaching covers narrative, breath control, body language, and calls to action. The reported effects include less risk of duplicated work or mistakes, cross-pollination of ideas, more confident speakers, increased interest in Shopify from some attendees, and a recurring pipeline in which storytelling generates further stories.
The Data and Science Behind GrabShare Part I: Verifying Potential and Developing the Algorithm
Expanding from point-to-point dispatch services to dynamic carpooling requires matching independent passenger requests traveling in similar directions without causing unacceptable delays. Grab evaluated the feasibility of its GrabShare service by analyzing historical trip data with DBSCAN clustering on coordinates projected into a Universal Transverse Mercator system. This analysis demonstrated that 35% to 46% of rides across typical daytime windows fell into tight geographic clusters with near-identical pickup and drop-off coordinates. The resulting matching framework adapts the baseline dispatch flow by searching for in-transit drivers and enforcing real-time seat reservation constraints. Route assignment decisions subsequently evaluate detour times, trip angles, and expected arrival times to ensure driver utilization improves while total driving distance decreases.
How Shopify Governs Containers at Scale with Grafeas and Kritis
Shopify describes using Grafeas and Kritis to govern container software across a platform handling more than 6,000 container builds per day and 330,000 images in its primary registry. Grafeas serves as a central store for metadata about container origins, builds, base images, layers, registry pulls, installed packages, vulnerabilities, and deployment status, supporting auditing and lifecycle strategies. During CI, audits produce attestations, which Shopify signs with PGP; Kritis uses named attestation authorities and Kubernetes policies to require specific attestations before deployment. Policies can require containers to be built by Shopify, sourced from a trusted repository, free of newly introduced vulnerabilities, tested, non-root, and deployed with the appropriate security context. Shopify also generates attestations from kubeaudit results for workload manifests, helping catch runtime-security regressions before deployment.
The Art of Hiring Good Engineers
Building and scaling engineering teams requires adapting recruitment strategies across different organizational sizes, from initial hires to hundreds of engineers. Teams often begin with small groups of battle-tested engineers before designing formal hiring pipelines. The recruitment approach outlines four key steps: sourcing top-tier talent from platforms like GitHub and LinkedIn, assessing technical fit through tailored screens or assignments, evaluating cross-functional and culture fit via group interviews, and optimizing the process. This optimization includes routing candidates strategically to strict bar-raiser interviewers to prevent burnout and expedite decisions.
In the Cloud, No One Can Hear Your OutOfMemoryError
Heroku introduces Heroku Exec and Language Runtime Metrics to improve production monitoring, inspection, and debugging for applications running on dynos. Heroku Exec creates secure TCP and SSH tunnels for SSH sessions, port forwarding, remote debugging, and diagnostic tools, while the Heroku Java CLI integrates jmap and VisualVM. A synthetic Java memory-leak example shows VisualVM tracking heap growth and jmap producing a downloadable HPROF heap dump that Eclipse Memory Analyzer identifies as Servlet-held FakeLeak objects. The metrics feature, enabled through the Heroku Metrics buildpack on paid Standard-or-above dynos, displays heap, non-heap, and garbage-collection activity in the dashboard through a Java agent using Prometheus; JVM-language support is generally available, while Go support is in beta.
Evolution of the Heroku CLI: 2008-2017
Between 2008 and 2017, Heroku's CLI evolved from Ruby (v1–v3) through Go/Node versions v4 and v5 to a pure Node v6 implementation. The rewrite addressed slower Windows performance, separation of the CLI's runtime from users' Ruby environments, migration to API v3, and limitations in Ruby-based plugin dependencies. V4 kept existing command syntax and output while adding features such as non-TTY color suppression and --json output; its Go core used Node for plugins, and the CLI became a collection of core plugins that could also be developed and installed independently. The team migrated commands incrementally, then added Flow, Babel, and improved tests, including Jest tests that run the flag parser. The resulting foundation supports modular, tested feature development, with cli-engine and more interactive interfaces planned.
Migrating Existing Datastores
Grab's Identity team faced imminent memory exhaustion on a single Redis node used to cache mobile authentication tokens under rapid user growth. Because read traffic outweighed write traffic by roughly 200 times, the team opted for an AWS ElastiCache cluster with three shards and two read replicas per shard. They executed a zero-downtime, six-phase migration plan while handling a peak load of 20,000 queries per second. The migration transitioned through initial one-time data replication, asynchronous shadow writes, synchronous dual writes, asynchronous read validation, switching primary reads, and final write cleanup. Controlled by feature flags and monitored with metrics at every stage, the migration completed without invalidating tokens or causing service disruptions.
So You Need to Hire Good Engineers
Hiring strategies in fast-growing technology startups dictate the quality and scalability of expanding engineering teams. Insights gathered from conversations with over one hundred engineering leaders identify crucial evaluation criteria across technical competence, cultural alignment, and problem-solving capability. Candidates rarely present a perfect technical fit, requiring hiring managers to weigh passion, willingness to learn, and risk appetite against baseline technical requirements. Rather than focusing solely on static factual questions, interviewers gain deeper insights by probing the rationale behind candidates' architectural and implementation choices. High-caliber engineers ultimately elevate team capability by simplifying complex challenges, writing cleaner code, and learning quickly.
Come and #hackallthethings at Grab
Grab has officially launched a public bug bounty program in partnership with HackerOne to strengthen the security of its platform. This rollout follows a private bounty initiative operated over the previous year, during which the organization worked with over 350 security researchers and resolved nearly 200 awarded bug reports. The new public program invites external researchers to scrutinize Grab's code for critical flaws, including remote code execution, SQL injections, and exportable cross-site scripting vulnerabilities. To support ethical and responsible disclosure, Grab offers payouts reaching up to $10,000 per valid vulnerability report based on severity and impact.
How We Scaled Our Cache and Got a Good Night's Sleep
Growing business load on the Common Data Service (CDS) created potential bottlenecks for its single-threaded Redis cache on ElastiCache, necessitating horizontal scaling for greater capacity and throughput. After ruling out master-slave replication and intermediate Twemproxy setups due to memory constraints and proxy I/O bottlenecks, the team implemented client-side sharding. Using an internal Go package for consistent hashing, CDS instances hash cache keys locally to determine the target shard. The implementation encapsulates hashing inside a thin `ShardedCache` wrapper sharing the original cache interface while supporting Ketama and custom hash functions. Deploying via double-writing cron jobs during off-peak hours reduced database read pressure and improved P99 latency.
Grab's Front End Study Guide
Rapid hiring and engineering expansion created a steep learning curve for newcomers and backend engineers unfamiliar with modern web architecture. Grab created an opinionated front-end study guide to help engineers navigate JavaScript tooling, state management, and modern application practices. The guide frames client-side single-page applications as a scalable alternative to traditional server-side rendering, enabling clear client-server separation for multiple client apps hitting the same backend API. It establishes core competencies in ECMAScript 2015 via Babel and explains how component-driven interfaces in React improve maintainability and performance through virtual DOM reconciliation. Ultimately, the resource aims to ramp up developer onboarding and accelerate shipping speeds across web platforms.
DNS Resolution in Go and Cgo
Go applications experiencing load balancing issues across AWS Elastic Load Balancer (ELB) nodes trace uneven traffic distribution to IP address sorting defined in RFC 6724. Comparing Go's native DNS resolver with Cgo and glibc's getaddrinfo shows that both initially sort destination addresses using Rule 9 longest matching prefix rules. Disabling IPv6 on the network interface causes C and Cgo resolvers to return IP addresses in randomized order, while the native Go resolver continues deterministic sorting. Examination of net/addrselect.go reveals that Go's native resolver implements only a subset of the RFC rules and omits dynamic source address selection. Achieving permanent parity requires modifying the Go source code directly.
Driving Southeast Asia Forward with AWS
Grab transitioned its transportation platform from a single Ruby on Rails monolith on Amazon EC2 and Amazon RDS MySQL to a microservices architecture hosted on Amazon Web Services. The platform processes multi-petabyte real-time data flows and hundreds of millions of GPS data points to match drivers with passengers and push proactive demand heat maps. Operational efficiency is sustained with fewer than ten full-time infrastructure engineers, supported by AWS managed services. The analytics backend also transitioned from MySQL to Amazon Redshift, eventually moving to an Amazon S3 data lake using Amazon EMR and Presto. These data-driven matching systems improved driver-passenger allocation rates by up to 30%.
How to Go from a Quick Idea to an Essential Feature in Four Steps
Grab engineered an in-app messaging platform, GrabChat, to help drivers and passengers coordinate pickups across Southeast Asian markets characterized by weak 2G connectivity and high packet loss. The team developed an in-house TCP messaging architecture consisting of a TCP gateway named Gundam and a message dispatcher named Hermes connected to internal backend services over HTTPS. To protect backend server resources from resend loops during poor connection states, the communication protocol adopts a "server only push once" model that delegates retry handling to the client. Data science evaluations using a pre-trained cancellation prediction model confirmed that GrabChat adoption correlated with reduced booking cancellations. Following early usage feedback, the team further iterated on the feature by introducing pre-written message templates to reduce driver distraction on the road.
Troubleshooting Unusual AWS ELB 5XX Error
Grab experienced intermittent HTTP 5XX alerts when its Gothena service sent driver location updates to the Astrolabe service through an AWS Elastic Load Balancer (ELB). CloudWatch metrics revealed that requests were failing to reach healthy backend instances because of an uneven load distribution favoring a single ELB node in one Availability Zone. The team verified that Route 53 was properly using Alias records and ruled out OS-level DNS caching since Linux does not cache DNS queries by default. Connection inspection with netstat across multiple Go services confirmed a heavily skewed distribution of connections toward specific ELB IP addresses. Comparative tests with cURL, tcpdump, Go, Python, and Ruby in an isolated environment demonstrated that Go reused connections across requests while other runtimes opened new connections per request.
Hello RedBeat: A New Celery Beat Scheduler
Heroku Connect relied on Celery Beat to trigger frequent periodic tasks while synchronizing Salesforce and Heroku Postgres data, but the default file-based scheduler developed intermittent pauses and hangs that required manual intervention. Existing SQL-backed scheduling introduced blocked transactions and pauses from periodically saving and reloading the full schedule, while an existing Redis-backed package retained similar reload behavior. RedBeat moves the active schedule and runtime metadata into Redis, allowing updates through Redis keys that Beat reads on its next tick without pausing or reloading. It also uses a lock to prevent concurrent Beat daemons, supports schedule updates from other languages, and has run in production for more than a year without recurring scheduler-related incidents.
Scaling Like a Boss with Presto
Grab experienced severe performance degradation, long queue times, and connection timeouts on its Amazon Redshift analytics cluster as user concurrency and reporting workloads expanded. Although an initial Amazon S3 data lake decoupled storage from compute, business users required standard SQL interfaces rather than Spark data pipelines. The team deployed Presto clusters on AWS EMR, switching their storage format from AVRO to Parquet to support ANSI SQL querying directly against S3. Utilizing a shared Hive metastore on Amazon RDS allowed Grab to adopt a shared-data multi-cluster architecture that isolated distinct workloads across dedicated compute clusters. This setup enabled rapid cluster scaling, streamlined failover, and matched Redshift performance on partitioned time-range queries.
Functional Swift Conference 2017 — Brooklyn
Kickstarter hosted the fourth Functional Swift Conference on April 15th, marking the third time the event had been held at its office. The all-day gathering brought together 10 speakers, including four Kickstarter engineers, and 90 attendees to discuss what functional programming can contribute to better Swift code. Sessions covered recursive enums and KD-trees, testing side-effects, domain-specific languages with GraphQL, algebraic laws, monoids, predicates, sorting functions, applicative functors, parsing, and type inference. Examples included using value types and recursive data structures to describe complicated objects and live-coding an applicative parsing API that generates its own documentation. The closing type-system talk examined why inference is hard and described the Swift compiler team’s use of graph theory to address inference problems practically.
Deep Dive into iOS Automation at Grab - Continuous Delivery
Grab manages continuous delivery for its iOS applications using four build configurations per target: Adhoc QA, Hot Dogfood, Dogfood, and Testflight. The engineering team moved away from Fastlane in favor of custom bash scripts under 100 lines that execute clean builds and archives via xcodebuild. To eliminate manual logins for build execution, Grab developed a server-side Swift application integrating SlackKit and a bot named Iris. Incoming Slack commands are parsed and scheduled onto a serial DispatchQueue to trigger the appropriate build scripts. Deployment and rollback of updates to Iris and its build scripts are handled through Capistrano.
Deep Dive into iOS Automation at Grab - Integration Testing
Grab replaced third-party cloud CI services with an in-house Xcode Server pipeline to run automated iOS integration tests across an expanding engineering team. Instead of performing fresh repository clones on shared cloud instances, Xcode Server pulls code incrementally and mirrors local developer configurations. The team automated testing bots through Ruby scripts and Arcanist hooks tied to Phabricator code reviews, dynamically provisioning bots on diffs and deleting them after merges. This architecture preserves step-by-step UI test execution logs and screenshots for easier failure diagnostics. To maintain stability on on-premises hardware, pre-integration scripts clear application cache states using simctl commands.
Top topics
- Architecture830
- Performance812
- LLMs649
- Machine Learning617
- Open Source561
- AI Agents543
- AI456
- Developer Experience447
Weekly digest
The best engineering reads, once a week.