Loading…
Latest reads
Every post, summarised
Grounded summaries of engineering blogs, newest first. Each links to the original.
- 4,538 posts
- 20 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Latest posts
Tourists on GrabChat!
Grab examined more than 3.7 million tourist messages across Singapore, Malaysia, and Indonesia sent between December 2018 and March 2019 to evaluate passenger communication patterns. The platform deployed in-house translation and prewritten, auto-translated chat templates to bridge language barriers between international riders and local drivers. Analysis showed that bookings utilizing chat templates experienced a 10% higher ride completion rate than those without. Image-sharing features were most heavily used in high-traffic hubs such as airports, shopping malls, and major tourist centers to aid driver location. Passengers also consistently used messaging to clarify luggage capacity, provide identifiable passenger descriptions, and check pet policies.
Mobile Release Engineering at Scale with Shipit Mobile
Shopify built Shipit Mobile to replace error-prone, manual mobile app releases with a shared platform for creating, viewing, and managing releases. The process creates a release branch from master, builds a candidate for each commit, increments build numbers, and starts separate CI pipelines for testing and release builds. Shipit Mobile uploads the same binary and metadata to distribution channels including Google Play, the App Store, and internal support access, using store APIs and Fastlane, while Slack notifications make release status visible beyond the release captain. Developers can bypass CI when necessary for emergencies, and configuration favors convention over configuration for project metadata. After more than six months in production, the platform reduced release complexity, shortened the release cadence from three weeks to one week, and helped more people become release captains.
CLI Flags in Practice + How to Make Your Own CLI Command with oclif
CLI commands commonly accept arguments, long flags, short flags, and flag arguments; environment variables and standard input are alternative data channels not covered in detail. GNU-style conventions use double dashes for long flags and single dashes for short flags; short flags can be stacked, and flag options may be separated by a space or equals sign, while short flags also allow no separator. The tutorial uses oclif to build greet-me, which prints a greeting based on a name supplied through -n or --name while registering version, help, name, and force flags. It scaffolds the project with npx oclif single greet-me, places handling code in src/index.ts, defaults the name to world, and runs the CLI through ./bin/run or an npm link.
Bubble Tea Craze on GrabFood!
GrabFood recorded a regional average order growth rate of 3,000% for bubble tea across Southeast Asia in 2018. Individual country growth rates ranged from over 250% in Malaysia to more than 8,500% in Indonesia over their respective tracking periods. The customer base for bubble tea expanded by over 12,000%, supported by a 200% increase in merchant outlets to nearly 4,000 locations representing over 1,500 brands. Southeast Asian consumers ordered an average of four cups per person per month, led by Thailand at six cups and the Philippines at five cups. Order timing concentrated primarily around lunchtime meals and midday afternoon breaks.
Pair Programming Explained
Pair Programming Explained presents pair programming as two people writing, debugging, or exploring code together, while explaining why Shopify uses it for knowledge sharing, collective ownership, learning, quality, and onboarding. It defines terms including problem, solution, driver, navigator, expert, novice, and disengagement, and stresses that the driver handles implementation while the navigator guides broader strategy and spots errors. Guidance covers when to pair, the trade-offs among expert-expert, expert-novice, and novice-novice combinations, and the risks of “watch the master” or disengagement. Suggested practices include switching roles with the Pomodoro Technique, using Strong Style or Digging for Gold, and holding a retrospective to discuss flow, learning, and tension. The article presents pairing as a focused collaborative practice that benefits from shared attention, regular breaks, and deliberate communication rather than an unstructured coding session.
Why You Should Organise an Immersion Trip for Your Next Project
Grab Ventures relies on on-the-ground immersion trips to uncover real consumer behaviors and validate hypotheses that desktop research cannot address. Placing cross-functional teams directly into target markets reveals nuanced consumer motivations, such as Indonesian grocery shoppers prioritizing physical product freshness and price sensitivity over mere convenience. Effective immersion trips require pre-fieldwork reconnaissance by local residents, collaborative hypothesis generation across business and tech disciplines, and non-leading question design. Research teams should operate in small groups of two or three alongside experienced local translators while holding structured end-of-day debriefs to synthesize contextual observations. Within the Double Diamond framework, these field insights support the Discover phase and directly feed into design sprint workshops for problem framing.
Preventing Pipeline Calls from Crashing Redis Clusters
A single Redis slave node failure caused Grab's Apollo booking service to suffer an over 95 percent call failure rate for one minute despite running a three-shard cluster with two replicas per partition. Investigation revealed that the service configured Go-Redis to route all read queries exclusively to slave nodes to offload master CPU usage. When the slave node dropped offline, batched HMGET pipeline calls failed completely because the client wrapper treated a single command failure as a failure of the entire pipeline. Furthermore, the Go-Redis client cached cluster topology and only lazily refreshed state every sixty seconds, continuing to direct traffic to the dead replica until the timer expired. Grab addressed this risk by recommending dedicated pipeline clients configured with latency-based routing to allow reads to fall back to responsive master nodes.
A New Kubectl Plugin for Kubernetes Ingress Controller ingress-nginx
Shopify contributed a kubectl plugin to ingress-nginx, an open-source Kubernetes Ingress controller built on NGINX, because debugging and upgrading the heavily used controller often required long, frustrating sequences of kubectl commands. The plugin’s lint subcommand identifies ingress configuration that may need changes for a new release, while richer ingress listings expose individual paths, target services, and TLS configuration. It also wraps common kubectl operations, automatically selects an ingress pod, and helps inspect generated nginx.conf and dynamic configuration managed with openresty lua-nginx-module. The author reports substantially faster upgrades and debugging, comparing a rollout to version 0.24.1 with an earlier upgrade to 0.22.0 that took days, while acknowledging that growing Kubernetes expertise also contributed. Released with ingress-nginx 0.24.0, the plugin should also be compatible with 0.23.0.
Getting to Know Python 3.7: Data Classes, async/await and More!
Python 3.7 is presented as an upgrade for developers seeking new language features, library improvements, and performance gains over earlier Python 3 releases. The article focuses on data classes, where the @dataclass decorator inspects typed attributes and generates an init method, reducing boilerplate for data-oriented classes. It also covers asyncio.run for simpler asynchronous execution, breakpoint() for entering pdb, module-level getattr for lazy loading, and context variables that keep values separate across execution contexts. Dictionary insertion order becomes part of the language specification in 3.7, while OrderedDict remains available for advanced behavior; reported improvements include 10–30% faster startup, 45–70% faster common sorting cases, 5.5-times-faster dict.copy(), and 4–6-times-faster namedtuple creation.
Building Shopify’s Application Security Program
Shopify's application security program is designed to preserve safety while products and services scale quickly across a platform serving more than 800,000 merchants. It organizes the work around scaling secure applications, scaling security teams, and scaling security interactions, with a culture that treats security as a non-negotiable priority. Standardizing on Ruby on Rails and other shared technologies lets tooling go deep, while service tracking provides visibility into applications, dependencies, code, and automated dependency updates. The security organization grew from one employee into specialized teams, supplemented by penetration testing, bug bounties, and audits, while incident response remains shared across the company. At the code level, tripwires detect dangerous patterns early; Shopify renamed html_safe to dangerously_output_as_html, links flagged pull requests to XSS training, and alerts Application Security, shifting review toward automation.
Optimizing Database Performance in Rails
Production database performance in Rails can degrade as application and data volumes grow, with slow queries, unnecessary queries, and other database bottlenecks becoming significant constraints. The post recommends establishing monitoring early, using Heroku Postgres events, data.heroku.com diagnostics, and the pg-extras pg:outliers command to locate expensive queries, then checking Rails access patterns such as unnecessary association loads. It shows how ActiveRecord includes can reduce a collection example from 11 queries to two, and explains when indexes may help, including Rails migrations for indexed columns and foreign keys, while noting storage and write costs. It also presents fragment caching with Redis as a further optimization and points to connection leaks, locks, and VACUUM as additional areas to investigate, urging benchmarking and metrics for every change.
Guiding You Door-to-Door via Our Superapp!
Grab addressed passenger navigation challenges at large Southeast Asian venues such as airports and shopping centers. Satellite signals weaken through concrete and steel, creating GPS inaccuracies that caused the rendezvous distance between passengers and drivers at large venues to exceed twice the average. While introducing Entrances previously mapped over 120,000 green dots to lower rendezvous distances, passengers still struggled to locate specific pickup spots indoors. The company launched Venues, an in-app feature delivering turn-by-turn text and photo directions to designated pickup points. To support this system, operations teams surveyed sites with cameras and scanners to capture landmarks, after which in-house teams masked faces and vehicle license plates.
Loki, a Dynamic Mock Server for HTTP/TCP Testing
Grab built Loki, a dynamic mock server written in Golang that simulates backend services on local developer machines and CI pipelines. Mobile app testing previously suffered from heavy dependencies on complex, brittle staging environments and interconnected services communicating over HTTP, HTTPS, and TCP. Loki handles both HTTP and TCP traffic on distinct ports while exposing a unified RESTful API to manage test expectations. It provides runtime flexibility through sandboxed JavaScript execution, configurable request sequence ordering, and an in-memory cron scheduler for TCP push messages. Adopting Loki decoupled mobile releases from staging stability, improving delivery cycles and enabling automated UI testing with Espresso and XCUITest.
Saved by the Schema: Using JSON Schema to Document, Test, and Debug APIs
Heroku uses JSON Schema to define Platform API response structures, generate documentation, and validate endpoint behavior. A CI-invoked rake task detects schema changes and uses prmd to generate Markdown API reference material, while committee tests API responses with assert_schema_conform. Those tests exposed an edge case in Enterprise usage data: dyno values with many decimal places were parsed by Oj as BigDecimal and serialized as quoted strings, whereas integer and ordinary float values remained numbers. Heroku traced this behavior through MultiJson and configured Oj with bigdecimal_load: :float, because the endpoint had sufficient precision at 15 decimal places and its schema required a number. The conforming test then passed, confirming that the schema and generated documentation matched returned API values.
Turn Your Code into Docker Images with Cloud Native Buildpacks
The post presents Cloud Native Buildpacks (CNB) as an alternative to hand-written Dockerfiles for turning application source code into Docker images. It argues that Dockerfiles make developers manage language runtimes, system packages, dependency caching, cleanup, and repeated maintenance, creating a leaky abstraction between application and deployment concerns. CNB uses app-aware buildpacks to detect a project’s language from files such as Gemfile or pom.xml, reuse work from previous builds, download dependencies, prepare the application, and export an OCI-compliant image. The beta buildpacks.io tooling provides pack build and a Heroku builder supporting Ruby, Node.js, Java, Python, PHP, and Go; pack rebase can update an image’s toolchain without rebuilding it. The stated conclusion is that this approach reduces operational complexity while retaining compatibility with Docker tooling and the broader container ecosystem.
One Million Dollars in Bug Bounties
Shopify announces that it has awarded more than $1M USD through its bug bounty programs, which complement its security strategy by bringing in researchers with varied perspectives to assess the platform. The company says it is the fifth public program among 176 on HackerOne to reach that milestone, after four years of working with researchers. It describes three resolved vulnerabilities: an SSRF in Shopify Exchange that could provide root access within one infrastructure subset, an admin authentication bypass involving duplicate partner accounts, and stored cross-site scripting from improperly sanitized sales-channel SVG files. Shopify disabled or audited affected services, deployed a metadata-concealment proxy, corrected account verification, removed unnecessary admin functionality, and checked for exploitation. It plans to experiment with ways to increase hacker engagement and make the program more attractive.
How We Harnessed the Wisdom of Crowds to Improve Restaurant Location Accuracy
Grab discovered that abnormally short driver wait times often indicated restaurants registered at incorrect coordinates due to moves or onboarding errors. To fix this, Grab used driver-partner GPS pings, timestamps, and order status updates to infer true food collection locations. The system cleans the data by filtering low-quality GPS pings and isolating the longest temporal streak a driver spends within a predefined radius of the venue. Clusters of inferred pick-up points are then ranked by order volume, the proportion of off-target pick-ups, and median distance errors before routing to mapping operations for verification. This periodic correction workflow achieved a fivefold reduction in order cancellations caused by unfound merchant locations.
Designing Resilient Systems Beyond Retries (Part 3): Architecture Patterns and Chaos Engineering
Building resilient systems requires architectural safeguards and proactive testing beyond basic retries and circuit breakers. Architectural patterns such as idempotency keys enable safe retries without creating inconsistent state during failures. Asynchronous responses and deferrable work isolate services from downstream dependency latency and errors, though they can conflict with the fail-fast principle. To validate system behavior under stress, chaos engineering introduces intentional failures in production to test hypotheses against a defined steady state. Selectively adopting complementary patterns reduces failure points while avoiding unnecessary architectural complexity.
Bug Bounties and Black Swans: How Heroku Expects the Unexpectable
Heroku frames security as an engineering discipline that must challenge happy-path assumptions, test continuously, and prepare for low-predictability, high-consequence “black swan” events. Its threat-modeling guidance focuses on challenging defaults, recognizing that attackers have resources and organizational support, and planning for unexpected failures such as a zero-day exploit. A control-plane instrument for Heroku Postgres detected privilege escalation to a superuser role, and the Data team isolated the instance within 17 minutes while investigating the activity. The event was traced to researcher Andrew Krasichkov (“buglloc”), whose testing exposed a vulnerability in the dblink and postgres_fdw extensions. Through the Bugcrowd bug bounty program, Heroku worked with him to understand the issue, reward the finding, and defend against the previously unknown flaw before a patch was available.
Designing Resilient Systems Beyond Retries (Part 2): Bulkheading, Load Balancing, and Fallbacks
Software systems require mechanisms beyond retries to maintain resilience during downstream outages and high traffic. Bulkheading isolates failures across infrastructure, processes, thread pools, and connection limits, preventing a single failing component from degrading an entire system. Load balancing distributes traffic across backend pools via proxies, client-side libraries, lookaside services, or sidecars, often pairing with health checks to eliminate single points of failure. When operations fail unrecoverably, fallback strategies like silent failures, local defaults, stale cache reads, and dedicated backup services enable graceful degradation. Organizations like Grab implement these approaches using internal client-side load balancers backed by etcd, cache fallbacks in microservice frameworks, and redundant core backup services.
Top topics
- Architecture830
- Performance812
- LLMs648
- Machine Learning617
- Open Source561
- AI Agents542
- AI455
- Developer Experience446
Weekly digest
The best engineering reads, once a week.