Loading…
Latest reads
Every post, summarised
Grounded summaries of engineering blogs, newest first. Each links to the original.
- 4,538 posts
- 20 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Latest posts
Up to 75% Faster Maintenances with Heroku Postgres and Redis Premium Plans
Heroku describes how it minimizes downtime during routine security and patching maintenances for high-availability Heroku Postgres and Heroku Redis plans. These plans run a writable primary alongside a hidden read-only standby, and maintenance requires moving application writes safely to the standby before promoting it. Previously, Heroku changed DATABASE_URL and restarted app dynos, then waited for the old primary to shut down and replicate remaining writes, which could extend downtime during long-running transactions. The new method redirects new connections to the new primary, kills existing connections to the old primary, and relies on reconnections plus a rolling restart to prevent further writes there. Heroku reports HA maintenances now complete in 15–40 seconds instead of 60 seconds or more, and says the behavior is default for Premium, Private, and Shield plans.
Building Resilient GraphQL APIs Using Idempotency
Shopify’s Payment Service uses API-level idempotency to make GraphQL payment mutations resilient to timeouts, connection failures, and ambiguous outcomes that could otherwise leave merchants unpaid or double-charge buyers. The design places a UUID-based idempotency key directly in each mutation input, identifies requests by client plus key, and locks concurrent duplicates so retries receive a 409 response while the original call is processing. An IncomingRequest record stores request state and completed recovery points, allowing completed responses to be replayed and incomplete attempts to recover before continuing. Handlers divide work into no-side-effect, transactional local-side-effect, and remote-side-effect steps, each optionally providing run and recover functions. The approach adds database-write overhead and requires careful compatibility testing when recovery points change, but the team reports that its structured handlers are clear and maintainable.
Designing for Accessibility: Contrast Ratio
Heroku’s accessibility initiative targeted WCAG 2.0 Level AA luminance contrast for its user interfaces, addressing reduced contrast sensitivity and color-vision deficiencies. The team audited its palette with Contrast Grid, proposed darker and revised colors, then chose a narrower proposal that preserved existing names and colors while meeting accessibility concerns within an affordable implementation effort. Because the palette affected GUIs built with Purple 3 and Malibu, pull requests generated Review Apps so teams could audit production-ready CSS and SVG assets across Dashboard components before release. Audits using axe-core 3.2.2 covered common and unusual elements, reducing color-contrast violations by an order of magnitude before rollout. The updated palette now spans the Heroku Dashboard, with planned accessibility testing through Ember tooling and further audits across other interfaces.
Living on the Edge of Rails
Shopify describes running its Core monolith on the unreleased Rails 6 master branch, or “Edge of Rails,” since February 2019, with the goal of continuously using the framework’s latest revision. Instead of annual upgrades, weekly changes reduce diff size and let developers use upstream features quickly, while a 130,000-plus-test suite finds edge cases and informs patches to Rails and related projects such as Zeitwerk. Solid Track automates weekly Rails bumps, opens GitHub pull requests, and triggers git bisect when CI fails to identify the breaking commit. Green builds move through ten-minute canary deployments before reaching the remaining servers. The approach trades the stability of released dependencies for faster fixes and features, relying on monitoring, tests, and controlled deployment to manage risk.
Pagination with Relative Cursors
Incremental page-number pagination becomes increasingly expensive for large offsets, because the database scans and discards earlier records before returning a page. Shopify describes relative cursor pagination, which remembers the last record seen and continues from that point, using an indexed id to keep query time nearly constant as pages increase. In tests on a shop with 14 million products, a 100,000 offset took 2,221.60 ms, compared with 5.24 ms using the last id, while offsets of 1,000,000 and above consistently timed out. Sorting by another field requires retaining both its last value and a unique secondary key such as id, with a matching compound index to avoid skipped records and maintain performance. Shopify introduced Link-header URLs for next and previous pages and began migrating REST endpoints from page numbers, reporting early /admin/products.json usage as about 11 times faster, though adoption remained low and full results were pending.
Data First, SLA Always
Grab's Data Engineering team transitioned from periodic batch ETL ingestion to a real-time change data capture architecture called Trailblazer after dataset sizes exceeded the petabyte mark. The previous system caused severe JDBC timeouts and heavy CPU loads when executing chunked or full-scan queries on unindexed upstream MySQL tables. To solve this, MySQL binary logs are captured via Debezium on Kafka Connect, buffered in Kafka, and ingested into a data lake using Spark Structured Streaming. Checkpoints are decoupled from local storage and persisted in a Redis cluster to simplify ingestion offset overrides and handle ephemeral compute clusters. The system incorporates extensive health monitoring across Airflow, Datadog, and custom services to maintain stream liveliness and avoid Kafka retention breaches.
Save Your Place with Grab!
Grab introduced Saved Places across Southeast Asia to eliminate the friction of repeatedly typing addresses and prevent selection errors between similarly named locations. Data analysis across transport and food orders revealed that consumers consistently visit only five to seven unique locations and order food to one or two addresses. To streamline repeat bookings, cross-functional teams designed a feature allowing users to bookmark locations under custom labels such as Home and Work. Following usability testing and release, more than 14 million users stored nearly 45 million addresses across the platform. Platform metrics also showed that while office destinations clustered in central districts across major cities, residential distributions varied significantly between markets like Singapore and Jakarta.
No More Forgetting to Input ERP Charges - Hello Automated ERP!
Grab launched an automated Electronic Road Pricing (ERP) fare calculation feature in Singapore to eliminate the need for driver-partners to manually track gantries and enter toll charges. Because Singapore gantries frequently adjust fares based on time and road conditions, manual entry often caused driver errors and revenue loss. Grab solved this by mapping precise geographical coordinates for every toll gate using satellite imagery and open data, matching frequent driver GPS pings against road layers and gantry locations. The engineering and operations teams also built an internal ERP Workflow tool to map ride trajectories and resolve driver dispute feedback within an average of one day. Following its rollout in Singapore, Grab began testing and planning regional expansion to Indonesia, Thailand, Malaysia, and the Philippines.
How We Built a Logging Stack at Grab
Grab needed a scalable logging platform to replace slow, fragmented systems that hindered debugging across their growing service fleet. Generating 25TB of daily logs, the team built a horizontally scalable Elasticsearch cluster configured via Ansible and monitored with Datadog. Although the initial proof of concept assigned all node roles (ingest, coordinator, master, and data) to every machine, operating at scale introduced major challenges with JVM heap exhaustion and cluster stability. The team resolved memory pressure and performance bottlenecks by tuning circuit breakers, lowering field data cache limits, adjusting shard allocations based on segment memory, and disabling translog compression during shard transfers.
Lessons from Leading a Remote Engineering Team
Managing remote engineering teams can expand hiring beyond office locations, support a more diverse workforce, retain employees through relocations, and enable distributed coverage such as 24/7 support. The guidance describes building such teams around relationships, trust, and communication, with “Default to open internally” and “Charge your trust battery” as supporting cultural values. It recommends asynchronous documentation and discussion in email, GitHub issues, Google Docs, and project systems, supplemented by a fixed weekly synchronous meeting and shared core hours. Managers should also support healthy home workspaces, maintain regular one-on-ones and timely contact, arrange selective in-person sessions, and actively champion remote employees’ visibility, recognition, and career progression. The approach requires deliberate effort but can provide hiring, composition, retention, and strategic benefits.
Dataclips Power Insights at Heroku
Heroku Dataclips let employees create SQL queries through a web GUI against Heroku Postgres databases, then share results through unique URLs or HTTP endpoints returning CSV or JSON. At Heroku, Dataclips reduce dependence on engineers or DBAs by letting departments retrieve fresh production metrics, build dashboards, track product usage, identify deprecated-feature users, and investigate suspicious signup patterns. To make querying safer, the organization runs dataclips against a read-only Postgres follower rather than the leader; writes propagate from leader to followers in close to real-time. Business Operations also teaches SQL and the internal schema through Office Hours, aiming to replace recurring one-off requests with self-service query skills.
Puma 4: Hammering Out H13s—A Debugging Story
Heroku investigated mysterious H13s—“Connection closed” without response—affecting Ruby applications during dyno deployments or scaling, especially at high traffic levels. The failure involved a race between SIGTERM shutting down Puma and the router’s separate system continuing to deliver a few requests, but comparison with other webservers indicated Puma needed to handle the condition. A reproduction used Rack applications that sent SIGTERM to Puma and themselves, plus parallel curl requests, reproducing curl exit code 52 and Heroku H13s. Debugging showed Puma relied on process termination to close accepted sockets; Puma 4 instead closes binder listeners before shutdown, allowing the router to retry connections while preserving responses for in-flight requests.
How Heroku Operates its Multi-Tenant Apache Kafka Services
Heroku’s Apache Kafka service supports thousands of developers and millions of transactions, including customers sharing multi-tenant clusters. The post explains how those clusters provide security, predictable performance, availability, and production-like topology at lower cost than single-tenant plans. Multi-tenant and single-tenant offerings use the same numbers of brokers and ZooKeeper nodes, while Kafka ACLs, per-tenant resource namespaces, and producer/consumer byte quotas isolate customers and limit resource contention. Heroku provisions purchased resources, expands storage at 80% of quota, and uses higher defaults for partitions, replicas, and in-sync replicas; exceeding capacity can eventually throttle throughput without stopping the cluster. Automated tests model 50 users with varied traffic, exercise throughput and partition configurations, and simulate broker failures and restarts to identify operational limits and adjust the infrastructure before customer impact.
Componentizing Shopify’s Tax Engine
Shopify’s tax engine grew from a simple system into difficult code as tax rules and ownership spread across Cart, Checkout, and Order. The team first learned the surrounding code and tax domain with help from a tax subject matter expert, then mapped dependencies using regular expressions, scripts, manual investigation, and tax-related bug work. They moved tax source files into a dedicated component behind explicit TaxesRequestSchema and TaxesReponseSchema interfaces, with simple-type request and response schemas, protected boundaries, and expanded unit, integration, and end-to-end tests. During release, the old and new engines calculated taxes concurrently so discrepancies could be measured; after iterative fixes, Shopify rolled the component out gradually from a small group of shops to all shops over several days, with no downtime or merchant impact reported.
Making Grab’s Everyday App Super
Grab manages an expanding superapp ecosystem comprising ride-hailing, food delivery, payments, and partner content surfaced through the Grab Feed. As content volume grows, the platform risks overwhelming users with irrelevant information. To address this, Grab built a recommendation engine that ranks cards using signals across user profiles, content metadata, and contextual factors such as time and location. The system employs multiple recommendation strategies—including popularity metrics, user favorites, collaborative filtering, habitual patterns, and cross-platform deep embeddings—which are selected or aggregated. Recommendation quality is evaluated via offline metrics like Recall@K and NDCG alongside online engagement experiments.
Catwalk: Serving Machine Learning Models at Scale
As machine learning adoption expanded at Grab, individual teams created fragmented model serving solutions that duplicated engineering effort and required data scientists to handle underlying infrastructure. To resolve these inefficiencies, Grab developed Catwalk, a self-service machine learning model serving platform. The system runs TensorFlow Serving containers across a managed Kubernetes cluster integrated with Grab's observability stack. Data scientists deploy or update models simply by saving files using the tf.saved_model API to dedicated Amazon S3 buckets, while Kubernetes automates orchestration, ingress routing, and pod autoscaling. Catwalk abstracts server management away from data scientists, shortens deployment timelines, and provides high availability during model version rollouts.
Six Strategies for Deploying to Heroku
Heroku offers six deployment strategies: Git push, GitHub integration, Review Apps, Docker, Terraform, and the “Deploy to Heroku” button. Each has different requirements and trade-offs, and strategies can be combined in a workflow. Git push is simple but requires a developer with access to both repository and app, while GitHub integration can automate deployments and block them until CI tests pass. Review Apps provide disposable instances for pull requests; Docker supplies self-contained environments but requires maintaining a stack and does not support pipeline promotions. Terraform uses declarative HCL to manage Heroku and external infrastructure despite lacking official Heroku support, and buttons offer one-click deployments from GitHub repositories but do not auto-update or connect deployed apps back to their repositories.
Implementing Android POS Receipt Printing on Shopify
Shopify’s Android POS lacked printed-receipt support until March 2019, leaving merchants who needed it to use iPads. The design targets a pragmatic, adaptive, composable, maintainable, and highly testable API, while handling more than 8 billion possible receipt content combinations across discounts, tips, transactions, gift cards, refunds, payment methods, and regional requirements. Its six-step pipeline aggregates models from SQLite or GraphQL, transforms them into a PrintableReceipt, defines view-based sections, creates printer-agnostic PrintRequest objects, renders section bitmaps at the required resolution, and passes them through the Hardware SDK to vendor-specific integrations. The PrintableReceiptComposer contains most business logic, with unit tests covering features and edge cases, while screenshot tests compare rendered output pixel by pixel in CI. This componentized separation isolates printer support from receipt generation and allows new printer models to be added in a couple of hours.
React Native in GrabPay
Following the release of the GrabPay Merchant App, Grab adopted React Native inside the Grab Passenger app to maintain a single cross-platform codebase across iOS and Android. Integrating the framework into native applications required establishing bridge communication guidelines and incorporating React Native modules into the Grablet architecture. To streamline network communication, API calls were migrated from axios to native bridges returning promises, which eliminated the need to pass access tokens into JavaScript. The team also established a shared internal library of approximately 20 UI components alongside Redux for state management and react-navigation for routing. Modules like BillPay and Transaction History successfully launched across Southeast Asia while maintaining the performance and feel of native software.
Connecting the Invisibles to Design Seamless Experiences
Service design operates as connective tissue within complex product ecosystems by bridging digital touchpoints, physical operations, and backstage technical workflows. At Grab, placing an order on GrabFood requires coordinating driver allocation, customer support paths, and long-term data storage rather than simply transmitting information to merchants. Focusing exclusively on singular features risks breaking broader network dependencies when modifications ripple into other operational systems. Grab addresses these interdependencies through participatory design processes and visual mapping across cross-functional teams. This holistic framework evaluates whether systemic issues, such as inaccurate restaurant operating hours, are best resolved through in-app feature changes or operational adjustments.
Top topics
- Architecture830
- Performance812
- LLMs648
- Machine Learning617
- Open Source561
- AI Agents542
- AI455
- Developer Experience446
Weekly digest
The best engineering reads, once a week.