---
title: "Latest reads"
description: "The engineering internet, summarised so you can actually read it."
---

# Latest reads
> The engineering internet, summarised so you can actually read it.

## Articles

### [Using an AWS microservice architecture for subscription management](https://yomu.fyi/post/using-an-aws-microservice-architecture-for-subscription-management.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: James Beswick
- Published: Apr 8, 2025
- Type: Explainer

Enterprise subscription management must handle mid-cycle plan changes, usage-based billing adjustments, synchronized service updates, atomicity, failed operations, data consistency, and auditability beyond basic recurring billing. This sample architecture uses an event-driven AWS design in which subscription changes are published to EventBridge and routed to Step Functions workflows. EventBridge provides at-least-once delivery, content-based filtering, dead-letter queue support, and integration with Step Functions, while Lambda functions validate changes, calculate Stripe proration, update external subscriptions, and trigger service-specific actions. Step Functions supplies retries with exponential backoff for StripeTemporaryError, DynamoDB stores current and historical subscription data using transactional updates, and Stripe calls use an event identifier as an idempotency key. The pattern is presented as a flexible foundation for reliability, maintainability, compensation for partial changes, and auditability in systems handling financial transactions and service provisioning.


### [Welcome Lena Waters, Notion’s new Chief Marketing Officer](https://yomu.fyi/post/welcome-lena-waters-notion-s-new-chief-marketing-officer.md)
- Company: [Notion](https://yomu.fyi/company/notion.md)
- Author: Erica Anderson
- Published: Apr 7, 2025
- Type: Announcement

Notion announces Lena Waters as its new Chief Marketing Officer while the company prepares to ship Notion Mail and enters a new phase of growth. Waters says she was attracted by Notion’s technical vision, user enthusiasm, mission, and volume of product releases, and brings experience from Grammarly and Docusign across consumer and B2B marketing, AI, IPO, acquisition integration, and multi-product expansion. She describes her operating theme as building scalable systems without sacrificing creativity, authenticity, or user connection. Her view of marketing’s AI role includes articulating a clear vision and roadmap, educating users, and collecting feedback as Notion AI changes how people work, think, and create. Her immediate priorities are listening, learning, meeting users, and understanding individual and enterprise customers.


### [Facilitating Docs-as-Code implementation for users unfamiliar with Markdown](https://yomu.fyi/post/facilitating-docs-as-code-implementation-for-users-unfamiliar-with-mar.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: David Khu
- Published: Apr 4, 2025
- Type: Problem & solution

Grab introduced a rich-text, WYSIWYG editor in its TechDocs portal to help non-engineering contributors work within a Docs-as-Code process despite limited familiarity with Markdown and GitLab. The editor keeps GitLab as the central storage system, uses Lexical to import and export Markdown, and supports page reordering, renaming, deletion, formatting, diagrams, and switching between normal and Markdown views. Three user journeys cover editing existing TechDocs and creating stand-alone or entity-linked documentation, with GitLab OAuth2, merge-request creation, repository access checks, and scheduler-based entity-catalog registration after merges. A phased rollout added drafts, image uploads, diagram integrations, concurrent-session visibility, and live preview, while pending-MR displays and conflict-resolution tools address collaboration risks.


### [Growth Intern to Procurement New Grad](https://yomu.fyi/post/growth-intern-to-procurement-new-grad.md)
- Company: [Ramp](https://yomu.fyi/company/ramp.md)
- Author: Aaron Abraham
- Published: Apr 3, 2025
- Type: Explainer

A 2023 summer internship on Ramp’s growth engineering team changed the author’s view of software engineering and led to a return as a new graduate engineer. The team accelerated outbound sales through fast experiments, metric ownership, and decisions about whether promising tactics deserved scalable systems, rather than following the thorough specification process typical of regular projects. His work included reducing third-party data-enrichment costs, building AI-driven outbound-sales tools, adding observability and service monitors, and conducting competitive intelligence projects. He returned to join the new procurement product line, where he helped match card transactions to purchase orders and led additional projects. The post also recommends rapidly building system context through code review and on-call shadowing, while full-time engineers should weigh velocity against maintainability and take initiative on less-defined problems.


### [A Fall Internship at Ramp: Improving Reimbursements](https://yomu.fyi/post/a-fall-internship-at-ramp-improving-reimbursements.md)
- Company: [Ramp](https://yomu.fyi/company/ramp.md)
- Author: Sat Arora
- Published: Apr 3, 2025
- Type: Explainer

During a fall internship at Ramp, a backend software engineering intern worked on the reimbursements team and describes projects that improved reimbursement flows while providing full-stack experience. One early change clarified multi-currency mileage reimbursements by showing both the expense currency and the payout currency, while later work overhauled mileage support with live navigation storage, frequent-trip recommendations, commute deductions, and automatic e-receipts. The post also discusses using LLMs to predict merchant categories from receipt text, enhancing merchant matching, and adding monitoring and alerts for OCR categorization errors. The reflection emphasizes measuring feature usage and outcomes, writing and revising specifications before implementation, leaving reusable code, and using Ramp’s transparent documentation and collaborative support. The internship concluded with a return to the University of Waterloo planned for 2025.


### [Tinder’s migration to Elasticsearch 8](https://yomu.fyi/post/tinder-s-migration-to-elasticsearch-8.md)
- Company: [Tinder](https://yomu.fyi/company/tinder.md)
- Author: Tinder
- Published: Apr 1, 2025
- Type: Problem & solution

Tinder modernized its recommendation system by migrating its core Elasticsearch infrastructure from an end-of-life Elasticsearch 6 deployment to Elasticsearch 8 on Kubernetes. The legacy setup relied on manual EC2 provisioning and custom scripts, preventing the engineering team from utilizing modern vector search capabilities while incurring substantial operational overhead. Tinder implemented Elastic Cloud on Kubernetes integrated with its in-house Infrastructure as Code framework, Scaffold, while refactoring custom Java ranking plugins to handle Elasticsearch 8 breaking changes like prohibited negative scoring. The migration was validated through offline benchmarking, online verification, and a three-wave A/B testing rollout. This transition completed with zero outages and under 0.2% data discrepancy, reducing p99 search latency by 12% to 56% and cutting annual infrastructure costs by over one million dollars.


### [Behind the Scenes - A Glimpse into Tax Calculations](https://yomu.fyi/post/behind-the-scenes-a-glimpse-into-tax-calculations.md)
- Company: [Etsy](https://yomu.fyi/company/etsy.md)
- Author: Nancy Arnold
- Published: Mar 29, 2025
- Type: Explainer

As jurisdictions increasingly require marketplaces to collect and remit tax, Etsy describes the infrastructure it uses to calculate, record, and report taxes on transactions. When a buyer adds an item to a cart, Etsy sends a Quotation Request API call to Vertex, using buyer and seller locations, product category, tax rules, and mappings to determine the amount shown. At checkout and throughout fulfillment, DistributeTaxRequest SOAP calls carry order, cancellation, and refund details to Vertex, whose records support reporting and audit workflows imported into Etsy’s bigdata. The integration also required mapping Etsy taxonomy categories to Vertex rule drivers and separating quotation, distribution, and reporting workloads across multiple instances for horizontal scaling and load balancing. Shadow requests, ramped gradually without applying calculated taxes to carts, let the team simulate added Vertex load and monitor checkout performance before releasing traffic-increasing features.


### [Using GitHub Actions with Heroku Flow for additional Security Control](https://yomu.fyi/post/using-github-actions-with-heroku-flow-for-additional-security-control.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: Andrew Fawcett
- Published: Mar 27, 2025
- Type: Tutorial

The pattern combines GitHub Actions with Heroku Flow for private repositories and stricter security requirements when GitHub organization IP restrictions cannot use Heroku IP ranges. A custom self-hosted GitHub Runner runs as a Docker image on Heroku, while the Platform API and Source Endpoint API upload source code to ephemeral storage, build and deploy apps, and manage Review Apps. Event-specific workflows handle pushes, manual runs, schedules, and pull requests; a weekly workflow rebuilds and autodeploys the runner image within GitHub’s 30-day update window. Setup uses a Heroku Private Space, its outbound IPs, a GitHub access token, and organization name, but omits the repository URL in the Review App UI, does not run conventional Heroku CI tests, and is incompatible with Fir.


### [Operationalizing Macaroons](https://yomu.fyi/post/operationalizing-macaroons.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: Mar 27, 2025
- Type: Explainer

Operationalizing Macaroons at Fly.io requires treating them as online-stateful bearer tokens whose nonces must be checked against database state, while keeping root secrets away from the primary API cluster. The team built tkdb, a roughly 5,000-line Go service backed by encrypted SQLite managed with LiteFS and Litestream, deployed on isolated hardware across the US, Europe, and Australia. Its HTTP/Noise interface supports verification and restricted signing, while FlyCast routing, retries, and client-side caching reduce the impact of unreliable global connectivity; verification caching exceeds 98%. Revocation uses blacklist state, and third-party caveats let services such as flyd obtain narrowly scoped secret-reading authority through tkdb rather than holding broad credentials. OpenTelemetry, standard metrics, and a retained OpenSearch audit trail provide visibility, while the post concludes that Macaroons’ internal infrastructure benefits outweigh limited user adoption.


### [Building Ramp's MCP server](https://yomu.fyi/post/building-ramp-s-mcp-server.md)
- Company: [Ramp](https://yomu.fyi/company/ramp.md)
- Author: Junho Choi
- Published: Mar 25, 2025
- Type: Explainer

Ramp built an open-source MCP server that exposes its developer API to LLMs through tools and a SQL interface, enabling natural-language access to business spend data. Using FastMCP in Python and Claude Desktop, the prototype could run analyses, generate visualizations, and issue cards on demand, but pagination struggled beyond a few hundred transactions because of miscalculations, context limits, input sizes, and token use. The team replaced bulk responses with an in-memory ETL that flattens API JSON into SQLite, then exposes load, transformation, and query tools; an OLAP-powered API was added for reporting workloads that caused timeouts. This let Claude analyze tens of thousands of spend events with aggregate and window functions while reducing token usage, although API latency, occasional reliability problems, tool-selection errors, and security risks remain; audit logs, OAuth scopes, and constrained tools provide mitigations, while write actions await a safety framework.


### [Simplifying JVM App Development with Heroku’s Buildpack Magic](https://yomu.fyi/post/simplifying-jvm-app-development-with-heroku-s-buildpack-magic.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: Andrew Fawcett
- Published: Mar 24, 2025
- Type: Explainer

Heroku’s Java buildpacks simplify JVM application deployment by detecting project languages and frameworks, fetching build tools, and configuring runtime settings automatically. They support Spring Boot, Quarkus, and Micronaut by inspecting Maven or Gradle build definitions, packaging applications, selecting framework-specific runner commands, and mapping service variables such as SPRING\_DATASOURCE\_URL, SPRING\_REDIS\_URL, DATABASE\_URL, and JDBC\_DATABASE\_URL. In many cases, a Procfile is unnecessary because the buildpack can identify the executable JAR and provide a default web process, while Quarkus and Micronaut receive their respective runner JAR and port settings. Heroku also automatically attaches a lightweight JVM agent for runtime metrics, including memory usage and garbage collection, and offers similar buildpack support for Scala and Clojure through sbt and Leiningen.


### [Unlocking powerful analytics with Stripe Data Pipeline and Google BigQuery](https://yomu.fyi/post/unlocking-powerful-analytics-with-stripe-data-pipeline-and-google-bigq.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: Sushant Jain
- Published: Mar 21, 2025
- Type: Tutorial

As businesses scale, the post addresses the need to move Stripe revenue data into BigQuery for recurring reporting and analysis. It describes configuring Stripe Data Pipeline to export transactions, invoices, subscriptions, and payments to Google Cloud Storage, then using a Python-based Google Cloud Run Job to discover the latest snapshot through data\_load\_times.json and enumerate each table’s Parquet files. The loader sends those files to BigQuery with the Google Cloud clients, creates missing tables, uses WRITE\_TRUNCATE to retain the latest snapshot, and parallelizes table loads with ThreadPoolExecutor while collecting errors. After deployment and a test execution, the workflow is verified in BigQuery and scheduled to run every six hours, providing an automated path to refreshed Stripe data for analytics.


### [Improving Hugo stability and addressing oncall challenges through automation](https://yomu.fyi/post/improving-hugo-stability-and-addressing-oncall-challenges-through-auto.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Shuguang Xiang
- Published: Mar 20, 2025
- Type: Problem & solution

Hugo, Grab’s data-ingestion platform for a data lake with more than 4,000 user-onboarded pipelines, faces failures that can leave users with limited visibility and on-call teams handling ad-hoc diagnosis, unclear resolution steps, and DPIs without clear RCA. The proposed automation architecture combines Signal, Diagnosis, RCA Table, Auto-resolution, Data Health API, and Data Health Workbench modules to monitor table health, identify root causes and assignees, execute fixes, and expose status to users and other platforms. Signals come from Airflow failure callbacks and Genchi’s SLA and data-completeness checks; diagnosis avoids parsing extensive logs, runs analysers in parallel according to signal type, and supports asynchronous custom handlers such as retry with backoff for transient errors. Reported outcomes include improved data visibility, faster issue resolution, reduced Hugo on-call workload, scalable handling of transient failures, and more precise data-contract incident management, while next steps include runtime optimisation, richer workbench details, and Flink diagnostics.


### [Scaling Nextdoor’s Datastores: Part 5](https://yomu.fyi/post/scaling-nextdoor-s-datastores-part-5.md)
- Company: [Nextdoor](https://yomu.fyi/company/nextdoor.md)
- Author: Slava Markeyev
- Published: Mar 19, 2025
- Type: Problem & solution

Nextdoor addressed database and cache consistency issues caused by missed cache writes and concurrent read-fill operations in their look-aside architecture. While forward row versioning prevents out-of-order write inconsistencies, writer failures and race conditions during cache misses can leave stale data persisted in Redis. To resolve this, Nextdoor built a reconciliation pipeline that consumes PostgreSQL WAL replication logs with pg-bifrost, streams changes through Apache Kafka, and executes conditional deletions in Redis. The Go-based reconciler operates in two passes using a time wheel, running one pass in near real time and a second pass after a delay exceeding web request timeouts. Because conditional deletion evaluates row versions directly in the cache, the system processes change streams out of order and scales horizontally.


### [Scaling Nextdoor’s Datastores: Part 4](https://yomu.fyi/post/scaling-nextdoor-s-datastores-part-4.md)
- Company: [Nextdoor](https://yomu.fyi/company/nextdoor.md)
- Author: Ronak Shah
- Published: Mar 19, 2025
- Type: Problem & solution

Look-aside caching systems can become inconsistent when concurrent database updates execute cache writes out of order, allowing stale data to overwrite newer modifications. To prevent these racing writes, Nextdoor introduced a unique, monotonic db\_version column to Postgres tables using database triggers that initialize version numbers on insert and increment them on update. Application updates retrieve this new version inside a transaction block and attach it as a metadata header to serialized cache values. Redis then executes custom Lua scripts, specifically set\_if\_version and del\_if\_version, to perform atomic conditional updates that reject any incoming payload with a version lower than or equal to the stored version. This serializable check ensures that out-of-order writes are dropped and the cache remains strictly aligned with the latest database state.


### [Scaling Nextdoor’s Datastores: Part 3](https://yomu.fyi/post/scaling-nextdoor-s-datastores-part-3.md)
- Company: [Nextdoor](https://yomu.fyi/company/nextdoor.md)
- Author: Ronak Shah
- Published: Mar 19, 2025
- Type: Problem & solution

Look-aside caching with object byte serialization can cause critical compatibility failures when application versions, dependencies, or database schemas change. Serialized cache entries bound to specific runtimes risk deserialization errors during deployments, triggering thundering herd queries against the underlying datastore. To resolve this, Nextdoor replaced runtime-bound serialization like Python pickle with MessagePack to serialize Django model objects. The team achieved forward compatibility by letting MessagePack ignore unrecognized new fields in older application code, while backward compatibility relies on mandatory default values for newly added model attributes. Nextdoor prepends a ten-byte header containing format metadata and version information before writing the serialized payload to cache stores.


### [Scaling Nextdoor’s Datastores: Part 2](https://yomu.fyi/post/scaling-nextdoor-s-datastores-part-2.md)
- Company: [Nextdoor](https://yomu.fyi/company/nextdoor.md)
- Author: Tushar Singla
- Published: Mar 19, 2025
- Type: Problem & solution

Nextdoor encountered scaling issues after adding read replicas when product engineers were initially tasked with deciding whether to route queries to the primary or replica databases. As business logic grew and gained abstraction layers, engineers struggled to track read-after-write consistency constraints across the call stack. To avoid replication lag race conditions, engineers routinely wrapped logic in database transactions, unintentionally directing all queries to the primary node and eroding read replica benefits over several years. The Core-Services team resolved this by injecting custom tracking logic into their Django ORM layer to monitor table writes during web requests and automate routing. They further optimized the system using a timing-based approach that restored replica read eligibility after the p99.9 replication lag elapsed.


### [Scaling Nextdoor’s Datastores: Part 1](https://yomu.fyi/post/scaling-nextdoor-s-datastores-part-1.md)
- Company: [Nextdoor](https://yomu.fyi/company/nextdoor.md)
- Author: Slava Markeyev
- Published: Mar 19, 2025
- Type: Explainer

Nextdoor relies on a Django backend connected to monolithic PostgreSQL databases and Redis look-aside caches. Migrating to distributed SQL datastores proved impractical because legacy business logic depends heavily on multi-table joins that could not be rewritten. Intermediate mitigations, including read replicas and data partitioning by severing foreign keys, extended infrastructure runways but left primary databases vulnerable to load bottlenecks. Furthermore, look-aside caching and replica usage introduced data staleness risks, atomicity losses across partitioned databases, and inconsistent cache-population behaviors. To address these limitations, Nextdoor initiated an architecture redesign focused on dynamic query routing to replicas, replica-driven cache hydration, time-bounded eventual consistency, and schema-resilient cache serialization.


### [Stripe’s new AI Assistant in VS Code](https://yomu.fyi/post/stripe-s-new-ai-assistant-in-vs-code.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: Mathew Varughese
- Published: Mar 18, 2025
- Type: Announcement

Stripe has introduced an AI Assistant in its VS Code extension to answer developer questions using current Stripe knowledge rather than relying on LLM pre-training. The extension retrieves API reference entries, integration guides, code examples, and curated summaries of developer Discord threads, while inserting the user’s API key into generated snippets and supporting either GitHub Copilot’s @stripe agent or its own chat interface. Its retrieval pipeline classifies queries, combines BM25 keyword and k-nearest-neighbor embedding search, reranks results, and sends selected sources, code, user code, and the question to Claude Sonnet through a RAG prompt. A nightly Temporal workflow refreshes indexed content, while evaluation uses golden and synthetic datasets, Mean Reciprocal Rank, and human and automated review. On its synthetic test dataset, the system includes the best source about 91.11% of the time and reports an MRR of about 78%.


### [Real-time payment analytics: Building a data pipeline from Stripe to AWS](https://yomu.fyi/post/real-time-payment-analytics-building-a-data-pipeline-from-stripe-to-aw.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: James Beswick
- Published: Mar 14, 2025
- Type: Tutorial

Payment processors such as Stripe emit numerous transaction events, while dashboard and API-based or batch approaches can limit historical analysis, real-time visibility, and correlation with business metrics as volumes grow. The proposed pipeline sends signed Stripe webhooks through API Gateway to Lambda for validation and enrichment, buffers them in Kinesis Data Streams, and uses a second Lambda to transform and index records in OpenSearch. OpenSearch Dashboards provides visualizations, while CloudWatch tracks pipeline health; Kinesis retention and replay support recovery and reprocessing. The post also describes a Kinesis Data Firehose route that can remove the consumer Lambda and reduce costs but may add batching latency. The resulting design is presented as scalable to millions of transactions per day, with sub-minute payment metrics visibility and flexible historical querying, while security guidance includes encryption, IAM configuration, and regular audits.


[Newer posts](https://yomu.fyi/posts/page/179.md) · [Older posts](https://yomu.fyi/posts/page/181.md)