Loading…
Latest reads
The engineering internet, summarised so you can actually read it.
Nextdoor ·
Scaling Nextdoor’s Datastores: Part 2
Nextdoor encountered scaling issues after adding read replicas when product engineers were initially tasked with deciding whether to route queries to the primary or replica databases. As business logic grew and gained abstraction layers, engineers struggled to track read-after-write consistency constraints across the call stack. To avoid replication lag race conditions, engineers routinely wrapped logic in database transactions, unintentionally directing all queries to the primary node and eroding read replica benefits over several years. The Core-Services team resolved this by injecting custom tracking logic into their Django ORM layer to monitor table writes during web requests and automate routing. They further optimized the system using a timing-based approach that restored replica read eligibility after the p99.9 replication lag elapsed.
Tushar SinglaNextdoor ·
Scaling Nextdoor’s Datastores: Part 1
Nextdoor relies on a Django backend connected to monolithic PostgreSQL databases and Redis look-aside caches. Migrating to distributed SQL datastores proved impractical because legacy business logic depends heavily on multi-table joins that could not be rewritten. Intermediate mitigations, including read replicas and data partitioning by severing foreign keys, extended infrastructure runways but left primary databases vulnerable to load bottlenecks. Furthermore, look-aside caching and replica usage introduced data staleness risks, atomicity losses across partitioned databases, and inconsistent cache-population behaviors. To address these limitations, Nextdoor initiated an architecture redesign focused on dynamic query routing to replicas, replica-driven cache hydration, time-bounded eventual consistency, and schema-resilient cache serialization.
Slava MarkeyevStripe ·
Real-time payment analytics: Building a data pipeline from Stripe to AWS
Payment processors such as Stripe emit numerous transaction events, while dashboard and API-based or batch approaches can limit historical analysis, real-time visibility, and correlation with business metrics as volumes grow. The proposed pipeline sends signed Stripe webhooks through API Gateway to Lambda for validation and enrichment, buffers them in Kinesis Data Streams, and uses a second Lambda to transform and index records in OpenSearch. OpenSearch Dashboards provides visualizations, while CloudWatch tracks pipeline health; Kinesis retention and replay support recovery and reprocessing. The post also describes a Kinesis Data Firehose route that can remove the consumer Lambda and reduce costs but may add batching latency. The resulting design is presented as scalable to millions of transactions per day, with sub-minute payment metrics visibility and flexible historical querying, while security guidance includes encryption, IAM configuration, and regular audits.
James BeswickGrab ·
Building a Spark observability product with StarRocks: Real-time and historical performance analysis
Grab redesigned its Spark observability platform, Iris, to overcome limitations associated with its previous Telegraf, InfluxDB, and Grafana stack. InfluxDB presented operational challenges due to limited SQL compatibility, poor handling of string metadata, and query degradation on high-cardinality identifiers. The team replaced InfluxDB with StarRocks to serve as a unified analytical engine for both real-time cluster metrics and historical analysis. StarRocks ingests metrics directly from Kafka via routine load tasks, storing worker and Spark event data in partitioned duplicate-key OLAP tables linked by worker and application identifiers. This architecture eliminated intermediate ingestion agents, simplified S3 data lake backups, and enabled a custom web application alongside Superset for consistent querying.
Huong VuongStripe ·
Load balancing Stripe API calls from multiple AWS regions
The guide describes a multi-region payment-processing architecture for applications that use Stripe’s API and need resilience against regional outages, latency, rate limits, and payment-data compliance constraints. It routes requests with Amazon Route 53 health checks and configurable latency-based, weighted, or geolocation policies, while identical regional API Gateway and Lambda deployments process payments. DynamoDB Global Tables replicate payment state and support distributed rate limiting, with optimistic locking, regional token-bucket allocation, retries, and exponential backoff for Stripe requests. The sample Lambda records a payment as PENDING, creates and confirms a Stripe PaymentIntent, then updates the state with the Stripe payment ID and returned status; the conclusion presents the design as a foundation for regional resilience, state consistency, and API-rate management.
James BeswickStripe ·
Importing sales data from Stripe into AWS
The article presents a low-code method for analyzing Stripe business and customer data and moving scheduled report results into AWS. Stripe Sigma lets users query payment, subscription, and plan-related data with SQL, while Sigma Assistant can generate and execute queries from natural-language prompts such as cancellation analysis. For delivery, the article uses the sigma.scheduled_query_run.created event and Stripe's Event Destination to Amazon EventBridge, where a partner event bus and event rule trigger AWS services without a separate public webhook API. Its Lambda example downloads the report CSV with the Stripe API key, parses it into JavaScript objects, and publishes the result through Amazon SNS; the same event-driven pattern can support Lambda, Step Functions, or Glue workflows.
Hidetaka OkamotoGrab ·
TechDocs at Grab: Cultivating a culture of quality documentation
Engineering organizations frequently struggle with fragmented documentation, stale content, and a lack of clear ownership across disparate tools. To address these issues, Grab established TechDocs on its central Helix platform, embedding a Docs-as-Code workflow into daily engineering routines. Feedback gathered from quantitative surveys and one-on-one sessions shaped governance policies, separating stable platform documentation stored in GitLab from collaborative artifacts like RFCs in Confluence. To sustain document freshness, the platform assigns mandatory points of contact, displays last-updated timestamps, and flags pages untouched for more than three months.
David KhuStripe ·
Building serverless usage notification with AWS
Pay-as-you-go services need a way to warn customers when consumption reaches a configured limit, but building usage aggregation, monitoring, and notification workflows adds operational complexity. This tutorial uses Stripe Billing Alerts to define a customer-specific usage threshold with a customer_id, meter_id, gte value, and recurrence, then sends billing.alert.triggered events through Stripe Event Destinations to an Amazon EventBridge partner event source. It tests the flow with Stripe Meter Events and shows AWS CLI rules routing the event to Amazon SNS, as well as a Lambda example that retrieves the customer's email from Stripe and sends mail with Amazon SES. The result is a minimal-code notification path using AWS services, with sandbox setup required because some APIs and features may not be available in conventional test mode.
Hidetaka OkamotoTinder ·
How the Tinder iOS App reduced the size of our localizations by 95% using Emerge
Supporting over 50 languages across numerous statically linked targets led to a substantial build size footprint in Tinder's iOS application. Because Apple's code signing mandates a minimum 4KB per file, shipping dozens of localized files per target inflated the final application package. Tinder addressed this by stripping comments and whitespace, merging localized strings into a single file per language via custom Bazel rules and Aspects, and compressing the strings using Emerge's SmallStrings tool into LZFSE files. The runtime decompresses these files dynamically through existing code-generated string accessors. Consequently, Tinder reduced download size by 10.7MB and install size by 51.3MB with no impact on developer workflows.
TinderCanva ·
Behind the scenes of Canva's DesignDNA campaign
Canva launched DesignDNA in December 2024 as a personalized year-in-review campaign to highlight user achievements and showcase generative AI capabilities. Because internal privacy rules strictly prohibited inspecting personal designs, the engineering and creative teams inferred user preferences from the style and theme metadata tags on public templates. An initial keyword-matching algorithm paired 95% of users with one of seven emerging design trends, while generative AI keyword expansion increased coverage to 99%. The team also synthesized over one million localized poems and distinct design personalities using tools like Magic Write and Dream Lab. Finally, dynamic Canva template elements populated through URL parameters assembled 95 million distinct multi-page stories across nine locales.
Divya PatelStripe ·
Securing Stripe API Keys in AWS with automatic rotation
The article presents a production-grade approach to securing Stripe API keys in AWS, motivated by the risks and operational limits of basic secret storage for payment processing. It compares AWS Parameter Store with Secrets Manager, emphasizing built-in rotation, CloudTrail audit trails, IAM integration, larger secrets, and cross-account access despite higher cost. The implementation uses environment-specific secret paths, tags, and IAM conditions to isolate development, staging, and production credentials, with separate rotation schedules. For zero-downtime rotation, applications refresh cached credentials after authentication failures while both old and new keys remain valid during a transition. The design also covers CloudWatch monitoring, emergency rotation procedures, cross-region replication, region-specific schedules, and cost considerations, concluding that ongoing review is necessary for secure and reliable payment operations.
James BeswickGrab ·
Grab AI Gateway: Connecting Grabbers to multiple GenAI providers
Grab built the AI Gateway to centralize access, cost control, and security across external and open-source Generative AI providers such as OpenAI, Azure, AWS, and Google. Designed as a set of lightweight reverse proxies, the gateway manages authentication, rate limiting, and authorization while translating payloads into a unified OpenAI-compatible interface. The platform archives request metadata and calculated per-call costs into a central data lake for auditing and showback, dynamically routing traffic across shared reserved capacity and regions to mitigate quota throttling. Supporting over 300 internal use cases, the system integrates directly with internal development notebooks and deployment tools to power applications ranging from real-time audio safety analysis to automated content moderation.
Bjorn JeeStripe ·
Tracking customer spend in an omnichannel or multiprocessor environment
Merchants operating across websites, mobile apps, physical stores, and multiple payment processors struggle to connect spending because customers use physical cards, digital wallets, and tokenized methods. Payment Account Reference (PAR), a 29-character alphanumeric identifier introduced by EMVCo, provides a non-financial, non-reversible reference that maps one PAN to multiple tokens without exposing sensitive payment data. Because the PAR is independent of payment method and processor, merchants can link transactions across channels and preserve continuity when a card is replaced with a new PAN. The post presents an omnichannel retail example and describes potential benefits including unified customer profiles, personalization, cross-channel reconciliation, reduced PAN-related PCI burden, and processor flexibility.
Andrew RobinsonStripe ·
Building rock-solid Stripe integrations: A developer's guide to success
The guide presents ten practices for building production-ready Stripe integrations, covering payment forms, subscriptions, and operational concerns. It recommends verifying webhook signatures with the Stripe-Signature header and endpoint secret, recording event IDs for idempotency, and returning a fast 2xx response while processing asynchronously. Test and live API keys, webhook endpoints, logs, and monitoring should remain separate, while error handling should distinguish card, request, and API failures and use exponential backoff for retries. The guide also recommends testing successful, declined, insufficient-funds, and 3DS scenarios with Stripe test cards, alongside simulated webhook events. Its conclusion emphasizes edge-case planning, logging, monitoring, and ongoing review as foundations for reliable, secure payment processing that can scale with business needs.
James BeswickAI Core Team Lead Mike Schuster on How to Get the Most From LLMs
Mike Schuster, Head of the AI Core Team at Two Sigma, advocates for grounding large language model adoption in practical tasks rather than speculative industry hype. Realistic enterprise applications focus on accelerating data processing, running faster experiments, and extracting domain-specific features from transcripts such as earnings calls and Federal Reserve speeches via prompt engineering. Because financial data faces inherent volume limits across trading days, successful deployments require multidisciplinary human teams to balance rapid technical experimentation with rigorous domain expertise and analytical reasoning. Schuster also dismisses predictions that programming will become obsolete, comparing coding to learning a musical instrument that cultivates structured thinking, problem decomposition, and scientific common sense essential for building complex predictive models.
Joy LooneyStripe ·
Building resilient webhook handlers in AWS: Implementing DLQs for Stripe events
Reliable Stripe webhook processing must account for lost deliveries from network or service outages, out-of-order events, and duplicates caused by Stripe retries. The proposed AWS architecture uses API Gateway for signature validation and throttling, an SQS FIFO queue for ordered delivery and content-based deduplication, Lambda for processing, DynamoDB for event-ID idempotency, and an SQS DLQ for failed messages. In the CloudFormation example, the main queue has a 300-second visibility timeout and sends messages to the FIFO DLQ after three receives; DynamoDB records expire through a seven-day TTL. Lambda retries failures with exponential backoff, while CloudWatch monitors queue depth, latency, and errors. The design is presented as a scalable foundation, with multi-region failover available at added cost and complexity, though single-region deployment may suffice for many applications.
James BeswickTinder ·
Sharing Tinder’s latest contributions to the open source community
Tinder open-sourced several iOS development repositories, including Layout, Nodes Architecture Framework, and CombineUI, to share the engineering patterns supporting its iPhone application. As the app expanded across diverse device profiles and scale, engineering teams faced reliability, consistency, and memory challenges under their legacy architecture. In response, Tinder developed Nodes, a plugin-based architecture framework using compile-time dependency injection and lifecycle hooks that enforce complete memory release upon feature dismissal. For interface construction, Tinder created Layout, a domain-specific Auto Layout wrapper offering declarative syntax for UIKit views to eliminate storyboard merge conflicts while preserving native capabilities. The resulting stack enables isolated testing of business logic, native reactive event binding, and incremental adoption of SwiftUI across the codebase.
TinderStripe ·
New to Stripe? Learn the key concepts for software developers.
The guide introduces software developers to Stripe’s main objects and workflows for integrating payments, recurring billing, and related account activity. It begins with Payment Intents, which specify an amount and currency, track a payment’s lifecycle, and produce a client secret for completing the frontend flow, then covers Payment Methods and Customers, including Setup Intents and attaching a default payment method. For subscriptions, it distinguishes Products, Prices, and Subscriptions, and explains using events and webhooks to react to asynchronous payment and subscription changes while verifying webhook signatures. It also describes Disputes, refunds, and expanded API responses, including retrieving related resources in one call. The guide recommends starting in test mode and using dashboard logs while consulting Stripe’s documentation and changelog as the platform evolves.
James BeswickCanva ·
Image replacement in Canva designs using reverse image search
Canva needed an automated way to replace media in design templates, such as when third-party licensing partnerships expire across more than 150 million images. Existing recommendation engines, perceptual hashing, and text metadata searches failed to capture visual similarity hierarchies or ensure replacement relevance. To build a reverse image search system, engineers evaluated embedding models including CLIP, ViTMAE, DreamSim, CaiT, and DINOv2 alongside an external vector database supporting metadata filtering. Evaluation on sample datasets identified DINOv2 as the best model for preserving subjects, background context, and color tones in photos. Integrated into the Template Assistant as a human-in-the-loop tool, the automated suggestions increased image replacement speeds by 4.5 times during initial pilot testing.
Sam JacobsCanva ·
The foundations of Canva’s continuous data platform with Snowpipe Streaming
As Canva expanded to over 200 million monthly active users, ingestion throughput reached 25 billion records daily, causing AWS Data Firehose costs to consume nearly half of the product analytics platform budget. To reduce these expenses and eliminate intermediate file staging, the engineering team integrated Snowflake's Snowpipe Streaming directly with their Java-based Kinesis Data Streams pipeline. The architecture streams records directly into Snowflake tables using logical channels with offset checkpoints while configuring client buffering up to a five-minute maximum lag. In handling Kinesis Client Library edge cases, record processors coordinate with the LeaseCoordinator to drop leases when channels become unhealthy. Operating in production for over six months, the system ingested more than 20.35 petabytes of data, decreased query latency to roughly ten minutes, and reduced overall cloud spend by 45 percent.
Jack Caperon