Loading…
Latest reads
Every post, summarised
Grounded summaries of engineering blogs, newest first. Each links to the original.
- 4,671 posts
- 20 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Latest posts
Unlocking powerful analytics with Stripe Data Pipeline and Google BigQuery
As businesses scale, the post addresses the need to move Stripe revenue data into BigQuery for recurring reporting and analysis. It describes configuring Stripe Data Pipeline to export transactions, invoices, subscriptions, and payments to Google Cloud Storage, then using a Python-based Google Cloud Run Job to discover the latest snapshot through data_load_times.json and enumerate each table’s Parquet files. The loader sends those files to BigQuery with the Google Cloud clients, creates missing tables, uses WRITE_TRUNCATE to retain the latest snapshot, and parallelizes table loads with ThreadPoolExecutor while collecting errors. After deployment and a test execution, the workflow is verified in BigQuery and scheduled to run every six hours, providing an automated path to refreshed Stripe data for analytics.
Improving Hugo stability and addressing oncall challenges through automation
Hugo, Grab’s data-ingestion platform for a data lake with more than 4,000 user-onboarded pipelines, faces failures that can leave users with limited visibility and on-call teams handling ad-hoc diagnosis, unclear resolution steps, and DPIs without clear RCA. The proposed automation architecture combines Signal, Diagnosis, RCA Table, Auto-resolution, Data Health API, and Data Health Workbench modules to monitor table health, identify root causes and assignees, execute fixes, and expose status to users and other platforms. Signals come from Airflow failure callbacks and Genchi’s SLA and data-completeness checks; diagnosis avoids parsing extensive logs, runs analysers in parallel according to signal type, and supports asynchronous custom handlers such as retry with backoff for transient errors. Reported outcomes include improved data visibility, faster issue resolution, reduced Hugo on-call workload, scalable handling of transient failures, and more precise data-contract incident management, while next steps include runtime optimisation, richer workbench details, and Flink diagnostics.
Scaling Nextdoor’s Datastores: Part 5
Nextdoor addressed database and cache consistency issues caused by missed cache writes and concurrent read-fill operations in their look-aside architecture. While forward row versioning prevents out-of-order write inconsistencies, writer failures and race conditions during cache misses can leave stale data persisted in Redis. To resolve this, Nextdoor built a reconciliation pipeline that consumes PostgreSQL WAL replication logs with pg-bifrost, streams changes through Apache Kafka, and executes conditional deletions in Redis. The Go-based reconciler operates in two passes using a time wheel, running one pass in near real time and a second pass after a delay exceeding web request timeouts. Because conditional deletion evaluates row versions directly in the cache, the system processes change streams out of order and scales horizontally.
Scaling Nextdoor’s Datastores: Part 4
Look-aside caching systems can become inconsistent when concurrent database updates execute cache writes out of order, allowing stale data to overwrite newer modifications. To prevent these racing writes, Nextdoor introduced a unique, monotonic db_version column to Postgres tables using database triggers that initialize version numbers on insert and increment them on update. Application updates retrieve this new version inside a transaction block and attach it as a metadata header to serialized cache values. Redis then executes custom Lua scripts, specifically set_if_version and del_if_version, to perform atomic conditional updates that reject any incoming payload with a version lower than or equal to the stored version. This serializable check ensures that out-of-order writes are dropped and the cache remains strictly aligned with the latest database state.
Scaling Nextdoor’s Datastores: Part 3
Look-aside caching with object byte serialization can cause critical compatibility failures when application versions, dependencies, or database schemas change. Serialized cache entries bound to specific runtimes risk deserialization errors during deployments, triggering thundering herd queries against the underlying datastore. To resolve this, Nextdoor replaced runtime-bound serialization like Python pickle with MessagePack to serialize Django model objects. The team achieved forward compatibility by letting MessagePack ignore unrecognized new fields in older application code, while backward compatibility relies on mandatory default values for newly added model attributes. Nextdoor prepends a ten-byte header containing format metadata and version information before writing the serialized payload to cache stores.
Scaling Nextdoor’s Datastores: Part 2
Nextdoor encountered scaling issues after adding read replicas when product engineers were initially tasked with deciding whether to route queries to the primary or replica databases. As business logic grew and gained abstraction layers, engineers struggled to track read-after-write consistency constraints across the call stack. To avoid replication lag race conditions, engineers routinely wrapped logic in database transactions, unintentionally directing all queries to the primary node and eroding read replica benefits over several years. The Core-Services team resolved this by injecting custom tracking logic into their Django ORM layer to monitor table writes during web requests and automate routing. They further optimized the system using a timing-based approach that restored replica read eligibility after the p99.9 replication lag elapsed.
Scaling Nextdoor’s Datastores: Part 1
Nextdoor relies on a Django backend connected to monolithic PostgreSQL databases and Redis look-aside caches. Migrating to distributed SQL datastores proved impractical because legacy business logic depends heavily on multi-table joins that could not be rewritten. Intermediate mitigations, including read replicas and data partitioning by severing foreign keys, extended infrastructure runways but left primary databases vulnerable to load bottlenecks. Furthermore, look-aside caching and replica usage introduced data staleness risks, atomicity losses across partitioned databases, and inconsistent cache-population behaviors. To address these limitations, Nextdoor initiated an architecture redesign focused on dynamic query routing to replicas, replica-driven cache hydration, time-bounded eventual consistency, and schema-resilient cache serialization.
Stripe’s new AI Assistant in VS Code
Stripe has introduced an AI Assistant in its VS Code extension to answer developer questions using current Stripe knowledge rather than relying on LLM pre-training. The extension retrieves API reference entries, integration guides, code examples, and curated summaries of developer Discord threads, while inserting the user’s API key into generated snippets and supporting either GitHub Copilot’s @stripe agent or its own chat interface. Its retrieval pipeline classifies queries, combines BM25 keyword and k-nearest-neighbor embedding search, reranks results, and sends selected sources, code, user code, and the question to Claude Sonnet through a RAG prompt. A nightly Temporal workflow refreshes indexed content, while evaluation uses golden and synthetic datasets, Mean Reciprocal Rank, and human and automated review. On its synthetic test dataset, the system includes the best source about 91.11% of the time and reports an MRR of about 78%.
Real-time payment analytics: Building a data pipeline from Stripe to AWS
Payment processors such as Stripe emit numerous transaction events, while dashboard and API-based or batch approaches can limit historical analysis, real-time visibility, and correlation with business metrics as volumes grow. The proposed pipeline sends signed Stripe webhooks through API Gateway to Lambda for validation and enrichment, buffers them in Kinesis Data Streams, and uses a second Lambda to transform and index records in OpenSearch. OpenSearch Dashboards provides visualizations, while CloudWatch tracks pipeline health; Kinesis retention and replay support recovery and reprocessing. The post also describes a Kinesis Data Firehose route that can remove the consumer Lambda and reduce costs but may add batching latency. The resulting design is presented as scalable to millions of transactions per day, with sub-minute payment metrics visibility and flexible historical querying, while security guidance includes encryption, IAM configuration, and regular audits.
Puzzle’s Notion Playbook: Eight Startup Hacks to Save 8+ Hours Weekly
Puzzle presents Notion as the central operating system for its startup, using it as a single source of truth for company knowledge, workflows, and collaboration. The playbook describes eight practices, including integrating tools and legacy content, standardizing recruiting and onboarding, supporting asynchronous communication, organizing teamspaces, building documentation habits, applying AI, and connecting task management to strategic priorities. Its implementation relies on templates, checklists, permissions, comments, @mentions, automations, relations, rollups, database views, formulas, and verification controls. For a team of 50+ employees, Notion AI reportedly saves about ten minutes per query; five daily queries per person are calculated as more than twenty-one hours saved weekly across the company. The conclusion recommends starting with one area, such as meeting documentation or onboarding guides, before expanding.
Building a Spark observability product with StarRocks: Real-time and historical performance analysis
Grab’s Iris project replaces a fragmented Spark observability stack with a centralised platform for real-time and historical analysis. The previous setup split data between Grafana, Superset, InfluxDB, and a data lake, creating usability, access-control, pipeline, metadata, and high-cardinality query challenges. The new architecture uses StarRocks as a unified store, ingesting Kafka directly, serving a custom Iris web application and Superset, and backing up to S3. Its relational data model links worker metrics through worker_uuid and Spark event metrics through app_id, while materialised views and dynamic partitioning support responsive queries and data retention as volumes grow. The resulting system consolidates metrics and metadata, simplifies ingestion and offline processing, and supports job-level showback/chargeback; the roadmap includes recommendations, advanced analytics, machine learning, and scalability improvements.
Load balancing Stripe API calls from multiple AWS regions
The guide describes a multi-region payment-processing architecture for applications that use Stripe’s API and need resilience against regional outages, latency, rate limits, and payment-data compliance constraints. It routes requests with Amazon Route 53 health checks and configurable latency-based, weighted, or geolocation policies, while identical regional API Gateway and Lambda deployments process payments. DynamoDB Global Tables replicate payment state and support distributed rate limiting, with optimistic locking, regional token-bucket allocation, retries, and exponential backoff for Stripe requests. The sample Lambda records a payment as PENDING, creates and confirms a Stripe PaymentIntent, then updates the state with the Stripe payment ID and returned status; the conclusion presents the design as a foundation for regional resilience, state consistency, and API-rate management.
Boolean Query Operators Guide, With Examples
Social listening queries filter relevant brand, product, industry, or consumer mentions from large volumes of online conversation, beginning with the research questions the analysis should answer. The guide explains Boolean operators including quotation marks for exact phrases, OR for alternatives, AND for context, NEAR/x for proximity, NOT for exclusions, wildcards for variations, and brackets for grouping. It recommends adding misspellings, hashtags, account handles, websites, and exclusions, then testing and adjusting the query while keeping its structure simple where possible. Brandwatch Consumer Research offers Boolean, non-Boolean, and blank query options, with editor validation and Iris AI suggestions for entities, keywords, hashtags, subreddits, and multilingual terms.
Importing sales data from Stripe into AWS
The article presents a low-code method for analyzing Stripe business and customer data and moving scheduled report results into AWS. Stripe Sigma lets users query payment, subscription, and plan-related data with SQL, while Sigma Assistant can generate and execute queries from natural-language prompts such as cancellation analysis. For delivery, the article uses the sigma.scheduled_query_run.created event and Stripe's Event Destination to Amazon EventBridge, where a partner event bus and event rule trigger AWS services without a separate public webhook API. Its Lambda example downloads the report CSV with the Stripe API key, parses it into JavaScript objects, and publishes the result through Amazon SNS; the same event-driven pattern can support Lambda, Step Functions, or Glue workflows.
TechDocs at Grab: Cultivating a culture of quality documentation
Grab’s TechDocs working group describes a two- to three-year effort to make high-quality documentation part of everyday engineering culture. Built on the Helix engineering platform and a Docs-as-Code workflow, TechDocs gives teams a central place to create, manage, and access technical content, while user research informs tool and process changes. The programme combines inventory-taking, policy, team enablement, and recurring measurement, using broad surveys and 1:1 observation to identify fragmented tools, weak ownership, search problems, and maintenance gaps. Research found that only 2% of engineers reported resolving 80–100% of on-call questions through documentation; later initiatives included an editor, writing training, practical workshops, and recognition tied partly to documentation quality and on-call-bot response accuracy. Planned work includes better navigation, training, feedback loops, and editing workflows.
Building serverless usage notification with AWS
Pay-as-you-go services need a way to warn customers when consumption reaches a configured limit, but building usage aggregation, monitoring, and notification workflows adds operational complexity. This tutorial uses Stripe Billing Alerts to define a customer-specific usage threshold with a customer_id, meter_id, gte value, and recurrence, then sends billing.alert.triggered events through Stripe Event Destinations to an Amazon EventBridge partner event source. It tests the flow with Stripe Meter Events and shows AWS CLI rules routing the event to Amazon SNS, as well as a Lambda example that retrieves the customer's email from Stripe and sends mail with Amazon SES. The result is a minimal-code notification path using AWS services, with sandbox setup required because some APIs and features may not be available in conventional test mode.
How the Tinder iOS App reduced the size of our localizations by 95% using Emerge
Supporting over 50 languages across numerous statically linked targets led to a substantial build size footprint in Tinder's iOS application. Because Apple's code signing mandates a minimum 4KB per file, shipping dozens of localized files per target inflated the final application package. Tinder addressed this by stripping comments and whitespace, merging localized strings into a single file per language via custom Bazel rules and Aspects, and compressing the strings using Emerge's SmallStrings tool into LZFSE files. The runtime decompresses these files dynamically through existing code-generated string accessors. Consequently, Tinder reduced download size by 10.7MB and install size by 51.3MB with no impact on developer workflows.
Faster and Local Shopping on eBay Just Got Simpler
eBay has introduced UX enhancements that make local listings, fast-shipping items, delivery estimates, and retail standards easier to find on Search item cards. Because its inventory comes from thousands of sellers in a global, decentralized network, eBay uses artificial intelligence that considers customer proximity, chosen shipping service, seller shipping history, and real-time data synchronization to improve estimate accuracy and coverage. Item cards now show delivery ranges for all fast-shipping items, whether delivery is paid or free, while the “Shipping and pickup” filter supports fast shipping, Free shipping, Local pickup, and Click & Collect options. For local pickup, cards display the exact distance between the buyer and item location. In Germany, the eBay Top-Service badge identifies items meeting standards including fast and free shipping, easy returns, and trusted sellers, simplifying overlapping retail signals.
Taming A Voracious Rust Proxy
Fly edge proxies began reporting elevated HTTP errors and sharply higher CPU utilization on several IAD hosts, with restarting fly-proxy providing only temporary relief. Profiling an affected process showed Rust tracing and repeated Future polls dominating samples while little time was spent in the kernel, pointing to an async busy-loop. Investigation narrowed the fully qualified polling type to a stack containing the proxy’s Duplex and Rustls’s TlsStream, whose mishandled Waker behavior after a TLS CloseNotify could spin when buffered data remained underneath. Tigris Data’s load test, involving tens of thousands of connections that sent small HTTP bodies and terminated early, triggered the condition; stopping the test and deploying the fix ended the spin-outs. The incident prompted plans to instrument spurious wakeups and reinforced that dependency updates require deliberate tracking, testing, and prioritization.
Behind the scenes of Canva's DesignDNA campaign
Canva launched DesignDNA in December 2024 as a personalized year-in-review campaign to highlight user achievements and showcase generative AI capabilities. Because internal privacy rules strictly prohibited inspecting personal designs, the engineering and creative teams inferred user preferences from the style and theme metadata tags on public templates. An initial keyword-matching algorithm paired 95% of users with one of seven emerging design trends, while generative AI keyword expansion increased coverage to 99%. The team also synthesized over one million localized poems and distinct design personalities using tools like Magic Write and Dream Lab. Finally, dynamic Canva template elements populated through URL parameters assembled 95 million distinct multi-page stories across nine locales.
Top topics
- Architecture834
- Performance816
- LLMs660
- Machine Learning622
- Open Source563
- AI Agents553
- AI470
- Developer Experience448
Weekly digest
The best engineering reads, once a week.