Loading…
Topic
379 posts about Scalability. Every summary links to the original.
See the idea
Consolidating agent memory, retrieval, and governance onto MongoDB Atlas removes sync and vendor layers, but ties scaling and control to one platform. This trade-off view shows the payoff, the cost, and the condition that decides it.
Autoscaling trades responsiveness, stability, and cost: this explainer shows how signal choice, scaling direction, and capacity limits shape the right control loop for a workload.
Apache Pinot is a distributed real-time OLAP datastore for low-latency analytics. This architecture view shows how streams, processing, partitioning, indexed storage, and replica-aware serving fit together.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Payment processors such as Stripe emit numerous transaction events, while dashboard and API-based or batch approaches can limit historical analysis, real-time visibility, and correlation with business metrics as volumes grow. The proposed pipeline sends signed Stripe webhooks through API Gateway to Lambda for validation and enrichment, buffers them in Kinesis Data Streams, and uses a second Lambda to transform and index records in OpenSearch. OpenSearch Dashboards provides visualizations, while CloudWatch tracks pipeline health; Kinesis retention and replay support recovery and reprocessing. The post also describes a Kinesis Data Firehose route that can remove the consumer Lambda and reduce costs but may add batching latency. The resulting design is presented as scalable to millions of transactions per day, with sub-minute payment metrics visibility and flexible historical querying, while security guidance includes encryption, IAM configuration, and regular audits.
The guide describes a multi-region payment-processing architecture for applications that use Stripe’s API and need resilience against regional outages, latency, rate limits, and payment-data compliance constraints. It routes requests with Amazon Route 53 health checks and configurable latency-based, weighted, or geolocation policies, while identical regional API Gateway and Lambda deployments process payments. DynamoDB Global Tables replicate payment state and support distributed rate limiting, with optimistic locking, regional token-bucket allocation, retries, and exponential backoff for Stripe requests. The sample Lambda records a payment as PENDING, creates and confirms a Stripe PaymentIntent, then updates the state with the Stripe payment ID and returned status; the conclusion presents the design as a foundation for regional resilience, state consistency, and API-rate management.
Canva launched DesignDNA in December 2024 as a personalized year-in-review campaign to highlight user achievements and showcase generative AI capabilities. Because internal privacy rules strictly prohibited inspecting personal designs, the engineering and creative teams inferred user preferences from the style and theme metadata tags on public templates. An initial keyword-matching algorithm paired 95% of users with one of seven emerging design trends, while generative AI keyword expansion increased coverage to 99%. The team also synthesized over one million localized poems and distinct design personalities using tools like Magic Write and Dream Lab. Finally, dynamic Canva template elements populated through URL parameters assembled 95 million distinct multi-page stories across nine locales.
Reliable Stripe webhook processing must account for lost deliveries from network or service outages, out-of-order events, and duplicates caused by Stripe retries. The proposed AWS architecture uses API Gateway for signature validation and throttling, an SQS FIFO queue for ordered delivery and content-based deduplication, Lambda for processing, DynamoDB for event-ID idempotency, and an SQS DLQ for failed messages. In the CloudFormation example, the main queue has a 300-second visibility timeout and sends messages to the FIFO DLQ after three receives; DynamoDB records expire through a seven-day TTL. Lambda retries failures with exponential backoff, while CloudWatch monitors queue depth, latency, and errors. The design is presented as a scalable foundation, with multi-region failover available at added cost and complexity, though single-region deployment may suffice for many applications.
As Canva expanded to over 200 million monthly active users, ingestion throughput reached 25 billion records daily, causing AWS Data Firehose costs to consume nearly half of the product analytics platform budget. To reduce these expenses and eliminate intermediate file staging, the engineering team integrated Snowflake's Snowpipe Streaming directly with their Java-based Kinesis Data Streams pipeline. The architecture streams records directly into Snowflake tables using logical channels with offset checkpoints while configuring client buffering up to a five-minute maximum lag. In handling Kinesis Client Library edge cases, record processors coordinate with the LeaseCoordinator to drop leases when channels become unhealthy. Operating in production for over six months, the system ingested more than 20.35 petabytes of data, decreased query latency to roughly ten minutes, and reduced overall cloud spend by 45 percent.
The DevRel Swag Store uses Stripe payments and AWS services to keep product inventory accurate and visible in near real time. Stripe products retain core payment information, while Amazon DynamoDB stores inventory and other attributes, keyed by Stripe product ID and store ID; EventBridge routes payment events to Lambda, which atomically decrements stock and publishes updates through AWS IoT Core to the frontend. DynamoDB condition expressions prevent decrements when stock is insufficient, while post-payment validation and refunds address the delay before a Stripe Payment Link is disabled, although Stripe fees may be non-refundable. A custom Payment Intent flow can check stock before payment, authorize funds with manual capture for up to seven days, update inventory after payment success, and then capture the charge.
Grab’s GrabUnlimited flagship membership program struggled as subscribers grew by over 1000% from January 2022 to June 2023. Its original design used a daily cron job for renewals, Amazon’s SQS to distribute state handlers, and five-minute Redis locks, but outages, corrupted states, concurrent cancellation and renewal, missed benefits, and non-idempotent retries created operational problems. The team migrated the service to a Temporal workflow-based system, enhancing its ability to handle millions of subscribers with greater efficiency and resilience. The source reports an 80% reduction in open production incidents and says the change improved the membership experience for users. This first part focuses on designing a robust and scalable membership system, while migration challenges and lessons are reserved for a subsequent installment.
Stripe Event Destinations, now in general availability and rolling out to all accounts, lets companies extend payment workflows without changing existing integrations. The post explains how Stripe events can be sent either to an AWS account through Amazon EventBridge or to a webhook endpoint, with EventBridge routing selected events to services such as AWS Lambda, Step Functions, and SQS. Event filtering, logging, transformation, and built-in security reduce the need to operate webhook servers and verify signatures in application code. Serverless Lambda functions can add custom logic and connect to DynamoDB, S3, or SNS, supporting examples including date-limited memberships, partial student payments, and real-time tipping broadcasts through AWS IoT Core and WebSockets. The resulting architecture is presented as flexible and scalable for variable or growing event volumes while preserving the underlying Stripe integration.
Grab developed and scaled Catwalk, an internal machine learning model serving platform, to address operational bottlenecks, low resource utilization, and deployment friction between data scientists and backend engineers. The platform transitioned from an admin-managed TensorFlow Serving setup into a low-code self-service system supporting PyTorch and ONNX, before replacing complex Helm charts with Kubernetes Custom Resource Definitions for declarative, blue-green deployment orchestration. To support complex business workflows and multi-model applications, Grab subsequently introduced Catwalk Orchestrator with bundled deployments that allow individual services to scale independently. Across two years, the orchestrator architecture expanded to 200 deployed applications serving approximately 1,400 production machine learning models.
Grab's Chimera team built Chimera Sandbox to address resource accessibility challenges and enable rapid machine learning experimentation across technical and non-technical teams. The platform runs on multi-cluster Kubernetes infrastructure capable of scaling to thousands of nodes per cluster during peak loads. Users can provision customizable Docker environments with varying CPU and GPU configurations, collaborate in real time, and query data using Spark-on-Kubernetes integrations and helper magic cells. The workspace integrates with Grab's AI Gateway for LLM access alongside tools like MLflow, Optuna, RStudio, and LabelStudio on shared storage, backed by user-level cost attribution.
MySQL requires locks before updating existing records, and concurrent upserts that mix new and existing rows can deadlock when gap locks overlap, even when processes target different records. The post describes ETL jobs partitioned by account whose updates nevertheless contend because sequential primary-key ordering places records in shared index gaps. It proposes replacing the existing primary key with a composite key of account and id, aligning InnoDB's physical row ordering with the processes' update partitions; gap locks remain necessary, but are largely compartmentalized within each process. An additional unique index on id preserves standalone record identification and compatibility with existing joins, while application and ETL logic remain unchanged. The reported outcome was removal of artificial delays and significantly higher import throughput through reduced job wait time.
Grab needed to classify sensitive data at the table and column level across petabytes of database tables and streaming schemas. Manual schema-tiering campaigns had resulted in half of all schemas receiving overly strict Tier 1 access controls, while an initial automated service using regex patterns and third-party machine learning produced high false-positive rates and lacked customizability. To address this, the Caspian data engineering and governance teams enhanced their internal orchestration service, Gemini, by integrating GPT-3.5 via Azure OpenAI. Gemini aggregates classification requests into mini-batches, handles API rate limits, and uses prompt engineering—including few-shot examples, curated tag libraries, and explicit JSON DTO schemas—to reliably tag columns for data owner verification.
Mae Santos, Head of Critical Applications and Data Reliability Engineering at Two Sigma, leads a global team across New York, Houston, London, and Tokyo to support critical applications and data pipelines. While software teams frequently prioritize functional features over operational stability, designing systems with reliability in mind from the beginning allows for progressive enhancements without complete rewrites. System architecture should be guided by concrete goals across four key dimensions: availability, observability, scalability, and supportability. In addition, mapping out interconnections among systems, infrastructure, data, and human processes helps teams manage dependencies effectively. Applying automation across support and delivery workflows further reinforces reliability by minimizing human error and accelerating execution.
Grab needed a scalable, centralised way to evaluate the business impact of millions of marketing messages sent across expanding campaigns. Manual analysis by marketing analysts produced inconsistent results and failed to scale, leading to the creation of multi-touch attribution and controlled experiment methodologies. The team designed a Marketing attribution model (MAM) that credits touchpoints based on relevance and recency, while using stratified hold-out groups to assess single campaign lift against baseline metrics. Statistical significance calculations, implemented using the jStat library based on the central limit theorem, ensure observed business metric differences like gross merchandise value are not random. The system provides campaign managers with direct access to impact analyses while the team works on challenges surrounding smaller sample sizes and hold-out group constraints.
Grab manages content moderation across GrabFood and GrabMart, where merchants update over 100,000 item listings daily across eight countries. The platform must adhere to internal rules, local government regulations, and external platform guidelines across these markets. To handle scale and regional nuance, Grab uses an in-house automated system powered by algorithms and machine learning to scan daily listings for violations. Items flagged as requiring subjective judgment or cultural awareness are routed to human moderators for manual review. Additionally, Grab collaborates with Google to maintain compliance with Play Store policies.
Shop app’s hockey-stick growth pushed its Ruby on Rails backend and MySQL primary datastore toward capacity, with multi-terabyte storage, week-long schema migrations, and throttled background jobs. After splitting databases and scaling surrounding systems, the team adopted Vitess to shard and re-shard MySQL while retaining SQL compatibility and self-managed primary datastores. They organized user-owned data into a sharded “users” keyspace using user_id, kept other data in an unsharded “global” keyspace, and built application-layer query verifiers to validate routing, distribution, and transaction correctness before migration. The move made schema migrations take hours instead of weeks, removed the reported capacity-driven throttling, and enabled further scaling by adding shards, while introducing concepts such as Sequences, Vindexes, and VSchemas that developers must learn.
Marketing communications across Grab's user base risked causing notification overload and consent revocations. To enforce personalized daily and weekly frequency caps across more than 270 million users, the team addressed segment membership storage and communication rate limiting. They adopted roaring bitmaps instead of Bloom filters to compactly store user segment data without hash collisions or costly rebuilds upon deletion. For frequency capping, they chose Amazon ElastiCache for Redis over DynamoDB, executing a sliding log rate limiting algorithm directly on the cluster using Lua scripts and sorted sets. Timestamps are stored as sorted set scores, and historical data is cleaned up via eviction ranges to prevent unbounded memory growth.
Grab needed to localise nearly 30 million road segments across more than 10,000 area boundaries in Southeast Asia to attach hyperlocal attributes like language, driving side, and vehicle access rules. Testing direct geometric inclusion between complex road polylines and boundary polygons proved computationally prohibitive for daily map generation pipelines. To optimize this process, Grab introduced geohashes as rectangular proxies, precomputing geohash coverage across borders and road segments before joining the datasets in parallel. To resolve misclassification errors near borders without sacrificing performance, the team evaluated geohash coverage percentages and refined boundary-touching geohashes with polygon clipping. A laptop benchmark on a subset of data reduced runtime from 38 minutes with geometric intersection to 78 seconds using the geohash approximation.
Grab's legacy lookalike audience generation platform suffered from long creation SLAs of two working days, high costs, and low weekly update frequencies. To resolve these bottlenecks, the engineering team designed an embedding-based platform powered by an in-memory retrieval service and automated update pipelines. The system creates audience representations by averaging constituent passenger embeddings, determining user membership through real-time cosine score thresholds. To eliminate feature store latency while fitting all embeddings into memory, a hash-based compression method cuts passenger embedding storage needs by roughly 90%. Consequently, audience availability dropped to within 15 minutes of campaign creation, audience generation costs fell by 98%, and ad impressions and clicks doubled.
Nextdoor transitioned its ranking and recommendation pipelines from traditional continuous and discrete interaction features to transformer-based representation learning. The engineering team deployed pre-trained Sentence-BERT models to generate multilingual post and comment representations, which were aggregated daily by interaction type to form user embeddings. To improve search recall, the team fine-tuned sentence transformers on unlabeled query session logs using contrastive learning and integrated HNSWlib for approximate nearest neighbor retrieval. Subsequent iterations incorporated labeled feedback, BERTopic for coarse personalization, and experiments with CLIP image embeddings. Infrastructure scaling challenges were addressed by performing embedding transformations directly within FeatureStore and optimizing feature payload formats to minimize microservice network bandwidth.
The best engineering reads, once a week.
