Loading…
Topic
375 posts about Scalability. Every summary links to the original.
See the idea
Consolidating agent memory, retrieval, and governance onto MongoDB Atlas removes sync and vendor layers, but ties scaling and control to one platform. This trade-off view shows the payoff, the cost, and the condition that decides it.
Autoscaling trades responsiveness, stability, and cost: this explainer shows how signal choice, scaling direction, and capacity limits shape the right control loop for a workload.
Apache Pinot is a distributed real-time OLAP datastore for low-latency analytics. This architecture view shows how streams, processing, partitioning, indexed storage, and replica-aware serving fit together.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
The post benchmarks pgvector 0.4.0 for approximate nearest-neighbor search and assesses whether PostgreSQL can support production-scale vector workloads. Using the ANN Benchmarks methodology, pre-warmed RAM, the dbpedia-entities-openai-1M dataset of 1 million 1,536-dimensional OpenAI embeddings, and Supabase ARM instances, it varies probes, lists, concurrency, and compute size. On an 8-core, 32 GB instance, pgvector reached 0.91 accuracy@10 at 380 QPS with 10 probes and 0.98 accuracy at 140 QPS with 40 probes; a 64-core, 256 GB server reached about 1,800 QPS at 0.91 accuracy. The results indicate predictable scaling, while higher lists can improve query speed at the cost of index-build time, and the roadmap identifies HNSW, product quantization, and parallel index builds as planned improvements.
Grab redesigned its messaging infrastructure from GrabChat to Message Center to overcome two-party chat limitations and support complex superapp requirements like group conversations and varied user roles. The architecture separates core processing logic from message delivery by splitting the system into a backend processor and an independently scalable postman service. Communication relies on an in-house TCP gateway named Hermes that proxies client payloads via gRPC, alongside Apache Kafka streams and Amazon SQS delay queues. Custom client-server acknowledgements and a DynamoDB event store ensure reliable message delivery even across dropped mobile TCP connections and offline reconnects.
Storage v3 adds resumable uploads, quality and format filters, a Next.js loader, WebP rendering, presigned upload URLs, and per-bucket size and file-type limits. Resumable uploads divide files into chunks, emit progress events, and allow interrupted transfers to continue; Pro Plan and higher users can upload files up to 50GB, up from 5GB. Rather than expose S3’s proprietary protocol, the implementation uses the open TUS protocol over HTTP through the Node.js Storage-API service and tus-node-server, with Postgres RLS integration. To support multiple TUS instances behind a load balancer, transactional Postgres Advisory locks serialize access to each S3 object and help address concurrent uploads, while the release rolls out gradually over the next month.
The post defines a staff developer at Shopify as an individual contributor whose scope and seniority can match a manager’s, while emphasizing that the role is a form of leadership rather than simply a more senior senior developer. Technical excellence comes from varied, challenging project experience, including large-scale systems, refactors, technology shifts, mistakes, and learning to choose appropriate solutions under constraints. The role also pairs with a team manager, translating product and business objectives into technical proposals, judging scope and infrastructure needs, finding recurring product problems, and saying no when effort does not match return. Its impact is measured partly through enabling others via reviews, mentorship, sponsorship, and support, leading to an adaptable leader comfortable with ambiguity whose responsibilities evolve with company needs.
Shopify’s Merlin machine learning platform was expanded to support online inference, allowing trained models to return predictions as input arrives rather than through periodic batch jobs. The design targets varied Shopify use cases and requirements, including support for TensorFlow, PyTorch and XGBoost, low latency, autoscaling, rolling deployments, observability and model updates. Each use case runs as a dedicated service on Google Kubernetes Engine, with a Python serving layer that loads models and artifacts, exposes an inference endpoint, and can use the Comet ML model registry and Pano feature store. Separate staging and production configurations let teams tune resources and test model versions before release; Shopify reports that online inference is providing the intended low latency, scalability and faster iteration, while ensemble inference, platform-based monitoring and continuous training remain planned areas.
HappyTeams, a bootstrapped startup, needed its database to support thousands of users and continued feature development without consuming scarce technical resources. Its Heroku Postgres deployment had developed scaling problems, including sluggish or uneven performance and limits on concurrent connections, prompting a database change during an application rebuild in early 2022. The company migrated to Supabase in less than an hour using pg_dump and pg_restore, and adopted PgBouncer for connection pooling; it selected the service for performance, connection flexibility, support and documentation, onboarding, and lower costs. The source reports that Supabase removed the performance issue, made pooling effortless, and helped HappyTeams release features faster, including a comments feature completed in two days, while the company considered expanding into Edge Functions and the API client.
Shopify treats Black Friday Cyber Monday (BFCM) as its most critical period, and uses full-scale BFCM Scale Testing to assess whether the platform can handle peak traffic. Preparation combines capacity planning, resiliency testing, single-application performance testing, and repeated exercises that scale core services to their BFCM profiles before generating load. The hybrid architecture centers on Shopify Core, a monolithic Ruby application supported by services for storefront rendering, analytics, fraud analysis, payments, webhooks, and more. Five load flows cover browsing and buying, admin activity, flash sales, Storefront APIs, and Hydrogen and Oxygen, with Cronograma layering requests gradually and providing an abort switch. The exercises surfaced connection-count pressures, a Kubernetes ingress-nginx endpoint limit, and uneven load-balancer distribution; mitigations included refactoring, scaling dependent services, increasing pod capacity, and dedicated nginx node pools.
Shopify presents three additional practices for optimizing large stateful Apache Flink applications, focusing on parallelism, sink capacity, and combining heterogeneous sources. It recommends starting with execution-environment parallelism, then matching task-manager capacity to the highest parallelism value; for example, parallelism 100 requires 25 task managers with four slots each. Sink bottlenecks can propagate backpressure upstream, so batching writes and correcting data skew—including with key bucketing when a low-cardinality or uneven key is unavoidable—can improve distribution, though bucketed results must later be combined. For archived Kafka topics, HybridSource exposes cloud-storage history and the live Kafka topic as one ordered logical DataStream, automatically switching after the archive is exhausted and using object-storage partitions to accelerate historical backfills.
Preparing for BFCM and other high-traffic events, Shopify describes risk mitigation for a globally distributed, complex, interconnected infrastructure platform that peaked at 75.98M requests per minute, or 1.27M per second. At this scale, identifying and mitigating every possible failure is impossible, so the process starts with “what could go wrong” exercises, then uses voting, expert review of likelihood and severity, and assigned ownership to prioritize action. Decision makers are empowered to weigh risks and rewards, while summarized findings, including key risks and single points of failure, are shared to maintain alignment and awareness. The post frames engineering choices as probabilistic bets: decision quality should be judged separately from outcomes, and uncertainty cannot be eliminated, only addressed through thoughtful prioritization and communication.
Shopify’s 2022 BFCM Live Map needed to deliver real-time sales and product data to thousands of concurrent users while processing data from millions of merchants. Its 2021 architecture used WebSocket delivery to a presentation layer, which stored messages in a mailbox that clients polled at least every 10 seconds, creating latency and bottlenecks. Shopify replaced that path with a streamlined Flink-based pipeline and a Golang SSE server that subscribes to Kafka topics and pushes JSON data to registered, authenticated clients as soon as it is available. The SSE server ran horizontally behind Nginx load balancers, with a configurable Java client used to load-test concurrent connections. The 2022 system maintained 100 percent uptime, delivered SSE data within milliseconds of availability, and visualized data within 21 seconds of creation, including pipeline processing.
Rapid expansion across multiple business verticals introduced diverse, evolving fraud patterns at Grab, making traditional rule engines and decision trees inadequate due to their heavy reliance on extensive feature engineering and labeled data. To address cold starts and detect unknown fraud, the engineering team implemented a semi-supervised Relational Graph Convolutional Network (RGCN) across millions of connected ecosystem entities. The model propagates information along neighborhood nodes through graph convolutional layers to produce node embeddings and output fraud probabilities. Trained on graphs where only a small percentage of nodes are labeled, the RGCN achieved an AUROC close to 1 and distinctly separated fraudulent embeddings from genuine ones. The approach also offers high explainability by surfacing dense clusters of shared physical devices and addresses ongoing challenges in real-time prediction and noisy connections.
The Shop Store team implemented a server-driven UI architecture for the Shop App’s Store Screen to personalize sections and layouts for different merchants. Previously, the client rendered a fixed layout, limiting customization, experiment timing, and the speed of fixes after weekly app releases. The new design uses template processing, orchestration, data loading, and a shop-server GraphQL layer, with ProductsSection and CollectionsSection types and configurable GridLayout or ShelfLayout options. On the client, ServerDrivenStoreScreen and StoreSectionContainer render the sections returned by the server, while default layouts allow older app versions to handle unfamiliar section or layout types. The architecture enables backend-controlled experiments and section changes without requiring an app release, while giving the product team a system intended to grow with the store experience.
During the infrastructure work for a marketing tool, Shopify's team found a BigQuery query that would have processed about 75,462,743,846 bytes per request and cost nearly $1 million monthly at an estimated 60 requests per minute. The query served a pipeline ingesting one billion rows through Apache Flink, with state managed by RocksDB and streaming requests from Apache Kafka; scaling beyond the release made ingestion unsustainable. To support general availability, the team evaluated an external SQL warehouse that could load Parquet atomically, handle 60 requests per minute, and export results to Google Cloud Storage, then clustered a dataset on two feature columns used in WHERE clauses. Running the same query on the clustered table reduced billed data to 508.1 MB and identified 108.3 MB scanned, lowering estimated monthly cost to about $1,370.67; the post also recommends selecting needed columns, partitioning tables, and using free previews instead of exploratory queries.
Dedicated vehicular inertial measurement units are prohibitively expensive to install across large fleets, while customer feedback lacks the coverage required to assess driving safety comprehensively. To monitor driver behavior at scale, Grab processes smartphone sensor data collected via a mobile telematics SDK at high sampling frequencies. A custom orientation algorithm calculates Euler roll and pitch angles from gravity and resolves yaw through principal component analysis to align phone measurements with vehicle axes. The pipeline applies low-pass or Kalman filtering to handle varying noise levels, followed by threshold peak-finding algorithms to detect harsh acceleration, braking, and cornering. Speeding events are identified by comparing GPS speeds against known map limits or aggregated speed profiles derived from free-flowing traffic.
Shopify’s Server Pixels service validates, processes, augments, and produces more than one billion customer events daily, but rising traffic caused Kafka input-topic consumption lag when downstream processing slowed. Its original design could spawn an unlimited number of goroutines while processing or producing event batches, consuming CPU and memory and threatening the five-minute event-processing SLO. The service was redesigned around fixed-size Go worker pools: workers receive Job values through channels, process them via reusable work functions, and use wait groups and context cancellation for orderly shutdown. Load testing showed 15 workers in both processor and producer raised throughput to 12.9 thousand events per second per pod, while 50 workers reached 19.3 thousand; subsequent improvements reached 21 thousand, a 170% increase over 7.75 thousand, and the service handled a maximum of 46 thousand events per second during BFCM 2021.
Supabase announces the general availability of Realtime’s multiplayer features, Broadcast and Presence, alongside a new channel-based architecture and supabase-js support. Realtime originally used Postgres logical replication, rather than LISTEN/NOTIFY, to avoid the latter’s 8,000-byte payload limit and the latency of querying each change from the database. In the new design, Channels act as rooms that support ephemeral publisher/subscriber Broadcast messages, shared Presence state, and Postgres change events; Presence uses Phoenix Tracker, a delta-based CRDT, for eventually consistent synchronization and removes disconnected users automatically. Realtime now runs as a distributed Elixir and Phoenix cluster on Fly, with clients connecting to the nearest node and nodes communicating across a backbone network to improve connection speed, stability, and latency. Free-plan active projects and projects created from August 18 have access, while other projects are scheduled for migration over the following weeks.
Supabase announced an $80 million Series B in May, bringing total funding to $116 million, and reported growth to more than 150,000 hosted PostgreSQL databases and 110,000 developers. Community metrics also increased, with Discord membership rising from 4,000 to 8,000, Twitter followers from 18,000 to 37,000, and GitHub stars from 19,000 to 36,000. The round is led by Felicis, with Coatue, Lightspeed, and Square Peg Capital joining, while Supabase describes its open-source support through contributor payments, maintainer hiring, sponsorship, and community collaboration. The company is moving into its third phase, focused on cloud-native PostgreSQL capabilities including branching, scalable storage, distribution, ephemeral compute, snapshots, and time travel. It invested $500,000 in OrioleDB and hired a developer to work on it, while stating that OrioleDB is not currently used on the Supabase platform and that the company will not run a PostgreSQL fork.
The post distills lessons from five years working on Shopify’s payment infrastructure into guidance for building systems that tolerate failure and large-scale traffic. It frames the advice partly around onboarding developers who were new to payments or Shopify’s scale, and notes that load testing for BFCM 2021 began in July with globally distributed load generation. It recommends low timeouts across HTTP clients and data stores, circuit breakers such as Semian, and capacity reasoning with Little’s Law, rate limiting, and load shedding. Monitoring should cover latency, traffic, errors, and saturation, while structured, machine-readable logs make centralized records searchable; load tests use benchmark gateways that mimic production partner latency and capacity. For incidents, Shopify uses defined roles, Slack-based coordination, and retrospectives that turn incorrect assumptions into safeguards and training material.
Tracking unique post views within Postgres presents trade-offs between counter accuracy, row bloat, and concurrent write performance. To evaluate options under an architecture constraint preventing external dependencies, a benchmark suite generated synthetic users, skewed post distributions, and replayed view actions. The benchmark tested four implementations: a naive counter column, an hstore key-value approach, an association table, and HyperLogLog (HLL). Benchmark results demonstrated that simple-hstore achieved the lowest average latency among deduplicating approaches at 2.15 milliseconds, closely followed by HLL at 2.16 milliseconds. Despite hstore's raw performance in prototypes, HLL is recommended for production because it circumvents row bloat as view counts grow while avoiding expensive row counts.
Nextdoor built a proximity-based autocomplete service to power typeahead search and mention features across its hyperlocal platform for hundreds of millions of entities, including users and businesses. The system shards geographic data using Uber's open-source H3 geohashing library and stores prefix indexes in memory using Redis sorted sets. By adopting a Command Query Responsibility Segregation architecture, ingestion writes are processed on Redis primary nodes and replicated to read-only search nodes with under 10 milliseconds of replication lag. Dedicated APIs handle indexing, typeahead lookups, and ranking before returning hydrated results. Operating since August 2021, the service processes hundreds of millions of monthly typeahead queries while maintaining a P95 search latency below 30 milliseconds.
The best engineering reads, once a week.
