Loading…
Browse by type
Problem & solution reads
Problem & solution posts on Yomu, summarised newest first.
- 916 posts
- 20 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Problem & solution posts
Building Zone Failure Resilience in Apache Pinot™ at Uber
Uber describes how it made Apache Pinot clusters resilient to zone failures while preserving query and ingestion availability for Tier-0 use cases. It maps Uber’s virtual resource isolation group IDs to Pinot server pools, uses pool-based instance assignment with replica-group segment assignment, and requires at least as many isolation groups as the table’s replication factor. During migration, automated pool registration, configuration backfills, granular APIs, selective rebalancing, and rollback support were used across more than 400 Tier 0-1 clusters and thousands of tables. An internal drill found that one remaining replica continued serving traffic without query degradation after a zone went down, while isolation-group-based claim and release policies enabled parallel restarts and sequential rollouts with baking time.
How we made ClickStack 5x faster for ClickHouse observability
ClickStack’s inherited OpenTelemetry Collector schema was struggling with slow searches, Top N queries, and aggregations that could time out on large observability workloads. The team prioritized common log and trace access patterns, including exact-attribute searches, body-text queries, histograms, individual log lookups, and distinct-key or value introspection. They used fixed production-like workloads and iterative testing with ClickCannon, Kubernetes-managed environments, ephemeral ClickHouse Cloud instances, and controlled schema, query, and version comparisons. The resulting redesign combined primary-key changes, text indexes, query rewrites, alias columns, materialized views, and newer ClickHouse features, reducing representative query latency by more than 5× while balancing ingestion, storage, and operational efficiency.
Any Apple update can break our app. Here's how we find out first.
Usage is an Apple ecosystem resource monitor that relies on undocumented APIs vulnerable to silent data structure changes across OS updates. To maintain stability across iPhone, iPad, and Mac without degrading device performance, the development team uses Sentry across separate component projects in a unified workspace. Beyond capturing standard crashes, the team wraps Swift errors with unexpected raw data payloads to quickly diagnose undocumented API structure drift while scrubbing sensitive user values. They also employ Sentry Size Analysis as a regression gate to monitor release payload deltas, preventing unintended asset omissions or bloat from slipping through code reviews. This multi-layered monitoring setup enables the team to detect and resolve platform-induced breakages before most end users encounter them.
Optimizing a Neural Reconstruction Pipeline Using NVIDIA Nsight Developer Tools
NVIDIA Omniverse NuRec reconstructs dynamic 3D environments from camera and lidar sensor data for autonomous vehicle simulation workflows, but high computational demands historically caused long reconstruction times. Engineers used NVIDIA Nsight Systems and Nsight Compute to systematically profile the PyTorch-based training pipeline and optimize its underlying CUDA kernels. Nsight Systems revealed GPU underutilization caused by numerous small kernels and blocking synchronization calls, leading to a fused interpolate kernel that accelerated execution from 4.184 milliseconds to 83.81 microseconds. Profiling with Nsight Compute uncovered resource mismatch in the renderBackward kernel, prompting the team to split it into dedicated camera and lidar implementations with tuned register and shared memory allocations. These adjustments raised kernel occupancy from approximately 15% to between 30% and 50% while decreasing the longest lidar kernel runtime from 31 milliseconds to 18 milliseconds.
A Quadrillion Rows across three Clouds: scaling LogHouse
LogHouse now stores 431 PiB of uncompressed data across 1.59 quadrillion rows, spans more than 30 regions across three cloud providers, and serves typical queries in a few hundred milliseconds. The platform grew 23x from 19 PiB in two years, while peak ingestion reached 80 GiB/s and 190 million rows per second. Its write architecture uses local geosharding, multiple cells in busy regions, and ClickHouse Async Inserts to combine small incoming writes into larger parts without overwhelming merges. On the read side, a single Distributed table hides the cell topology, while unavailable shards remain visible by default rather than producing silently incomplete results. Current work includes reducing index and metadata memory, improving async-insert durability, collecting system data from object storage, and expanding OpenTelemetry traces and metrics.
Agentic Risk Operations
Ramp describes an architecture for scaling risk operations as payment volume, payment rails, partners, and country coverage expand, rather than letting headcount grow with the business. Agents handle universal intake, gather business context, classify requests, and route work, while machine-learning models trained on millions of historical data points and approved policies make autonomous risk decisions. Operators can modify agent skills and tool configurations without engineering involvement, deploy changes in shadow mode, and rely on asynchronous execution, provider failover, and centralized observability. In payment-risk workflows, structured operator feedback produced a benchmark of more than 1,000 operations covering tool-call trajectories, operator alignment, and downstream outcomes, enabling separate evaluation of agents and policies. Ramp rolls agents out beside operators, then scales operation types and dollar volume using performance thresholds and capped exposure budgets.
Clever ingests 200x more logs at the same cost with ClickHouse Cloud
Clever, an identity platform for K-12 education, faced a fragmented logging setup as its 400 services and 200 AWS Lambdas generated about 150 TB of uncompressed logs monthly. Datadog indexed only about 10% and retained it for three days, while the remainder was in Athena and required SQL. Clever migrated to ClickHouse Cloud on AWS, indexing all logs with 60-day retention and reporting 200× more searchable log volume at no additional cost, with 10× compression. Its design extracts frequently queried JSON fields into columns, stores remaining attributes in a map alongside raw JSON, uses skipping indexes, date partitions, a 60-day TTL, and reverse Unix timestamp ordering; this ordering produced an 8–10× query speedup. A custom query layer translates LogQL to SQL, while separate development and production services and multi-region deployment address isolation and availability.
How Jua delivers the world’s most accurate physics simulations 3x faster with ClickHouse Cloud
Jua uses ClickHouse Cloud to deliver data from EPT-2, its physics foundation model, to energy traders and grid operators faster than its previous file-based pipeline. Jua’s forecasts combine outputs from multiple models and add roughly four terabytes of new data daily, producing about one petabyte compressed and over 10–12 petabytes uncompressed. Replacing Zarr and other file-based storage plus manual pre-fetching with ClickHouse Cloud reduced flagship forecast delivery from one hour to 20 minutes, with about 20 seconds of upload overhead. The platform also cut compute costs by roughly a third, reduced two-year single-location historical queries from hours to seconds, and supports both point lookups and regional aggregations; Jua selected the managed service for operational simplicity, separated storage and compute, and cost efficiency.
How to Govern Autonomous Agents in Enterprise AI Factories
Autonomous AI agents require governed environments to safely inspect code, execute tests, query enterprise systems, and handle sensitive business data. The NVIDIA Secure Agent Workspace Reference Design shifts agent execution away from user laptops to dedicated, single-user virtual machines managed on Red Hat OpenShift Virtualization or Microsoft Azure. Perimeter defenses enforce enterprise single sign-on access broker connections, default-deny network egress rules, and human approval gates for critical state-changing actions. Inside the workspace virtual machine, active sandboxing through NVIDIA OpenShell evaluates signed security policies before each tool invocation while credential proxies prevent direct access to raw secrets. GitOps workflows continuously reconcile infrastructure profiles and network configurations while all operational and security telemetry streams to centralized SIEM platforms in OCSF format.
GenPage: Towards End-to-End Generative Homepage Construction at Netflix
Netflix traditionally constructs its structured two-dimensional homepage through a complex, multi-stage recommender pipeline that separates candidate generation and ranking across rows and entities. To simplify this architecture and optimize directly for whole-page user satisfaction, Netflix developed GenPage, an end-to-end generative transformer model that autoregressively builds the entire homepage from raw tokenized context. The system relies on a domain-specific tokenizer to compress engagement history, context injection for cold start, hybrid row decoding to minimize decoding steps, and reinforcement learning post-training. In online A/B testing against the production baseline, GenPage delivered statistically significant gains in core user engagement metrics while cutting end-to-end serving latency by 20 percent. Offline evaluations further showed that enriching context prompts improved recommendation quality more effectively than increasing model capacity in the current operating regime.
How Databricks is turning video into searchable, actionable intelligence
Databricks presents video analysis as a data engineering problem for organizations with terabytes of footage that is difficult and expensive to review manually. An app accepts a video and a natural-language prompt, then triggers a Lakeflow job on Serverless GPU Compute to run Meta’s SAM3 segmentation model frame by frame and retain matching moments. Those clips preserve original timestamps and are sent through the Databricks Foundation Model API for summaries that can be written to tables or passed into downstream workflows. In one example, 26 minutes of traffic footage became one minute and 55 seconds of relevant video. The model-agnostic pipeline uses MLflow signatures to support interchangeable or custom models, while event-driven execution and independent serverless GPUs allow concurrent processing without cluster management or idle GPU costs.
How the English Office for Students leverages Databricks to enhance higher education standards and drive better student outcomes
The Office for Students, which regulates more than 400 higher education providers in England, modernised its data and analytics environment after a legacy platform could no longer handle growing volumes, varied sources, or emerging analytical demands. Its data spans millions of student records collected over 15 to 20 years, and a workflow processing about 300 million records took eight hours. Moving to Databricks consolidated structured, qualitative, and near-live data with analytics and AI workflows, while Unity Catalog added lineage, access controls, and security patterns for governed use. Genie Code reduced a student segmentation analysis from at least two weeks for two analysts to half a day, and a provider-registration triage proof of concept flags missing submissions earlier. The organisation frames AI as decision support rather than decision-making, keeping humans responsible for regulatory judgments.
Creating the NVIDIA Nemotron 3 Ultra NVFP4 Checkpoint with NVIDIA Model Optimizer
Moving large model weights efficiently requires quantization, but mapping weights to NVFP4's eight positive values risks severe precision loss or outlier-induced underflow. To address this, NVIDIA quantized the 550B Nemotron 3 Ultra model using NVIDIA Model Optimizer with a multi-precision strategy rather than quantizing all layers uniformly. The recipe applies four-over-six adaptive block scaling to Mixture of Experts routed weights, while assigning FP8 to shared experts and retaining BF16 for sensitive attention linears and embeddings. This process reduced the checkpoint size from 1,121 GB down to 352.3 GB and enabled single-checkpoint execution across Hopper using W4A16 and Blackwell using native W4A4. As a result, the model achieves up to 5.9x higher inference throughput than the GLM-5.1 754B FP4 model on decode-heavy workloads while matching BF16 benchmark accuracy.
Achieving Near-Linear Training Scalability for Pinterest’s Foundation Models
Pinterest’s foundation ranking models store approximately 99% of their parameters in embedding tables, causing cross-node NCCL all-to-all communication to severely bottleneck multi-node distributed training. Initial multi-node scaling stalled at 1.21x on four nodes even after adopting AWS Elastic Fabric Adapter for OS-bypass networking. To resolve this, engineers profiled GPU activity with PyTorch Profiler and systematically addressed communication bottlenecks across multiple layers. The team implemented FP8 quantized communications using FBGEMM, balanced table sharding across GPUs, halved embedding dimensions while doubling row counts, and adopted a 2D parallel communication topology alongside PyTorch Distributed Checkpoint. These combined interventions achieved near-linear scaling factors of 3.9x on four nodes and 7.5x on eight nodes across 64 GPUs.
Why we rewrote WAL-G for Postgres backups in Rust: Meet WAL-RUS
ClickHouse Cloud built WAL-RUS, an open-source Rust implementation of PostgreSQL backup and WAL archival tooling, after encountering difficulty predicting WAL-G’s memory usage in resource-constrained environments. The project preserves WAL-G’s core functionality and compatibility while using bounded worker pools, controlled concurrency, persistent object-storage connections, and a streaming pipeline that limits buffering and data copies. It uses the same WALG_ configuration variables, and WAL-G and WAL-RUS can read each other’s archives. In a sustained WAL-heavy benchmark with four archival workers, WAL-RUS stayed below 1 GB of peak virtual memory versus nearly 2.8 GB for WAL-G, while both maintained minimal backlog and comparable CPU usage. The authors plan to make WAL-RUS the default backup and archival mechanism for ClickHouse Cloud’s managed Postgres offering and are working to upstream its Postgres 17 wal summaries support to WAL-G.
How Vibe.co handles billions of ad impressions with ClickHouse Cloud
Vibe.co, recently acquired by Walmart, uses ClickHouse Cloud to provide real-time campaign reporting for thousands of Connected TV advertisers across billions of ad impressions. It migrated from Postgres after aggregation-heavy OLAP queries made its pre-aggregation layer a bottleneck, limiting interactive slicing and requiring new jobs for reporting use cases. The team evaluated self-hosted ClickHouse, another managed option, and ClickHouse Cloud, choosing the latter for operational simplicity, support from ClickHouse engineers, and separation of storage and compute with more predictable costs. Vibe uses refreshable materialized views for hot/cold tiering, keeping the last 30 days in a hot table; stored data grew from about 100 GB to over 2 TB without rearchitecting, while more than 90% of client campaign reports respond in under 100ms.
Modal's serverless Servers
Modal introduces Servers, a low-latency HTTP serving solution designed for applications like LLM inference where milliseconds matter. Servers reduce p50 latency from 39ms to 6ms compared to Modal Web Functions by removing control-plane lookups and queuing from the hot path. The architecture uses a stateless proxy (fprs) for domain association and load balancing, with Envoy for TLS termination and HTTP/2 multiplexing. Two design principles governed development: maximizing resource sharing while minimizing interference, and making no network calls on the request path. The system supports autoscaling, regionalized routing, proxy authentication, and mirroring for A/B testing and continual learning.
Automated Schema Evolution in Pinterest’s Next-Generation DB Ingestion Framework
Pinterest's CDC ingestion platform relies on a multi-stage pipeline built with Kafka, Flink, Spark, and Iceberg, where upstream schema updates risk disrupting data transformations and offline storage. To prevent pipeline failures and schema drift, Pinterest introduced an automated schema evolution framework based on a staged convergence model. Upstream changes are detected via push-based DDL messages and daily pull-based validation checks. The workflow automatically restricts automated updates to additive changes and numeric precision widening, updating Iceberg table definitions and regenerating Flink and Spark code through auditable pull requests. By decoupling schema divergence, code deployment, and data convergence, the system maintains continuous ingestion availability while working toward eventual schema consistency across all storage layers.
How Daikin Applied Americas builds consistent data pipelines at scale with Genie Code
Daikin Applied Americas needed to scale reliable data pipelines across growing analytics and AI use cases involving operational, manufacturing, and service data while coordinating development across teams. It adopted Databricks Genie Code within a structured operating model, using Unity Catalog context, reusable MECE skills, and explicit checkpoints across Bronze, Silver, and Gold layers to guide planning and execution. The framework defines competencies such as source grain, transformation patterns, canonical alignment, governance, and business-entity modeling, moving standards out of long prompts and into the development environment. The team reports that pipelines that once took days to prototype could be generated in minutes, with faster iteration, more consistent outputs, less structural correction, reduced architectural drift, and greater trust in AI-assisted results.
Accelerating BEV Pooling on NVIDIA GPUs for Physical AI Applications
Bird's-eye-view (BEV) pooling frequently becomes an inference latency bottleneck in spatial AI pipelines due to irregular memory access patterns and scatter-reduce operations. BEVPoolV3 addresses this by tailoring kernel execution to the GPU memory regime, specifically distinguishing between DRAM-bound and L2-cache-resident working sets. The implementation reduces redundant scatter traffic via depth-outer traversal, precomputes indices to eliminate runtime integer division, minimizes duplicate depth loads, and adopts interval-owned writes to bypass atomic operations. On a canonical 49 MB nuScenes working set, BEVPoolV3 accelerates TensorRT plugin execution from 274.0 µs down to 16.4 µs on an NVIDIA RTX PRO 6000 Blackwell Max-Q GPU. Furthermore, profiling with Nsight Compute reveals that L2-resident scatter-reduce operators achieve optimal efficiency with FP8, avoiding the ALU instruction overhead of NVFP4 decoding.
Top topics
- Architecture834
- Performance816
- LLMs660
- Machine Learning622
- Open Source563
- AI Agents553
- AI470
- Developer Experience448
Weekly digest
The best engineering reads, once a week.