Loading…
Browse by type
Problem & solution reads
Problem & solution posts on Yomu, summarised newest first.
- 891 posts
- 20 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Problem & solution posts
Scaling Verify with Wallet for Identity Verification at Uber
Uber integrated Apple’s Verify with Wallet API into its cross-app Identity Verification Platform to support government-backed digital IDs across driver onboarding, age-gated ordering, and car rental use cases. The central challenge was allowing each use case to request only its Apple-approved identity data while operating through a scalable, platform-agnostic service rather than separate integrations. Server-side configurations map Uber identity models to PassKit objects, set one-time or persistent storage intent, and validate responses using HPKE decryption, CBOR payloads, session transcripts, MSOs, device signatures, and issuer certificate chains under ISO/IEC 18013-5. A registry of IACA certificates, sourced from jurisdictional portals and AAMVA’s VICAL, supports overlapping certificate rotations, while the platform is intended to accommodate new document types and cross-platform digital credential integrations.
Synthetic Data Generation for Financial AI Research with NVIDIA NeMo
Real-world financial natural language processing datasets often overrepresent common events like earnings while underrepresenting rarer occurrences such as credit-rating changes. To address this imbalance, an iterative generation pipeline produced over 500,000 unique headlines across 13 categories using NVIDIA NeMo Data Designer, NeMo Curator, and Nemotron 3 Nano. Instead of relying on a single large batch that yielded 65 percent near-duplicates, the workflow executes iterative cycles of category-weighted generation, rule-based filtering, global semantic deduplication, and dynamic category reweighting. Farthest-from-centroid few-shot example selection guided subsequent rounds toward novel semantic spaces across 82 total iterations. Executed over approximately six days on an eight-way NVIDIA B200 node, the resulting corpus enabled fine-tuning compact student language models to achieve 95 percent of teacher model performance.
A Practical Guide to GPU-Initiated Communication for Molecular Dynamics at Scale
Molecular dynamics packages like GROMACS face severe scaling bottlenecks when CPU-orchestrated MPI communication introduces blocking synchronizations during halo exchanges. To resolve this, researchers implemented GPU-initiated communication using NVIDIA NVSHMEM to eliminate CPU-GPU handoffs. The redesigned approach fuses packing, remote memory access, and completion waiting into device-side kernels using fine-grained signaling. Benchmarks on NVIDIA DGX H100 systems demonstrated up to 1.5x performance gains intra-node and up to 2x on GB200 NVL72 multi-node clusters, particularly on latency-bound systems. While device-initiated communication introduces a slight 1–3% overhead on very large compute-bound configurations at low node counts, it consistently improves strong scaling for production workloads.
GitFarm: Git® as a Service for Large-Scale Monorepos
At Uber’s monorepo scale, traditional Git workflows create bottlenecks through multi-gigabyte clones, local checkouts, repeated synchronization, and independent fetch or push operations across automation systems. The post describes GitFarm, a Git as a Service platform exposed through a high-performance gRPC API that executes standard Git commands in secure, ephemeral sandboxes backed by pre-warmed repository and container pools. Each backend maintains a bare clone, receives push-based updates and periodic git fetch operations every five minutes, while clients can explicitly fetch when they need current state. The design provides a full checkout in under 500 milliseconds and reduced client-side resource utilization by over 80%, while avoiding local clones and long initial clone delays.
How Trigger.dev is using ClickHouse to scale observability for long-running AI workflows
Trigger.dev, an open-source platform for serverless asynchronous tasks, needed to scale observability for increasingly complex, long-running AI workflows spanning thousands of tenants. Its initial Postgres design stored logs, traces, and events in a growing table, making indexes, analytics, and operational reliability increasingly difficult to manage. The team moved observability workloads to ClickHouse after evaluating performance, scalability, open-source compatibility, deployment flexibility, and support for dynamic telemetry, including its native JSON data type. Migration occurred gradually at the task-execution level, routing each run’s logs to either database without downtime or customer intervention. ClickHouse now supports real-time telemetry analysis, approximately 200-millisecond P95 latency for queries returning tens of thousands of rows, and a dramatically smaller telemetry storage footprint.
Automating cross-repo documentation with GitHub Agentic Workflows
Maintaining documentation across separate repositories often leads to severe lag because technical writers must reverse-engineer shipped features weeks after release. To address this in the Aspire project, the team implemented an automated pipeline using GitHub Agentic Workflows to bridge the product and documentation repositories. When product pull requests merge, a bash step maps milestones to docs release branches before an LLM agent evaluates the diff, drafts documentation updates, and emits structured pull request intents. A dedicated safe-outputs handler materializes these drafts via a scoped GitHub App and assigns the original code reviewers to verify accuracy. Across 396 product pull requests, the system generated 82 documentation pull requests that all merged with a median turnaround time of 44.8 hours.
Navigating a Synapse Migration to Databricks
The post presents migration from Azure Synapse to Databricks as a way to reduce platform complexity while supporting data engineering, analytics, machine learning, real-time pipelines, and AI applications. It distinguishes migration paths for Dedicated SQL Pools, serverless SQL, Spark Pools, orchestration, governance, and downstream BI connectivity, noting that accumulated T-SQL logic and dependencies require the most redesign, testing, and validation. The proposed programme uses discovery and assessment with Lakebridge Profiler and Analyzer, then selects an architecture, sequences workloads, pilots a real use case, and scales migration in waves. It recommends automating code conversion, reconciling row counts, aggregations, hashes, and tolerances, avoiding direct Synapse distribution-to-Delta partition mappings, and planning compaction, enablement, and rollback. Reported examples include Casey’s halving operational data delivery times from eight hours to four and Italgas cutting workload costs by 73%.
How Imperial College London is accelerating dementia research with a modern data platform
Imperial College London’s Care Research and Technology Centre (CR&T) needed to scale Minder, a platform that combines in-home sensors, sleep monitors, and electronic health records to support dementia care and research. Growing data volumes caused competing workloads, tightly coupled storage and compute, and limited access for researchers and clinicians. The re-architected platform validates IoT data through Kubernetes, stores it in Delta Lake on Azure Data Lake Storage using bronze, silver, and anonymized gold layers, and adds Databricks analytics with Unity Catalog governance while retaining FHIR-based EHR interoperability and Kubeflow for model deployment. Research-to-production workflows, reusable pipelines, dashboards, and timestamped data improve collaboration, accessibility, and reproducibility. During migration, uptime remained 100%; new sources integrated in as little as one month, model development fell to about one month, and insights reached 581 people living with dementia in five months.
Building an Analysis AI Agent for Industrial Alarm Management with NVIDIA Nemotron
Industrial machinery produces high volumes of alarms and sensor readings that exceed manual triage capacity. To address this bottleneck, an end-to-end GPU-accelerated AI agent is designed using NVIDIA NeMo libraries, Nemotron open models, and the NVIDIA OpenShell runtime behind a single HTTP endpoint. When triggered with an alarm payload and sensor frame, the agent gathers historical context using cuDF, cuVS, and NeMo Retriever, queries data warehouses via Text-2-SQL, and executes specialist signal checks with tools like cuFFT and cuML. Nemotron 3 Nano manages orchestration tasks while Nemotron 3 Super synthesizes the gathered evidence. The workflow outputs a structured package containing an observation, root-cause hypothesis, remedy, and recommended action within a seconds-long latency budget.
Barracuda makes security logs conversational with Genie
Barracuda Managed XDR needed to make security-log investigations accessible beyond analysts who knew each vendor’s schema and could write SQL. Its AI-Powered Log Search integrates Databricks Genie into the XDR dashboard: Genie translates natural-language questions into SQL, runs them against customer logs, and returns tabular results with the generated query. Security-specific metadata in Unity Catalog helps map requests across network, server, cloud, email, and endpoint sources, while multi-turn context supports iterative filtering. Secure views enforce organization-level filtering before execution; scoped service principals and SQL validation add further controls against cross-tenant access. Barracuda reports reducing routine investigation time from hours to minutes and estimates that self-service could free 83 to 100 SOC analyst hours monthly, or 1,000 to 1,200 annually.
Enhancing Goodput in Large-Scale LLM Training with Nonuniform Tensor Parallelism
Large-scale LLM training jobs spanning thousands of GPUs frequently encounter transient hardware interruptions that bottleneck tightly coupled tensor parallel groups. Nonuniform Tensor Parallelism addresses this bottleneck by dynamically adapting the tensor parallelism degree within affected scale-up domains rather than dropping entire data replicas. When a GPU drops out, the system reconfigures the remaining functional devices to process larger shards and resumes execution from the latest checkpoint. To prevent the downsized replica from stalling global synchronization, advanced rack designs temporarily power-boost the active GPUs to increase clock frequencies and match baseline throughput. Overlapping the necessary tensor resharding with backward computation and parameter synchronization reduces overhead on healthy replicas to under one percent.
Scaling Security Alert Triage With Specialized Agents on Databricks
Databricks describes a security-alert triage system designed to review low-severity alerts that historically received less attention because of their volume and low fidelity. An initial single-agent prompt escalated 50% of alerts, so the team built a fleet of 17 source-specific agents plus a Threat Intelligence agent, using Structured Streaming to ingest alerts, enrich evidence, and route each alert. The pipeline combines deterministic benign-signal filtering, historical and behavioral context, specialized prompt functions, optional tools, structured dispositions, and cost controls before sending escalations to human analysts. Those analysts provide ground truth through MLflow traces and labels, supporting ongoing evaluation across escalate, monitor, and close decisions. After triaging more than 18,000 alerts, the system reached a 3.2% escalation rate, a 10.5-second median triage time, and more than 6,500 analyst hours saved in 30 days; escalated low-severity alerts were roughly 10 times more likely to be true positives than HIGH and MEDIUM alerts.
98% faster queries, 50% lower cloud costs: Verihubs’ journey from Postgres to ClickHouse
Verihubs rebuilt its data warehouse after a Postgres-based, daily-batch pipeline became slow, unreliable, and difficult to correct when late vendor callbacks required backfills. The new architecture uses Kafka, Kafka Connect, and Debezium for real-time ingestion into ClickHouse, with MergeTree for append-only logs and ReplacingMergeTree for datasets updated through change data capture. Materialized views create daily and monthly rollups, projections accelerate existing query patterns, and Dagster handles complex detection workflows that can notify Slack. The migration, completed in about six months, reduced query times from 20–30 minutes to two or three seconds on datasets up to 18 million rows, representing as much as 98% faster queries and 50% lower infrastructure costs. Verihubs says the resulting real-time OLAP system gives finance and operations fresher data for traffic analysis, reconciliation, performance reporting, and invoicing.
Migrating Counter Service storage: Design choices and learnings
Grab’s Counter Service supports time-windowed fraud queries at tens of thousands of QPS and about a billion requests per day, but an infrastructure review prompted migration from a wide-column database to Aerospike. The team first separated Rust business logic from storage, then added an enum-dispatched facade with Single, WithShadow, and WithSplit modes so traffic could be compared and shifted without downtime. They also redesigned the writer path around Aerospike, including a map-based schema and a single atomic increment, while testing index placement and reverting from an NVMe-backed primary index after hot-key p99 spikes. Production p99 read latency improved by roughly 50%, disk usage fell from about 3 TB to 1 TB, and per-node cost dropped 45–50%; the rollout reported zero downtime and no data-integrity issues.
Presto® on Apache Kafka® At Uber Scale
Uber connected Presto and Apache Kafka to provide lightweight, interactive SQL queries over Kafka topics for ad-hoc analysis of real-time data. The work addresses operational requests such as checking whether an order UUID appears in a service’s input or output streams, where streaming processors are poorly suited to historical point lookups and real-time OLAP requires onboarding and dedicated resources. The enhanced connector discovers Kafka clusters, topics, and schemas on demand, supports multiple Kafka clusters, caches metadata, enforces _timestamp or _partition_offset filters, and uses a shared Kafka client ID with broker quotas to control consumption. After rollout, example lookups returned within a few seconds, daily usage reached 6,000 queries, and the teams reported improved productivity, while concurrent queries may take longer under quota control.
Building Zone Failure Resilience in Apache Pinot™ at Uber
Uber describes how it made Apache Pinot clusters resilient to zone failures while preserving query and ingestion availability for Tier-0 use cases. It maps Uber’s virtual resource isolation group IDs to Pinot server pools, uses pool-based instance assignment with replica-group segment assignment, and requires at least as many isolation groups as the table’s replication factor. During migration, automated pool registration, configuration backfills, granular APIs, selective rebalancing, and rollback support were used across more than 400 Tier 0-1 clusters and thousands of tables. An internal drill found that one remaining replica continued serving traffic without query degradation after a zone went down, while isolation-group-based claim and release policies enabled parallel restarts and sequential rollouts with baking time.
How we made ClickStack 5x faster for ClickHouse observability
ClickStack’s inherited OpenTelemetry Collector schema was struggling with slow searches, Top N queries, and aggregations that could time out on large observability workloads. The team prioritized common log and trace access patterns, including exact-attribute searches, body-text queries, histograms, individual log lookups, and distinct-key or value introspection. They used fixed production-like workloads and iterative testing with ClickCannon, Kubernetes-managed environments, ephemeral ClickHouse Cloud instances, and controlled schema, query, and version comparisons. The resulting redesign combined primary-key changes, text indexes, query rewrites, alias columns, materialized views, and newer ClickHouse features, reducing representative query latency by more than 5× while balancing ingestion, storage, and operational efficiency.
Any Apple update can break our app. Here's how we find out first.
Usage is an Apple ecosystem resource monitor that relies on undocumented APIs vulnerable to silent data structure changes across OS updates. To maintain stability across iPhone, iPad, and Mac without degrading device performance, the development team uses Sentry across separate component projects in a unified workspace. Beyond capturing standard crashes, the team wraps Swift errors with unexpected raw data payloads to quickly diagnose undocumented API structure drift while scrubbing sensitive user values. They also employ Sentry Size Analysis as a regression gate to monitor release payload deltas, preventing unintended asset omissions or bloat from slipping through code reviews. This multi-layered monitoring setup enables the team to detect and resolve platform-induced breakages before most end users encounter them.
Optimizing a Neural Reconstruction Pipeline Using NVIDIA Nsight Developer Tools
NVIDIA Omniverse NuRec reconstructs dynamic 3D environments from camera and lidar sensor data for autonomous vehicle simulation workflows, but high computational demands historically caused long reconstruction times. Engineers used NVIDIA Nsight Systems and Nsight Compute to systematically profile the PyTorch-based training pipeline and optimize its underlying CUDA kernels. Nsight Systems revealed GPU underutilization caused by numerous small kernels and blocking synchronization calls, leading to a fused interpolate kernel that accelerated execution from 4.184 milliseconds to 83.81 microseconds. Profiling with Nsight Compute uncovered resource mismatch in the renderBackward kernel, prompting the team to split it into dedicated camera and lidar implementations with tuned register and shared memory allocations. These adjustments raised kernel occupancy from approximately 15% to between 30% and 50% while decreasing the longest lidar kernel runtime from 31 milliseconds to 18 milliseconds.
A Quadrillion Rows across three Clouds: scaling LogHouse
LogHouse now stores 431 PiB of uncompressed data across 1.59 quadrillion rows, spans more than 30 regions across three cloud providers, and serves typical queries in a few hundred milliseconds. The platform grew 23x from 19 PiB in two years, while peak ingestion reached 80 GiB/s and 190 million rows per second. Its write architecture uses local geosharding, multiple cells in busy regions, and ClickHouse Async Inserts to combine small incoming writes into larger parts without overwhelming merges. On the read side, a single Distributed table hides the cell topology, while unavailable shards remain visible by default rather than producing silently incomplete results. Current work includes reducing index and metadata memory, improving async-insert durability, collecting system data from object storage, and expanding OpenTelemetry traces and metrics.
Top topics
- Architecture830
- Performance812
- LLMs648
- Machine Learning617
- Open Source561
- AI Agents542
- AI454
- Developer Experience446
Weekly digest
The best engineering reads, once a week.