Loading…
Browse by type
Problem & solution reads
Problem & solution posts on Yomu, summarised newest first.
- 910 posts
- 20 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Problem & solution posts
Our vision for the ClickHouse Grafana plugin
ClickHouse is widely used with Grafana for observability and real-time analytics, but requiring users to write raw SQL creates onboarding and exploration friction. To simplify the user experience, planned updates to the ClickHouse Grafana plugin introduce native interfaces such as click-to-filter actions, categorized OpenTelemetry attribute displays, and guided variable editors. A new compact search mode leverages the hasToken function for fast log filtering, while a visual metrics builder simplifies aggregating OpenTelemetry and Kubernetes metrics without manual queries. Prebuilt dashboards for logs and traces will accelerate initial setup alongside automated annotations derived from service version changes and Kubernetes lifecycle events. Future prototypes also explore bidirectional SQL AST parsing for seamless query editing, JWT user identity forwarding for fine-grained access control, and conversational query generation.
Amplitude ships 3x more production code with Cursor
Amplitude’s engineering team built toward an autonomous development pipeline that moves software from idea to production with minimal developer intervention, using Cursor to connect customer feedback, observability, code reviews, and other lifecycle context to agents. Cloud agents run in isolated, scalable VMs with complete development environments, enabling parallel, long-running, self-testing work that local machines could not support at the same scale. Automations now turn Slack bug reports and feature requests into Linear tickets and pull requests, perform hourly legacy migrations, and classify pull-request risk, while Bugbot reviews and autofixes low-risk changes. Since adopting cloud agents, Amplitude reports a threefold increase in weekly production commits, more than 1,000 unprompted agent runs per week, and Cursor as a top-three contributor by commit volume. About 60–70% of low-risk pull requests merge without additional developer work, while high-risk changes are routed to appropriate reviewers; the next goal is automating CI/CD, build validation, and deployment.
Autoresearch isn’t just for training models
Repeated visual-regression failures made Polaris CI feedback cycles take about 30 minutes, prompting an attempt to optimize build time rather than fix individual tasks. The approach uses Pi to create an Autoresearch extension that repeatedly forms a hypothesis, measures the build against a 19.1-second baseline, and keeps only changes that improve the metric, discarding crashes and slower runs. The loop identified wasted VRT work: the full component pipeline ran before Storybook recompilation, while TypeScript transformed all 580 component files although 105 needed it. Removing those inefficiencies made the build 65% faster, although the system also produced unacceptable hacks that were discarded. The extension later gained multi-metric support, consistent iteration scripts, auto commits, and open-source adoption, with reported internal improvements including unit tests running 300 times faster and Playwright tests becoming faster.
Agentic coding with ClickHouse. One person, one data stack, one full-stack application
A Solutions Architect at ClickHouse built ClickShop, a full-stack retail analytics platform, within a few days using Cursor to demonstrate production-level capabilities beyond static slides. The platform pairs ClickHouse for sub-second analytical queries across billions of rows with PostgreSQL for transactional workflows, using ClickPipes for real-time change data capture synchronization. LibreChat orchestrates 18 persona-tailored AI agents, while dual observability layers—ClickStack for OpenTelemetry infrastructure metrics and Langfuse for LLM prompt and cost tracking—store their telemetry directly in ClickHouse. Developing with an AI-assisted IDE was accelerated by open-source documentation and community skills, though it required continuous human review to handle unintended code refactoring and token consumption.
Mintlify boosts NPS 30% and saves 60% with real-time analytics on ClickHouse Cloud
Documentation platform Mintlify experienced major dashboard performance degradation and rate limiting under PostHog as multi-tenant analytics traffic grew rapidly. Because PostHog lacked incremental materialized views, queries scanned full raw datasets and took tens of seconds to load. Mintlify migrated its customer-facing analytics infrastructure to ClickHouse Cloud, ingesting events through Amazon MSK and ClickPipes into a single events table backing incremental materialized views. The team executed the transition over two weeks by dual-writing events and replicating the existing PostHog schema to preserve API surface compatibility. The migration reduced dashboard query latency to sub-one-second, decreased related support bug reports to zero, lifted NPS by an estimated 30%, and lowered analytics infrastructure costs by approximately 60%.
Managing context in long-run agentic applications
Long-running multi-agent systems struggle with context management because accumulating raw message histories degrades inference quality, increases latency, and exceeds context window limits. In a collaborative security investigation platform, passing unrestricted history can also introduce confirmation bias across specialized agents. To maintain coherence across unbounded rounds of investigation, the system eliminates raw message history carryover between invocations. Instead, it coordinates agents through three structured context channels: a Director's Journal for orchestration memory, a Critic's Review that scores findings to filter hallucinations, and a Critic's Timeline of validated chronological events. This architecture provides agents with tailored context without overwhelming their inference capacity.
Automating Receipt Collection: Apple Intelligence for On-Device Inference
Ramp built automatic receipt detection for its iOS app to reduce the effort of finding receipt photos while keeping private images on device. An initial design used PhotoKit, Vision OCR, background processing, local storage, and exact checks for merchant, amount, and date, but formatting differences, aliases, and OCR errors produced many false negatives. FoundationModels enabled a local structured-output LLM, yet a single prompt performed poorly in evals because the small 4096-token model hallucinated and took shortcuts. Splitting the merchant, date, and amount questions and requesting rationale improved evaluation results, but the alpha release reached 66% precision and about 18% recall. Version 3.0 combined one LLM merchant-name check with deterministic iOS 26 data detectors for dates and monetary values; it cut processing time, more than doubled recall, and raised precision to 87%.
Using AI agents to build Stripe-Salesforce integrations faster
An experiment using Claude Code to build a Stripe App for Salesforce Flow exposed context gaps: the generated integration misnamed fields, targeted a nonexistent custom field, and omitted AsyncAfterCommit, causing a “Callout from triggers not supported” deployment error. The proposed remedy is a repository of grounding context files that present invocable signatures, request models, platform constraints, LWC patterns, and validated examples in concise, machine-readable form. Core files are about 35K tokens, while the full reference catalog is about 230K tokens and can be loaded on demand. With the context loaded, the agent selected the v02_CreateCustomers invocable directly and generated a helper Apex class for nested Checkout Session inputs, along with callout-safe Flow structure and other safeguards. The author reports minutes instead of hours and fewer iterations in personal usage, while noting that org-specific details still require human verification.
Re-imagining ML Serving Infra: My Winter Internship at Ramp
Ramp’s ML Platform team needed to separate flexible model development in ml-pipelines from the Intelligence Platform’s 99.9%+ uptime and low-latency requirements. The internship project, Bring-Your-Own-Model-Weight Serving (mw-serve), built a dedicated serving layer that pulls model weights from MLflow and a training-environment base image from ECR, then runs each model in an isolated instance. To protect uptime, it added deployment safeguards, pre-traffic liveness checks, and monitors, while latency work instrumented client TCP/TLS connections and server processing. The investigation traced extreme tail latency to connection churn, which improved with longer keepalives and startup warmups, and to garbage-collection pauses associated with memory drops; tuning thresholds and freezing objects removed more outliers. By the final day, one Intelligence Platform model was serving production traffic through mw-serve, with additional models in shadow testing.
The cure for "execution tax" | Notion
Brainlabs CEO Dan Gilbert describes “execution tax” as the gap between a decision and its outcome, where meeting action items are recorded, chased, or lost. He argues that AI has improved engineering productivity while leaving strategists, account managers, and operators with fragmented context and a wide gap between thinking and doing. Brainlabs chose Notion as a company-wide context layer and built “Get Stuff Done,” a factory of Notion Custom Agents that captures action items, assigns owners, and executes suitable work. Its automation uses Notion Custom Agents for native tasks, Notion Workers for external integrations, company-specific Skills and selected LLMs for complex tasks, and human-in-the-loop workflows when judgment remains necessary; Gilbert says some decisions now become completed work in minutes rather than weeks.
Selective Test Execution at Stripe: Fast CI for a 50M-line Ruby monorepo
Stripe’s Ruby monorepo contains about 50 million lines, 100,000 test files, and roughly 1.2 million test units, making sequential execution impractical at an estimated four months. To keep CI within minutes, Selective Test Execution (STE) runs tests whose recorded file accesses intersect files changed since a suitable baseline, averaging about 5% of the suite and under 10% of always-run compute. A dynamically linked C++ library loaded with LD_PRELOAD intercepts file opens, associates them with hierarchical test scopes, and propagates tracing to child processes; logs are aggregated into a selection index outside the syscall hot path. Guardrails cover tests affected by directory discovery, previously failing tests, selective linting, generated files, and reproducible database-backed baselines ordered by Monotonic Revision IDs. The system is presented as a way to preserve safety and affordability while scaling CI for a highly dynamic language.
The three villains to agentic observability: retention, sampling and rollups
Observability practices increasingly depend on data sampling, short retention windows, and rollups to control storage costs. While human operators historically compensated for these compromises with intuition, AI agents lack implicit operational context and require full-fidelity telemetry to diagnose issues and justify conclusions. Head sampling drops unobserved spans early, tail sampling risks missing implicit non-error anomalies, and aggressive log expiration prevents long-term cyclical trend analysis. Addressing these constraints requires decoupling storage and compute via high-compression object storage and dedicated compute pools. Deploying real-time analytical databases such as ClickHouse enables high-concurrency, low-latency SQL querying, log clustering, and structured in-database summarization necessary for automated reasoning loops.
How Eisan made POS analytics faster, cheaper, and more reliable with ClickHouse Cloud
Eisan System Development moved a retail analytics workload from a QlikView-based platform to ClickHouse Cloud after rising ID-POS data volumes caused slower processing, failed analyses, and increasing licensing and server costs. Client records from supermarkets and drugstores support basket analysis, cross analysis, and real-time personalization, with some datasets nearing 200 million records annually. The team compared open-source options including PostgreSQL and YugabyteDB, then tested ClickHouse configurations, including a 30-core, 120-GB setup, while accounting for cache memory usage through load testing. In comparison, ClickHouse ran on slightly lower-spec hardware under high-load conditions without observed performance deterioration, and Eisan reports no cases so far where it failed to return a result; the team is now targeting 100 billion records and exploring off-hours processing and scheduled scaling.
MongoDB Predictive Auto-Scaling: An Experiment
MongoDB Atlas initially relied on a reactive auto-scaler that took minutes to adjust instance sizes and only transitioned between adjacent tiers. Because these delays caused transient server over-utilization or unnecessary customer costs, researchers prototyped predictive auto-scaling to forecast demand. The experimental architecture used short-term and long-term forecasters for customer-driven metrics, an estimator using boosted decision trees to project CPU utilization, and a planner to pick the most cost-effective tier. In testing, the prototype kept replica sets closer to target CPU ranges while saving an average of nine cents per replica set hourly. MongoDB subsequently implemented a production version that scales replica sets up proactively before expected spikes while retaining reactive downscaling.
Zero downtime Upgrade: Yelp’s Cassandra 4.x Upgrade Story
Yelp's Database Reliability Engineering team upgraded over a thousand Cassandra nodes from version 3.11 to 4.1 without downtime. The upgrade delivered reliability, observability, and performance improvements, including Java 11 support, configurable guardrails, and fixed incremental repairs. To avoid the high costs, consistency compromises, and data streaming delays of standing up separate datacenters, the team selected an in-place rolling upgrade strategy. The rollout orchestrated staged node updates alongside compatible versions of the Stargate proxy and custom data pipeline connectors without requiring client code modifications. Although Stargate 2.x introduced regressions on range and multi-partition queries that required downgrading to 1.x, the finished migration achieved up to a 58% reduction in p99 latency across key clusters.
How Respan is scaling LLM observability with ClickHouse Cloud
Respan scaled its LLM gateway and observability platform from hundreds of daily requests to 50 million daily events, causing its original Django and Postgres architecture to stall under transaction contention at 50 to 100 requests per second. To overcome Postgres write-ahead logging bottlenecks, the team migrated the ingestion and analytics pipelines to ClickHouse Cloud. Engineers minimized row sizes by truncating large prompt text and writing structured metrics to MergeTree tables, then built incremental minute- and hour-level materialized views alongside scheduled refreshable views for user metadata. Distributed tracing spans are ingested asynchronously with analytical parent-child resolution, utilizing functions like argMax rather than the FINAL modifier to prevent costly table merges. This architectural shift allows Respan to achieve sub-second search across tens of millions of records and support customer dashboards with predictable query performance.
How Notion catches breaking schema changes before they reach production
Schema drift in typed API requests and queue tasks can cause incidents when contracts change without detecting whether old payloads remain valid. Notion's CI guardrail extracts schema types from the base branch and compares them with PR types using TypeScript's compiler API, checking whether the old type is assignable to the new one. The check auto-discovers registered API endpoints and queue task types, uses git archive for base files, and currently covers approximately 1,300 endpoints and 296 task types. Backward compatibility blocks changes that reject old data, while queue schemas also receive an advisory forward check for rollout ordering. The rollout began with warnings, then made backward checks blocking and added a GitHub label for intentional exceptions; type checking still cannot catch extra runtime validation or semantic behavior changes.
Rill and ClickHouse: real-time operational BI for a metered world
Granular operations across cloud computing, payment platforms, and artificial intelligence workloads produce high-volume event streams that traditional business intelligence tools struggle to unify and aggregate. Operational BI platform Rill addresses this challenge by combining declarative data loading through the dlt framework with ClickHouse's high-performance analytical aggregation engine. Data transformations, SQL-based measure expressions, and dashboard configurations are defined entirely as code using SQL and YAML. This declarative, metrics-first architecture allows developers to validate analytics locally, track changes in Git, and eliminate intermediate caching layers when querying live data. By querying ClickHouse directly and linking AI-generated insights back to underlying queries, the system provides traceable, real-time operational analytics for enterprise environments.
Headless Screenshot Testing in Thumbtack’s Android Apps
Thumbtack required a faster, emulator-free method to execute over 1,500 Android screenshot tests during continuous integration. Previously, running UI tests on physical devices or emulators proved too slow for code review verification jobs, causing tests to run post-merge and allowing regressions to persist. After evaluating Paparazzi and Google's experimental Compose Preview Testing, the team selected Roborazzi, which runs on top of Robolectric's native graphics engine within a standard JVM. This headless setup accommodates custom Activities, Dagger field injection, and legacy View systems while executing alongside standard unit tests. Moving screenshot tests to the unit test level enabled gating merges on passing tests, colocating tests inside feature modules, and increasing total CI job runtime by only twenty percent.
From Custom to Open: Scalable Network Probing and HTTP/3 Readiness with Prometheus
Slack encountered a lack of client-side observability when introducing HTTP/3 at its edge. Because HTTP/3 runs over QUIC on UDP instead of TCP, neither existing commercial SaaS tools nor the internal Prometheus Blackbox Exporter could probe the new endpoints. To address this gap, an engineering intern integrated the open-source quic-go library into Prometheus Blackbox Exporter to build a native, configurable HTTP/3 probe. Slack deployed an internal implementation using the upstream functionality while waiting for pull request reviews, ultimately landing the contribution upstream. The resulting setup provides a unified view of HTTP/1.1, HTTP/2, and HTTP/3 metrics within Grafana, enabling reliable alerting and faster correlation with other telemetry.
Top topics
- Architecture832
- Performance814
- LLMs658
- Machine Learning620
- Open Source563
- AI Agents549
- AI465
- Developer Experience448
Weekly digest
The best engineering reads, once a week.