Loading…
Browse by type
Problem & solution reads
Problem & solution posts on Yomu, summarised newest first.
- 891 posts
- 20 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Problem & solution posts
From "What Happened?" to "What Will Happen?"
Databricks Genie makes descriptive analytics accessible in natural language, but predictive questions still require specialized data science workflows and carefully prepared datasets. This post presents a multi-agent supervisor deployed as a Databricks App, combining Genie, TabPFN, and Agent Bricks to turn business questions into predictions. The orchestrator asks Genie to use governed Lakehouse data, schemas, relationships, and semantics to generate labeled training data through SQL, then sends it to TabPFN, which predicts in a single forward pass without feature preprocessing, model selection, or hyperparameter tuning. The resulting conversational experience supports descriptive and predictive analytics with Unity Catalog lineage and access control, while an MLflow GenAI evaluation harness monitors reliability and regressions. Its central limitation is that predictions depend on Genie producing a meaningful dataset with a clear label, so missing signals, joins, outcomes, or agent omissions can make results unreliable.
How Partition Access Visualizations Reduced our Data Lake S3 Cost by 33%
Data teams at Yelp struggled to identify table consumers and track how datasets were accessed across their petabyte-scale data lake. To solve this, the team built a usage attribution system using Amazon S3 server access logs to visualize time-based partition keys against access timestamps and IAM roles. These visualizations revealed distinctive patterns for daily batch jobs, backfills, and ad hoc queries, guiding migrations of thousands of tables to Apache Iceberg format. Furthermore, Yelp implemented a Default Access Retention strategy gated by IAM bucket policies to prevent accidental queries from resetting S3 Intelligent Tiering transitions. Combining these granular usage insights with targeted S3 storage classes reduced Yelp's S3 storage costs by 33 percent.
Unlock seamless and cost-effective marketing campaigns with Lakebase
Retail marketing teams often store customer segments in OLTP databases, creating underused capacity between campaigns and synchronization work whenever new segments are requested. Lakebase, Databricks’ implementation of a lakebase architecture, separates storage from compute and uses elastic, serverless Postgres that can scale to zero when idle and up for bursty campaign traffic. The integration with the Lakehouse uses managed Synced Tables for customer segments and Lakehouse Sync for continuous CDC-based replication of operational data into Unity Catalog Delta tables. The SAP Engagement Cloud example configures a Lakebase Autoscaling project, native Postgres credentials, an ISRG Root X1 certificate, and snapshot synchronization for datasets with more than 10% updated. The post concludes that this design lowers idle and sizing costs, reduces pipeline maintenance, and supports low-latency point lookups, while warning that Lakebase is not optimized for large scans or classic OLAP.
Databricks for Good and Virtue Foundation: Partnering to Connect Medical Volunteers to Critical Health Services in 72 Countries
Virtue Foundation’s VF Match platform connects medical professionals with volunteer opportunities across 72 low and low-middle income countries, but scaling its initial LLM proof of concept into production required a robust data pipeline. The Databricks-based Foundational Data Refresh combines Overture Maps and Bright Data, processes more than 25 million web pages with targeted GPT extraction steps, and uses Spark, Photon, and Lakeflow Jobs to orchestrate checkpointed, modular workloads. Splink resolves duplicate facilities and nonprofits across messy sources; Photon reduced the slowest entity-resolution partition from 30 minutes to approximately two minutes, a 15x improvement. A prototype VF Agent adds natural-language access through LangGraph, Databricks Model Serving, AI Search, and Genie, while the resulting system provides an up-to-date global view of healthcare infrastructure and volunteer needs.
Scaling reinforcement learning at Applied Compute
Applied Compute, founded by former OpenAI Codex and o1 researchers, builds custom AI agents for enterprises through a post-training approach they call Specific Intelligence. Their core training mechanism is Reinforcement Learning, where agents repeatedly attempt tasks in replayable environments and are scored against reward functions to refine model weights. For clients like DoorDash and Cognition, this means training specialized agents for merchant onboarding and bug-catching that improve from continuous use. The company evaluated multiple infrastructure platforms before selecting Modal, which provides ephemeral sandboxes for complex environment simulation, sub-second cold starts to keep GPUs busy, and serverless fan-out for parallel grading. Modal’s primitives allow each phase of the RL loop—rollouts, evals, and inference—to operate with appropriate resources while sharing state efficiently.
Optimizing Our Build Times by Migrating from Webpack to Rspack
As Yelp's JavaScript monorepo expanded over time, Webpack build speeds became a significant engineering bottleneck. To resolve this without rewriting complex configurations, the team selected Rspack for its direct compatibility with existing Webpack setups and plugins. Yelp implemented an adapter pattern to generate Rspack configurations dynamically from original Webpack files, enabling a staged opt-in rollout across development tools and CI environments. Following the initial migration, engineers addressed performance issues in barrel files using automated codemods to replace star re-exports and import-then-export patterns with named re-exports. Combined with Rspack's portable persistent caching, the migration reduced integration build times by approximately 52 percent and warm cache builds by up to 80 percent.
How Databricks Genie improves supply chain visibility with real-time AI analytics
Supply chain leaders often have extensive operational data yet still respond reactively because predictive signals such as supplier lead-time trends, inventory velocity, weather, and commodity prices remain siloed and difficult to synthesize. Databricks Genie is presented as a plain-language intelligence layer that lets leaders interrogate operational and external data in the flow of work rather than relying on analyst-led BI sessions. Users can ask questions combining supplier tiers, lead-time changes, inventory coverage, contract terms, and production schedules, then follow with what-if queries about financial exposure if conditions worsen. The post says Genie returns answers in seconds, supports near-real-time monitoring and proactive alerting, and enables shared, governed answers for procurement, operations, and finance, shifting decisions from reactive reporting toward earlier evidence-based action.
Stop rogue AI: How Unity Catalog secures your agent actions
Agentic AI systems connected to external tools can take destructive production actions while operating within delegated authority, yet without restrictions on tool selection or records of their calls. Unity Catalog now governs MCP tools with fine-grained access control, service policies, and payload logging, while Unity AI Gateway enforces decisions in real time. Policies are SQL functions receiving actor and context information; they can allow, deny, or require user consent, inspect arguments and caller properties, and block calls before execution. Every tool call is recorded in a Unity Catalog-managed Delta table with the tool name, arguments, result, user identity, and allow-or-deny status, making the activity queryable with SQL. The service policies and payload logging are available as a Gated Beta, with early access offered through Databricks account teams.
How Deutsche Börse built a generative AI tool to tackle the large-scale migration of Zeppelin notebooks to Databricks
Deutsche Börse Group is migrating a high volume of Zeppelin notebooks used by more than 2,000 users from Cloudera to Databricks, after Cloudera’s 2027 Zeppelin decommissioning and the group’s move to cloud analytics. Because the notebooks combine SQL, Python, custom interpreters, data-source references, visualizations, widgets, and scheduling logic, the team rejected fully rule-based rewriting. Its Zeppelin to Databricks Notebook Converter, delivered as a Databricks App, deterministically maps paragraphs to cells, translates interpreter syntax, reformats metadata as .ipynb JSON, and preserves original content. A generated, environment-specific prompt then gives Genie context about interpreters, data sources, and configuration patterns so it can reconstruct logic, ask clarifying questions, and handle the variable work. The hybrid workflow reduced redevelopment to 15–20 minutes per notebook, while keeping SQL, Python, visualizations, widgets, Oracle and HDFS references, scheduling, and custom business logic untouched; large-scale testing is next.
One Stone, Three Birds: Finer-Grained Encryption @ Apache Parquet™
Apache Parquet modular encryption is presented as a way to unify fine-grained access restriction, column-specific retention, and encryption at rest for large Hadoop-based data platforms. The approach encrypts selected columns with different keys at the Parquet file-format layer while preserving column projection, predicate pushdown, encoding, and compression; schema-controlled encryption avoids extra RPC calls and supports tag-driven policies. The post describes operational challenges including multiple access paths and Parquet versions, hard versus soft denial, KMS reliability, historical backfills, changing schema tags, and mixed data formats, alongside auto-onboarding and a high-throughput tool for existing data. In an evaluation using Java 8, CTR mode, and 60% encrypted columns, encryption added 5.7% to writes and 3.7% to reads, while the tool encrypted data 20 times faster; the changes were merged upstream for Parquet 1.13.0.
Cost Efficiency @ Scale in Big Data File Format
Uber’s rapid data growth increased storage and access costs, along with hardware demand, resource use, and risks such as out-of-memory conditions and long GC pauses. The work focuses on Parquet-level file-size reduction through ZSTD compression, unused-column pruning, column ordering, decimal precision reduction, and advanced encodings, while benchmarking storage and query costs across representative workloads. Converting top Hive tables from GZIP to ZSTD cut storage by about 7%, and from SNAPPY by about 39%; previously uncompressed tables saw about 79% reduction, while benchmark queries generally used fewer vCore seconds with ZSTD. For offline conversion, level 19 was selected because higher levels did not improve reduced size while increasing write time. The work also reports a 32% reduction from suitable four-column sorting and upstream Parquet changes, including faster translation and high-throughput pruning.
Coding Agent Horror Stories: The Security Crisis Threatening Developer Infrastructure
Autonomous AI coding agents increasingly handle developer workflows, but their execution model introduces critical security risks. Operating with the executing user's full permissions, these agents can read ambient secrets, inherit cloud credentials, and run destructive shell commands without human intervention. Documented incidents between October 2024 and February 2026 reveal systemic hazards, including unintentional home directory deletions and increased security vulnerabilities in generated code. To address these vulnerabilities, Docker Sandboxes isolate agent execution within dedicated microVMs, preventing access to the host filesystem and blocking common credential paths by default. Routing changes through Git worktrees and restricting network egress ensures that agent actions remain strictly bounded and reviewable before affecting main branches or production infrastructure.
How ChatFeatured migrated from PlanetScale Postgres to Postgres Managed by ClickHouse to power AI brand discovery
ChatFeatured, a Toronto-based startup, helps brands track and improve how AI models discover and recommend them, including by analyzing cited sources and guiding content production. As prompt ingestion grew to about 1,000 prompts daily across six AI models, managed Postgres instances became slow, CPU reached 90% during ingestion windows, and PlanetScale hit an IOPS limit; analytics queries remained the deeper constraint. The team migrated from PlanetScale to Postgres managed by ClickHouse using the Postgres-to-Postgres ClickPipes connector, which handled initial load, schema transfer, continuous sync, and a five-minute maintenance window; the migration took 30 minutes overall. This setup retains Postgres for transactional workloads while providing a built-in path to ClickHouse for analytics, and a 30-day LifestyleRx query fell from about two and a half minutes to less than a second, with 20x compression reported for CDN log data.
PipelineIQ: Forward‑Looking Sales Intelligence That Drives Action
PipelineIQ addresses the administrative drain and unreliable revenue predictability caused by incomplete, inconsistent, and backward-looking CRM data in B2B sales. Rather than build another forecasting system, it applies prescriptive analytics to identify forward signals and produce immediate actions for reps and managers. Built on Databricks, it uses Foundation Model APIs, Unity Catalog, Delta Lake, and AI/BI Dashboards; its confidence scorer sends CRM fields to ai_query() with a Gemma 3 12B model, scores eight MEDDPICC dimensions from 0–10, and limits missing fields to scores of 3 or below. Weighted confidence is refreshed daily, with a fail-safe override to Low when a use case has more than three active blockers. Dashboards and Genie queries connect evidence-based risk explanations, remediation steps, and portfolio views to sales execution.
amazon.science
Making LLMs faster without sacrificing accuracy
Standard scaling laws optimize parameter count and training data budgets to minimize loss but ignore internal Transformer architecture configurations. To address throughput disparities among equally sized models, researchers developed a scaling framework incorporating hidden size, the multilayer perceptron to attention parameter ratio, and grouped-query attention. The method calibrates a correction surface over standard Chinchilla scaling laws to independently optimize separable architectural parameters for target accuracy. Evaluated across more than 200 models up to three billion parameters, the framework produced the Panda model family for maximum accuracy and the Surefire family for Pareto efficiency. Surefire models matched or exceeded LLaMA-3.2 accuracy while increasing inference throughput by 12% to 47% across vLLM and SGLang serving systems on modern GPUs.
Scaling developer experience: How we improved Android Studio in a large monorepo
Grab’s passenger Android monorepo contains roughly 2,000 modules and 11 million lines of code, making Gradle-based Android Studio syncs increasingly slow and memory-intensive even though Bazel handles builds. A cold full sync could exceed 35 minutes, and a survey of 55 developers found that 82% wanted an option to exclude modules from syncing. The team built a custom Focus plugin that uses a dependency graph generated during migrateToBazel, avoids repeating Gradle’s configuration phase, and integrates with Android Studio through a one-click Select → Focus → Sync flow; a Groovy include-shadow technique avoids requiring teams to reorganize settings files. Typical focused syncs fell below two minutes, while reported results show reductions from 35 minutes to under one minute depending on module size, memory usage from 10 GB to 2 GB, and improved Compose preview updates, with broader IDE operations requiring expanded focus or full sync.
From latency to instant: Modernizing GitHub Issues navigation performance
GitHub Issues addressed navigation latency by shifting workloads to the client using a local-first, stale-while-revalidate architecture. To evaluate perceived delays, the team tracked user transitions through Highest Priority Content thresholds, aiming for instant rendering in under 200 milliseconds. The core implementation added a persistent client-side cache using IndexedDB, paired with a synchronous in-memory tier to serve hot issue payloads without asynchronous overhead. A selective preheating mechanism resolves missing cache entries for high-intent links without overburdening backend capacity with redundant requests. Following broad rollout, the proportion of instant React soft navigations increased from 4% to approximately 22%, yielding an overall cache-hit ratio of around 33%.
amazon.science
Promptimus: Improving already good LLM prompts with zero manual engineering
Enterprise applications rely on mature large language model prompts that incorporate intricate business logic and regulatory constraints, making further optimization and cross-model migration challenging. Promptimus automates prompt refinement through a four-step iterative loop that evaluates candidate prompts against decomposed metric checkpoints, diagnoses failure bottlenecks, and generates targeted improvement strategies. The system operates either via full prompt rewrites in standard mode or through surgical, programmatic find-and-replace edits in edit mode to preserve structured taxonomies and schemas. Tested across multimodal classification and code generation tasks, the framework achieves performance gains with small development datasets of 20 to 50 samples. Amazon is making the model-agnostic optimization system available through Amazon Bedrock to support enterprise generative-AI model migration without manual engineering.
The Rosetta stone of CPS: Claroty’s AI-powered library
CPS asset identification is difficult because 88% of CPS assets do not transmit an exact product code and 76% use product codes that differ from the vendor's official records. Claroty's AI-Powered CPS Library addresses this entity-resolution problem with a hybrid architecture that combines classic matching, statistical inference, generative AI, NLP, reasoning agents, and human-in-the-loop review to reconcile protocol data, vendor records, manuals, and vulnerability information. Its Databricks-based lakehouse uses Delta Lake, Unity Catalog, Spark pipelines, Model Serving, MLflow evaluations, Lakebase, and Databricks Apps to maintain governed, auditable mappings across a catalog of more than 17 million assets. The post reports 25% improved vulnerability-attribution accuracy and says early tests gave 56% of analyzed devices new or updated recommendations for outdated firmware, while low-confidence mappings remain subject to expert review.
Clinical operations intelligence belongs on the Lakehouse
The Site Feasibility Workbench addresses fragmented clinical operations data by placing a site-selection decision-support application, its models, and data on Databricks. Released as a fully open-source Databricks App, it combines a FastAPI backend, React frontend, Unity Catalog-governed data, Lakebase operational state, and AI/BI Genie within the workspace. TA-segmented LightGBM models use the organization’s CTMS, EDC, and IRT history alongside real-world evidence, patient access, qualification, Open Payments, and protocol-execution signals to rank sites. Each prediction stores SHAP attributions in a governed Unity Catalog Delta table, while MLflow tracks training runs and model versions; saved shortlists persist to Lakebase and Genie answers governed-data questions. The six-step workflow supports protocol selection through final shortlisting, including diversity considerations; the release is a decision-support layer rather than a source of record, and deployment reportedly takes about 30 minutes before security review and validation.
Top topics
- Architecture830
- Performance812
- LLMs648
- Machine Learning617
- Open Source561
- AI Agents542
- AI454
- Developer Experience446
Weekly digest
The best engineering reads, once a week.