Loading post…
NVIDIAJiwei LiuExplainer
Building Reliable Data Analytics Agents: Lessons from the KDD Cup
Summary
The NVIDIA KGMON team placed second in the KDD Cup 2026 Data Agents competition by building a reliable agent system around a small, fixed LLM. The system constrained the agent’s action space through a harness that normalized data access by converting CSV and JSON files into SQLite tables, exposed a limited set of custom functions for schema inspection, SQL queries, and answer writing, and seeded the agent with schema context before analysis. Documents and videos were handled through targeted tools like prose_helper for extracting rules from documents and keyframe extraction with aligned transcripts for video evidence, keeping the main LLM context focused. The team emphasized inspectability by recording execution traces and running repeated attempts, using evaluation and trajectory review to identify failures and refine the harness. The key lesson is that reliability for data analytics agents often comes from the harness design, not the model’s capabilities.
Context
The KDD Cup 2026 Data Agents competition required agents to answer natural-language questions over heterogeneous data sources (databases, CSV, JSON, documents, PDFs, videos) using a small, fixed LLM. The NVIDIA KGMON team placed second by focusing on harness design rather than model capability.
Approach / What changed
KGMON built a harness that constrains the agent’s action space: converts CSV and JSON into SQLite tables for a single SQL interface, provides custom functions (schema, sql, write_answer, prose_helper), seeds schema context upfront, preprocesses videos by extracting keyframes and aligning transcripts, and makes every attempt inspectable through execution traces and repeated attempts with answer selection.
Takeaways
Normalize structured data sources into a single query surface (e.g., SQLite) before the agent starts, reducing routing failures and wasted turns.
Provide a small, opinionated tool harness with limited functions and error repair, keeping the model focused on analysis and reducing syntax or file-handling errors.
Make every attempt traceable by recording tool calls and intermediate results, enabling evaluation and refinement of the harness based on trajectory inspection.
FAQ
How can a small fixed LLM handle data analytics reliably?
By building a constrained harness that normalizes data access, limits tools, and makes attempts inspectable. The harness reduces the action space and provides a stable environment for the model to reason over data.
What are the key techniques for building reliable data agents?
Normalize data access, seed schema context, use a small tool harness, treat prose separately with tools like prose_helper, preprocess videos, use repeated attempts selectively, and keep humans in the loop for review and improvement.
Why is harness design more important than model capability?
Because the harness constrains the action space and makes failures inspectable, allowing the model to focus on reasoning. Reliability comes from reducing opportunities for errors and making the system easier to verify.
Source and freshness
- Source
- developer.nvidia.com
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.