Building a knowledge assistant over code

Summary

Building a knowledge assistant over code requires more careful chunking than indexing ordinary business documents, because code has nested structure, cross-file dependencies, and multiple file formats. The comparison built three Databricks Knowledge Assistants over the Casper’s Kitchens repository: fixed-size character chunks, LangChain language-aware splitting, and AST-based splitting with structure and metadata headers. The AST approach used ASTChunk to preserve units such as functions, classes, or statement blocks and prepend file paths plus class/function hierarchies to chunks. MLflow GenAI evaluation ran 46 questions with retrieval sufficiency, groundedness, and custom answer-correctness judges; AST chunking achieved 89% retrieval sufficiency and 70% fully correct answers, versus 85% and 59% for the naive baseline. The advantage was strongest for disambiguating structurally similar code, while value lookups and complete definition retrieval were comparable across methods.

Takeaways

1. 1

   AST-based chunking produced 70% fully correct answers, compared with 59% for fixed-size chunks and 61% for the language-aware splitter.

2. 2

   Retrieval sufficiency was high for every strategy: 85% for the naive baseline, 87% for language-aware splitting, and 89% for AST-based chunking.

3. 3

   AST-based chunking helped most with disambiguating structurally similar code across modules; value lookups and complete definition retrieval were comparable across strategies.

FAQ