Loading post…
JetbrainsAdam MalekExplainer
Building a RAG Pipeline for Semantic Code Search: A Developer Diary and Field Notes
Summary
JetBrains Context is presented as a production RAG pipeline for semantic code search, designed to give coding agents precise, citable evidence from repositories instead of relying on keyword tools such as grep. This first installment covers parsing, structure-aware chunking, and vectorization, with the goal of retrieving meaningful functions, symbols, or snippets rather than whole files or insignificant lines. Language parsers produce syntax nodes, and the algorithm retains the largest units that fit, splits oversized leaf nodes when necessary, groups small adjacent units, and keeps documentation, annotations, modifiers, and closing syntax with their constructs. JetBrains Context supports this approach for Kotlin, Java, Python, JavaScript, TypeScript, C#, PHP, Go, and Rust; other languages use line-based splitting. Vectors are generated with an open-weight model on company-operated GPUs, while indexed records store coordinates rather than source code, and the post reports that this setup outperformed hosted embedding APIs on internal code-retrieval benchmarks.
Context
Large codebases make keyword search and fixed-size file splitting unreliable for coding agents: relevant code may not contain the searched term, while whole-file or line-level retrieval can provide either too much or too little context. The post focuses on the first stages of JetBrains Context’s pipeline and notes that storage, low-latency querying, evaluation, and agent integration are covered later.
Approach / What changed
Source files are parsed into syntax nodes, then structure-aware chunking keeps the largest semantically coherent units that fit the preferred size. Oversized nodes are subdivided when possible, small adjacent units are grouped, and documentation, annotations, modifiers, and closing syntax remain attached to their declarations. Chunks are vectorized with an open-weight embedding model running on operated GPUs; indexed metadata stores references and offsets, while snippets are reconstructed locally from the user’s checkout. Paths are abbreviated consistently in indexed chunks and scoped queries.
Takeaways
Fixed-size line chunks can combine unrelated code, while line-level chunks can lose essential context; structure-aware chunking aims to keep semantically meaningful units together.
JetBrains Context uses specialized parsing and chunking for Kotlin, Java, Python, JavaScript, TypeScript, C#, PHP, Go, and Rust, with line-based splitting for other languages.
The index stores references, offsets, and vectors rather than source content, and embeddings are produced on operated GPUs with an open-weight model rather than sent to external providers.
FAQ
How does structure-aware chunking avoid returning misleading code fragments?
The parser exposes syntax nodes and their relationships. The chunker keeps the largest units that fit, groups small adjacent units, and preserves documentation, annotations, modifiers, and closing syntax with the constructs they describe.
What happens for languages without specialized parsing support?
JetBrains Context falls back to naive, line-based splitting for other languages or documents. Structure-aware parsing and chunking are currently described for nine languages.
How does the design protect source code during indexing?
Indexed records contain a cluster reference, item type, path, offsets, a vector reference, and optional metadata, but not the source code itself. Search results are coordinates, and the displayed snippet is assembled locally from the user’s checkout; embeddings are generated on operated GPUs.
Source and freshness
- Source
- blog.jetbrains.com
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.