Building AI Search on Heroku

Summary

The reference architecture addresses a common RAG failure: vector search can return semantically similar documentation that does not answer the user's question. Its query pipeline retrieves the top 20 chunks with cosine similarity in Heroku Postgres and pgvector, then sends those candidates to Cohere Rerank 3.5, a cross-encoder reranker, before Claude 3.5 Sonnet generates an answer. The indexing pipeline crawls URLs, filters likely boilerplate, splits documents into 1,000-character chunks with 200-character overlap at natural boundaries, embeds them with Cohere Embed Multilingual, and batch-inserts them. Server-Sent Events stream embedding, search, reranking, sources, and generated text to avoid a blank screen during the multi-step request. The reranking stage can be made optional when lower latency matters more than precision.

Takeaways

1. 1

   Two-stage retrieval first selects the top 20 chunks by cosine similarity, then uses Cohere Rerank 3.5 to score query-document relevance and return the top five.

2. 2

   Documents are split into 1,000-character chunks with 200 characters of overlap, prioritizing paragraph or sentence boundaries before embedding and storage.

3. 3

   Server-Sent Events expose embedding, searching, reranking, sources, and generated text while the multi-step RAG request is running; reranking may be disabled when latency matters more than precision.