---
title: "Building AI Search on Heroku"
description: "The reference architecture addresses a common RAG failure: vector search can return semantically similar documentation that does not answer the user's question. Its query pipeline retrieves the top 20 chunks with cosine similarity in Heroku Postgres and pgvector, then sends those candidates to Cohere Rerank 3.5, a cross-encoder reranker, before Claude 3.5 Sonnet generates an answer. The indexing pipeline crawls URLs, filters likely boilerplate, splits documents into 1,000-character chunks with 200-character overlap at natural boundaries, embeds them with Cohere Embed Multilingual, and batch-inserts them. Server-Sent Events stream embedding, search, reranking, sources, and generated text to avoid a blank screen during the multi-step request. The reranking stage can be made optional when lower latency matters more than precision."
---

# Building AI Search on Heroku

[Heroku](https://yomu.fyi/company/heroku) · Anush DSouza · Jan 29, 2026

**Type:** Tutorial

## Summary

The reference architecture addresses a common RAG failure: vector search can return semantically similar documentation that does not answer the user's question. Its query pipeline retrieves the top 20 chunks with cosine similarity in Heroku Postgres and pgvector, then sends those candidates to Cohere Rerank 3.5, a cross-encoder reranker, before Claude 3.5 Sonnet generates an answer. The indexing pipeline crawls URLs, filters likely boilerplate, splits documents into 1,000-character chunks with 200-character overlap at natural boundaries, embeds them with Cohere Embed Multilingual, and batch-inserts them. Server-Sent Events stream embedding, search, reranking, sources, and generated text to avoid a blank screen during the multi-step request. The reranking stage can be made optional when lower latency matters more than precision.

## Context

Vector search may return documents that are semantically similar to a query but do not actually answer it, creating a gap between a RAG demonstration and a production-grade enterprise search system.

## Approach / What changed

The architecture uses separate indexing and query pipelines with Heroku Managed Inference and Agents, Heroku Postgres with pgvector, two-stage retrieval, natural-boundary chunking, batch inserts, and Server-Sent Events for progress and answer streaming.

## Takeaways

- Two-stage retrieval first selects the top 20 chunks by cosine similarity, then uses Cohere Rerank 3.5 to score query-document relevance and return the top five.
- Documents are split into 1,000-character chunks with 200 characters of overlap, prioritizing paragraph or sentence boundaries before embedding and storage.
- Server-Sent Events expose embedding, searching, reranking, sources, and generated text while the multi-step RAG request is running; reranking may be disabled when latency matters more than precision.

**Tags:** [AI Search](https://yomu.fyi/topic/ai-search), [Heroku](https://yomu.fyi/topic/heroku), [pgvector](https://yomu.fyi/topic/pgvector), [Retrieval-Augmented Generation](https://yomu.fyi/topic/retrieval-augmented-generation)

- Source: [Heroku](https://www.heroku.com/blog/building-ai-search-on-heroku)
- Source URL: https://www.heroku.com/blog/building-ai-search-on-heroku
- Ingested by Yomu: 2026-09-10T04:05:17.676Z

[Read original post](https://www.heroku.com/blog/building-ai-search-on-heroku)