3x Faster Search: Parallel Test-Time Scaling with Instructed-Retriever-1

Summary

Instructed-Retriever-1 is introduced as the retrieval-specialized model behind an update to Agent Bricks Knowledge Assistant, targeting search quality and latency. It parallelizes query and filter generation for broader candidate recall, then applies multi-pivot groupwise reranking to improve precision, using one model for both retrieval stages. The model was trained on synthetic enterprise-style environments and evaluated on realistic Knowledge Assistant workloads, where it scored 81.0 nDCG@10 for reranking and matched Claude Sonnet 4.5 retrieval quality on KARLBench. Serving uses a Mixture-of-Experts architecture, FP8 quantization, speculative decoding, and infrastructure tuning; the update reports more than 3x faster search, 2x faster answer generation, and TTFT around two seconds. End-to-end latency was consistently below 10 seconds in the offline evaluation setup.

Context

Knowledge Assistant needed higher-quality retrieval without the substantial latency and cost associated with sequential agentic search operations. The update was also evaluated against realistic enterprise-style workloads and production latency constraints.

Approach / What changed

Instructed-Retriever-1 uses parallel query and filter generation to broaden recall, followed by multi-pivot groupwise reranking to improve precision. A single retrieval-specialized model supports both stages, with Mixture-of-Experts serving, FP8 quantization, speculative decoding, and infrastructure tuning used to improve serving efficiency.

Takeaways

• Parallel query and filter generation improves candidate-set recall, while increasing the number of pivots provides a precision-oriented test-time scaling control for reranking.
• On the internal realistic-workload evaluation, Instructed-Retriever-1 achieved 81.0 nDCG@10 for reranking, compared with 80.1 for Claude Sonnet 4.5; both improved over no reranking.
• FP8 quantization showed no statistically significant quality degradation relative to BF16, and speculative decoding added more than 30% speed-up to the combined query-generation and reranking path.

Related reading