---
title: "How Hugging Face Inference Endpoints, Jobs, and Buckets Power Search on Papers with Code"
description: "Papers with Code needs a search engine that supports exact titles and arXiv identifiers, semantic requests, navigational queries, typos, and fast responses when model services are cold or unavailable. The system uses hybrid retrieval: PostgreSQL full-text search provides a lexical baseline, pgvector stores dense embeddings, and reciprocal rank fusion combines them. Full-corpus embeddings for more than 110,000 papers run in burstable GPU Jobs from repeatable-read database snapshots, with checksummed JSONL inputs and resumable Parquet outputs stored in Buckets. An Inference Endpoint serves low-latency query embeddings and bounded hourly updates for changed papers, while lexical retrieval remains a fallback. The embedding contract pins the model revision, dimension, prompts, normalization, and source hash; production uses Qwen/Qwen3-Embedding-0.6B with 256-dimensional L2-normalized vectors, and stored vectors also support related-paper retrieval without a request-time model call."
---

# How Hugging Face Inference Endpoints, Jobs, and Buckets Power Search on Papers with Code

[Hugging Face](https://yomu.fyi/company/hugging-face) · Niels Rogge · Aug 21, 2026

**Type:** Explainer

## Summary

Papers with Code needs a search engine that supports exact titles and arXiv identifiers, semantic requests, navigational queries, typos, and fast responses when model services are cold or unavailable. The system uses hybrid retrieval: PostgreSQL full-text search provides a lexical baseline, pgvector stores dense embeddings, and reciprocal rank fusion combines them. Full-corpus embeddings for more than 110,000 papers run in burstable GPU Jobs from repeatable-read database snapshots, with checksummed JSONL inputs and resumable Parquet outputs stored in Buckets. An Inference Endpoint serves low-latency query embeddings and bounded hourly updates for changed papers, while lexical retrieval remains a fallback. The embedding contract pins the model revision, dimension, prompts, normalization, and source hash; production uses Qwen/Qwen3-Embedding-0.6B with 256-dimensional L2-normalized vectors, and stored vectors also support related-paper retrieval without a request-time model call.

## Context

Papers with Code aims to make open AI research accessible through search, but research queries require more than regular text matching. The search must handle exact titles and identifiers, semantically related concepts, navigational requests such as an original paper, incomplete titles, typos, and model-service cold or unavailable states.

## Approach / What changed

The architecture separates offline corpus construction from online retrieval. PostgreSQL full-text search supplies lexical retrieval, pgvector supplies dense embeddings, and reciprocal rank fusion combines both. Hugging Face Jobs perform large GPU embedding runs, Storage Buckets hold immutable and checksummed artifacts between database export, computation, and import, and an Inference Endpoint handles query embeddings plus bounded hourly updates for changed papers. A versioned embedding contract records model revision, dimensions, prompts, normalization, input format, and source hashes.

## Takeaways

- Hybrid retrieval combines PostgreSQL keyword matching with pgvector semantic recall through reciprocal rank fusion, allowing exact mentions and fuzzier conceptual matches to contribute to search results.
- Jobs are used for full rebuilds, new model generations, and large backfills, while Inference Endpoints handle interactive queries and small incremental document updates; lexical search provides a fallback during endpoint cold starts or failures.
- In a 5,000-paper pilot, 256-dimensional Matryoshka embeddings preserved ANN recall while materially reducing storage compared with 1024 dimensions; stored document vectors also enable related-paper retrieval without a model call at request time.

**Tags:** [Machine Learning](https://yomu.fyi/topic/machine-learning), [Performance](https://yomu.fyi/topic/performance), [Postgres](https://yomu.fyi/topic/postgres), [Reliability](https://yomu.fyi/topic/reliability), [Search](https://yomu.fyi/topic/search)

- Source: [Hugging Face](https://huggingface.co/blog/pwc-search)
- Source URL: https://huggingface.co/blog/pwc-search
- Ingested by Yomu: 2026-08-29T05:41:52.898Z

[Read original post](https://huggingface.co/blog/pwc-search)