Hugging Face has detailed the architecture behind search on the relaunched Papers with Code, revealing that a hybrid system built on three of its own products — Jobs, Storage Buckets, and Inference Endpoints — now maintains vector embeddings for more than 110,000 papers sourced from arXiv and Daily Papers. The disclosure offers a rare production case study of how the company’s infrastructure products work together, and explains how search stays fast and available even when GPU services fail.

According to the engineering post, the system splits search into two deliberately separated parts. Expensive, throughput-oriented work — embedding the full paper corpus — runs offline as Hugging Face Jobs, burstable GPU compute that only consumes resources while running. Durable artifacts such as input snapshots and output vectors live in a Storage Bucket. Only the small query-embedding step sits on the live request path, served by a protected Inference Endpoint. If that endpoint is cold, busy, or unhealthy, search immediately falls back to PostgreSQL full-text retrieval, keeping the site responsive.

The search itself is hybrid: PostgreSQL’s native full-text capabilities provide a fast lexical baseline for exact matches, while pgvector supplies dense embeddings for semantic recall. The two result sets are merged using the reciprocal rank fusion (RRF) algorithm. Hugging Face said this design draws on prior experience at ML6, where hybrid systems typically outperformed keyword-only or vector-only search. The team cited a 2023 Microsoft Azure AI Search analysis showing hybrid retrieval with reranking outperforming vector search alone, while noting that rerankers add overhead and latency.

A central design decision was a strict, versioned embedding contract. Every paper is encoded as a normalized title plus abstract, and every vector generation records the model repository and exact revision, output dimension, input-format version, whether the input is a query or document, normalization method, and a content hash. Production uses Qwen/Qwen3-Embedding-0.6B pinned to an exact revision, producing 256-dimensional L2-normalized vectors. The team chose 256 dimensions using the model’s Matryoshka Representation Learning (MRL) support, which allows a trade-off between quality and speed or storage cost. Qwen’s separate document and query prompts are used for corpus embedding and live queries respectively.

The corpus build itself is a batch pipeline: a repeatable-read PostgreSQL snapshot exports every paper’s latest version in streamed rows, writes bounded JSONL shards with a manifest of row counts and SHA-256 checksums, and syncs the immutable run directory to a private bucket. That bucket is mounted directly into an l4x1 Job — an NVIDIA L4 GPU with 24GB of VRAM — using hf-mount, so the worker sees it as an ordinary filesystem.

Why This Architecture Matters

The post matters on two levels. For Hugging Face, it is a public proof point that its newer infrastructure products — Jobs, Buckets, and Inference Endpoints — can run a real production search workload at scale, not just demos. Papers with Code serves researchers and increasingly AI agents, which can query search through the pwc search CLI command via a Skill, so reliability of the search path directly affects both human and automated users.

For engineering teams building retrieval systems, the write-up documents concrete patterns: versioned embedding contracts to prevent silent model-drift failures, RRF to combine lexical and semantic retrieval, and graceful degradation to full-text search when GPU services are unavailable. It also shows how MRL-supporting models like Qwen3 let teams shrink vector dimensions — here to 256 — to keep online search fast. The team framed the broader goal as making open AI research accessible enough to ‘power the wave of research that leads to the next Transformer.’

The Papers with Code Revival

Papers with Code, a site for linking research papers to code implementations and state-of-the-art benchmarks, was relaunched by Hugging Face roughly three months before the post, after the original site went offline following its acquisition by Meta. Hugging Face framed the revival’s goal as making open AI research accessible and digestible — letting users find artifacts related to a paper, track SOTA results across AI domains, and build on each other’s work.

The team emphasized that research search differs from ordinary text search: a useful engine must resolve exact titles and arXiv identifiers, but also handle fuzzy queries like ‘small language models for code generation’ where the words never appear together in a paper, recognize navigational requests such as ‘the original BERT paper,’ tolerate typos and incomplete titles, and stay fast when a model service is cold or temporarily down. Those requirements shaped the hybrid design and the full-text fallback.

“Its goal is to power the wave of research that leads to the next Transformer.”

— Hugging Face engineering post

Details Not Yet Published

Several operational specifics remain undisclosed. The post does not publish latency figures, search-quality benchmarks, or reranking evaluation results for the production system, so claims of hybrid superiority rest on the cited Microsoft analysis and the team’s prior ML6 experience rather than Papers with Code measurements. The cost of running Jobs and Inference Endpoints at this scale is not stated.

The source material also appears truncated mid-example — the Jobs invocation command for the embedding pipeline is cut off before its output arguments — so the full job configuration, including output handling and any post-processing steps, is not fully documented in the published excerpt. How frequently the corpus is re-embedded, and how incremental updates are merged into the live index, are described only at a high level.

Scaling Search Beyond 110,000 Papers

Hugging Face said the post is the first of a series explaining the architecture, design decisions, and lessons learned taking the system to production, so further technical installments — likely covering the online serving path, reranking experiments, and agent-facing tooling — can be expected. As the arXiv and Daily Papers corpus grows, the incremental-update workflow on Inference Endpoints will carry more of the load, and the versioned embedding contract will govern any future model upgrades beyond the pinned Qwen3-Embedding-0.6B revision.

For users, the practical developments to watch are search-quality improvements on the site, deeper agent integration through the pwc search CLI Skill, and any published benchmarks validating the hybrid approach on Papers with Code data itself.

Key Questions

What is hybrid search on Papers with Code?

It combines PostgreSQL full-text search, which finds exact keyword matches, with pgvector semantic search, which finds fuzzy, semantically similar content. Results are merged using the reciprocal rank fusion algorithm.

Which Hugging Face products power the system?

Jobs provides burstable GPU compute (an NVIDIA L4) for embedding the corpus offline, Storage Buckets holds durable input and output artifacts, and Inference Endpoints serves low-latency embeddings for live search queries.

What happens if the GPU embedding service goes down?

According to Hugging Face, if the Inference Endpoint is cold, busy, or unhealthy, search automatically falls back to PostgreSQL full-text retrieval, so results still return, just without semantic recall.

Which embedding model is used?

Production uses Qwen/Qwen3-Embedding-0.6B pinned to an exact revision, producing 256-dimensional L2-normalized vectors. The 256-dimension size was chosen via the model’s Matryoshka Representation Learning support to keep search fast.

How many papers are covered by the search index?

The system maintains embeddings for more than 110,000 papers sourced from arXiv and Hugging Face’s Daily Papers.

Source: Hugging Face