Q2D-Web: Evaluating First-Stage Retrievers at Scale

Q2D-Web: Evaluating First-Stage Retrievers at Scale

Source: Perplexity Engineering, September 9, 2026

Perplexity has introduced Q2D-Web (Query2Doc-Web), a private benchmark with a public leaderboard for evaluating retrieval inside agentic RAG systems. It extends the original Q2D benchmark along the dimensions that matter for production web search: corpus size, judged-query volume, and annotation depth.

The benchmark is designed for embedding models operating over large-scale web search. It combines 190 million web documents with 69,721 agent-reformulated queries in ten languages. The queries were sampled over nine months of PII-free production search traffic, and the evaluation is deliberately not limited to one definition of relevance.

At this scale, query-document relevance is sparse. A single labeling source can quietly encode one pipeline's preferences, so Q2D-Web provides three relevance-judgment sets: agent citations, production web rankings, and additional LLM judgments for previously unjudged pairs. The extra judgments reduce false negatives while multiple sources limit single-pipeline bias.

Why realistic retrieval evaluation is hard

A production retrieval system searches billions of pages and returns thousands of candidates. Those candidates define the evidence an agent can see, so the first-stage retriever shapes everything downstream. Realistic evaluation therefore has to scale along three dimensions: documents in the corpus, judged queries, and relevance judgments per query.

More documents bring harder negatives: pages that resemble a query semantically but omit the required date, entity, version, quantity, or aspect. If a benchmark shrinks its corpus while keeping the known positives, it can remove those distractors and inflate recall.

More judged queries reduce dependence on one particular sample. Deeper judgments reduce false negatives, where a retrieved but unlabeled document is treated as irrelevant. The original post points to MS MARCO evidence that 70% of manually inspected top-retrieved but unlabeled passages were in fact relevant.

Most existing benchmarks optimize one or two of these axes. Web-scale collections often have few judged queries; query-heavy collections tend to have smaller corpora and sparse labels. They also usually contain human-written queries, while agentic RAG searches with reformulated queries whose wording, structure, and specificity differ from the original user message.

Q2D-Web combines all three: 190 million documents, 69,721 agent queries, and an average of 99.6 positive judgments per query in the combined set. Its closest public comparison, MS MARCO Web Search, has 100.9 million documents, 9,374 test queries, and one click-derived positive label per query.

How the benchmark was built

Q2D-Web starts with 23,000 PII-free production searches collected over nine months in ten languages and dozens of domains. A search contains all web-search tool calls an agent performs in response to a user message. Each tool call may include multiple queries.

There are two query roles. A primary query restates the user's information need. Support queries explore alternative phrasings, background information, or related entities. Because they can require different documents, each query is evaluated independently with its own judgments.

Privacy filtering is applied before inclusion. Queries must come from users who allowed their data to be used, PII detection using perplexity-ai/pplx-pii-masking excludes personal information, and exact duplicates, very short queries, and queries with operators such as site: or quoted phrases are removed.

English accounts for 65.8% of queries, followed by Spanish, Russian, German, French, Portuguese, Italian, Korean, Japanese, and Chinese. Topic coverage includes consumer goods, programming, law, health, business, science, education, technology, finance, travel, entertainment, politics, local information, and news.

For the corpus, Perplexity retrieved the top 5,000 documents for each query with a production retrieval system and took the union. Documents were deduplicated with MinHash–LSH, clustering items whose token 5-gram sets have Jaccard similarity of at least 0.975.

The collection is intentionally not a random web sample. Every document was a plausible result for at least one benchmark query. That produces a dense pool of difficult distractors: pages matching the query's topic or language but missing a required date, entity, version, quantity, or aspect.

Three judgment sets, not one ground truth

Citation marks a document relevant when an agent cited it in a response. This is closest to downstream use, but high precision and low recall by construction: once an agent has sufficient support, it has little reason to cite every other useful page.

Web Ranking contains up to 50 documents per query, averaging 43.1. They were identified by an internal retrieval stack using BM25 and dense retrieval in the first stage, followed by cross-encoder reranking. This captures useful pages that were not cited.

Combined + LLM-Judged takes the union of Citation and Web Ranking, then adds LLM judgments for previously unjudged candidates. The pipeline pools results from BM25, ColBERTv2, and seven dense retrievers released before January 1, 2025; merges their rankings with reciprocal rank fusion; selects the top 500 unjudged documents; and applies a strict binary relevance judge using DeepSeek-V4-Flash.

Subsampling for practical evaluation

Full-corpus evaluation is expensive. One Q2D-Web run with pplx-embed-v1-4b requires 4,608 H200 GPU-hours; even EmbeddingGemma-300M requires nearly 200. To make evaluation practical, Perplexity built a subsampled corpus.

All positively judged documents are retained in every strategy; only the selection of unjudged distractors changes. The important distractors are those a retriever is likely to rank above a relevant document, because those are the ones that change its score and leaderboard position.

The team compared uniform random sampling, fixed-depth pooling across construction retrievers, and per-query RRF at different depths. RRF-based sampling uses 31.7% of the full corpus. It preserves the full-corpus model ranking and stays closer to absolute scores, inflating mean Recall@1000 by only 4.5 points versus 11.1 for random sampling at the same size.

With the RRF subcorpus, the cost drops to about one third: roughly 1,500 H200 GPU-hours for pplx-embed-v1-4b and under 70 for EmbeddingGemma-300M. Every submission is evaluated first on the sampled subcorpus, split by parameter count into groups of up to 1B and above 1B. A model reaching its group's top ten is then evaluated on the full corpus; models outside the top ten appear only on the sampled subcorpus leaderboard.

What the first results show

Q2D-Web's primary metric is Recall@1000, because a first-stage retriever only needs to place relevant documents in the candidate pool; subsequent reranking determines their order. Across the three relevance sets, the leaderboard provides a consistent view of web-scale retrieval while also showing how results change under narrower citation-based labels and broader relevance judgments.

Table 1. Recall@1000 and nDCG@10 for thirteen retrievers across Citation, Web Ranking, and Combined relevance sets. Higher scores are better.

Model	Recall@1000: Citation	Recall@1000: Web Ranking	Recall@1000: Combined	nDCG@10: Citation	nDCG@10: Web Ranking	nDCG@10: Combined
BM25-tantivy	43.92	42.50	44.77	5.74	11.45	30.30
EmbeddingGemma-300M	58.17	58.93	65.45	9.20	17.30	43.56
mDenseOn	51.78	52.56	59.31	7.63	15.06	40.20
Nemotron-3-Embed-1B	54.40	54.37	61.68	7.54	14.83	40.21
Nemotron-3-Embed-8B	61.68	61.73	68.58	10.16	18.69	47.44
pplx-embed-v1-0.6b	58.35	62.15	67.02	9.37	19.76	44.36
pplx-embed-v1-4b	61.22	65.73	69.11	10.33	21.29	45.84
Qwen3-Embedding-0.6B	50.10	51.82	57.89	6.58	14.11	37.38
Qwen3-Embedding-4B	54.95	55.60	62.01	7.67	14.97	40.21
Qwen3-Embedding-8B	57.38	58.34	64.53	8.30	16.27	42.69
voyage-4-nano	49.38	50.14	56.67	5.41	11.77	31.63
mLateOn	52.85	53.12	60.98	8.44	15.51	43.11
pplx-embed-v1-late-0.6b	56.56	58.14	63.40	9.13	17.25	43.39

No model leads on all three judgment sets. On Recall@1000, pplx-embed-v1-4b is highest on Web Ranking (65.73) and Combined (69.11), while Nemotron-3-Embed-8B leads Citation (61.68). When attention shifts to Combined Recall@100 and nDCG@10, the order changes: Nemotron-3-Embed-8B scores 30.03 and 47.44, above pplx-embed-v1-4b's 29.82 and 45.84.

Within families, larger models score higher on Combined Recall@1000. Qwen3-Embedding rises from 57.89 at 0.6B to 64.53 at 8B, Nemotron-3-Embed rises from 61.68 at 1B to 68.58 at 8B, and pplx-embed-v1 rises from 67.02 at 0.6B to 69.11 at 4B.

Perplexity models use the same evaluation pipeline as submitted models. But because Q2D-Web is derived from Perplexity production traffic, those models may benefit from an in-distribution advantage. The evaluation queries and corpus were excluded from model training, yet results should still be interpreted with that potential advantage in mind.

Submitting a model

To request an evaluation, submit a publicly available Hugging Face model through the evaluation request form. Perplexity follows the model card for model-specific inference settings, including recommended query and document instructions or prefixes, pooling, normalization, and similarity function. All queries and documents are truncated to 512 tokens with the model's own tokenizer, including instructions, prefixes, and special tokens, so a longer supported context confers no advantage.

Eligible models must load through standard transformers or sentence-transformers APIs. Models that do not run with trust_remote_code=False require manual review of both the custom code and the submitter, which takes considerably longer. Private or gated repositories are not eligible. The technical report and the public leaderboard provide the methodology, findings, and current results.

---

Source: Q2D-Web: Evaluating First-Stage Retrievers at Scale, Perplexity Engineering, September 9, 2026.