How we measure
Real questions agents asked, an answer key checked on the primary source, deterministic grading, and a date on every number.
Real questions, not a benchmark set
The search eval starts from questions AI agents actually sent to search tools while doing real software and research work. We wrote 48 questions from them, eight in each of six classes: dev, docs, news, policy, price and spec. Personal projects were excluded, and the full question list isn't published; six samples are.
An answer key checked on the primary source
For every question we found the primary source (the vendor's docs, pricing page, changelog or the regulation itself) and quoted the passage that answers it. A script then re-fetched every source page and string-matched the quote: 28 matched exactly, 6 matched closely, and in 10 every required fact was on the page. We checked one datasheet by hand. Three questions were dropped (one had no primary source, one had its only source in an image, one had its source page change), which leaves 45 graded questions.
Deterministic grading
No model judges the search results. Each question lists the facts a correct answer must contain, and grading is a string match:
- fact@5: the share of required facts present in the text an API returned for its top five results.
- Full answers: questions where every required fact appeared.
- Answer page first and in top 3 pages: the same check after reading the top one or three result pages in full, the way an agent searches and then fetches.
- A call that fails scores zero on every measure, because it fails the agent too. Rows with failures say how many questions they answered. A tool held back by a free tier's rate limit is shown separately, scored on what it answered, and not ranked.
- Latency and context size are medians over the calls that returned.
The one answer API, Exa /answer, returns a paragraph rather than results. Its answers were string-matched first, and the rest were read by hand against the key.
One run, dated
Each API ran once, on Nov 7, 2026, within minutes of the answer key being checked, so every API saw the same live web. Search results change daily; a number without a date isn't comparable with anything.
Evidence badges
We ran it ourselves with the method on this page. Today that is web search, Exa /answer, a partial run for Firecrawl, and Context7 and DeepWiki on the dev and docs questions.
A third party published it. We cite the publisher and the date beside every figure, note who runs the board, and record where boards disagree.
Published by the vendor itself, or by a study a vendor paid for. Shown only beside independent figures, and never used to order a list.
Desk research from vendor pages, GitHub and secondary sources, compiled on a stated date; prices are not individually re-verified. Ordered by adoption (stars) where one metric applies to the list, otherwise alphabetically. Never shown as a score.
Re-measurement
Rankings in this field go stale within months: one code-review board had four different leaders in 2026. We re-run our measurements when a tool ships a major change and at least quarterly, show the date of every run, and keep earlier results rather than overwriting them. You.com was tried but not measured yet: the account's free credit read as depleted. Docs tools (Context7, DeepWiki) were run on the 15 dev and docs questions; see the docs page.