Evals

Retrieval runs the 97-query golden set (known titles, people, half-remembered plots, facets, Spanish) against the live api after every deploy and weekly. Recall@5 is the share of each query's correct titles in the top 5; MRR is how high the first one ranks.

Groundedness gives a 70B LLM judge each blurb and the sources it was written from, and scores the share of its claims a source supports. Runs are started by hand from eval-groundedness.yml. The same judge gates new blurbs at ingest: only a fully supported, properly cited blurb is approved without an editor.

Tool selection connects to the live MCP server like any client, reads its instructions and tool list, and gives them to Llama 3.3 70B with each test request (some carry an earlier turn, as in "tell me more about the second one"). Some requests are written to trip a model up - a film set in the 1940s is not a 1940s film - and other models are scored on the same requests. A request passes when the model calls the right tool, or none for an off-topic question, with valid arguments, the expected title id or filter, and no search filter the person didn't ask for - filters exclude, so a guessed one can hide the answer. Runs are started by hand from eval-mcp-tools.yml, after a change to a tool's wording.

Runs marked invalid stay on the record. Their judge was given each source's reference (a title slug and a director's name) instead of its text, so it never saw the synopsis, and it ran at a non-zero temperature. Scores only compare within one judge version.

Unscored in the latest run: 10-years-of-taking-the-sky-by-storm-1991 (The judge's reply wasn't valid JSON, so the blurb went unscored rather than failed.)