Build a small judged query set before tuning weights

Store intent beside each query

Give every evaluation query an ID, text, task category and short intent description. Record the authorised audience or filter context where it affects eligible results.

Use stable document identities and a documented rating scale. Keep unjudged documents distinct from judged irrelevant ones. Otherwise a new useful result can be penalised simply because nobody has reviewed it yet.

This synthetic fixture describes an application-owned evaluation format, not a direct request to a particular search API.

{
  "queryId": "search-example-7",
  "text": "undo deployment after database change",
  "intent": "Find the current rollback limits for a schema-changing release",
  "category": "operationalProcedure",
  "ratings": [
    { "documentId": "release-recovery-v3", "grade": 3 },
    { "documentId": "archived-release-v1", "grade": 0 }
  ]
}

Run a fixed baseline

Record the corpus version, index configuration and query settings. Save the ordered result IDs for each query before calculating metrics.

Include enough candidates for the chosen evaluation depth and any later manual review. Preserve errors and timeouts as outcomes rather than dropping those queries from the average.

Use deterministic tie-breaking where the product requires stable ordering and the search engine supports it. Document remaining variation so small changes are not mistaken for a meaningful improvement.

Compare a bounded change

Alter one relevant mechanism, such as title weighting or a specific synonym mapping. Run the same query set and produce per-query result differences alongside aggregate metrics.

Review newly retrieved unjudged documents before drawing a conclusion. Add appropriate judgments through the same review process rather than automatically labelling them bad or good.

Keep the harness connected to the user path

Check destination URLs and snippets for the judged results. A result with the right ID but a broken link does not complete the task.

Add a small set of end-to-end journeys for filtering, keyboard navigation and opening a result. The retrieval harness explains ranking, while those checks establish that the interface actually exposes the useful output.