Compare cost after retries and validation failures

Use a useful denominator

For an extraction workflow, count cost per accepted extraction with the required fields correct. For an agent, use a completed authorised task. A generated response is only an intermediate product.

Track initial model usage, retries, repair prompts, tool calls and evaluator work. Include the cases that fail permanently rather than spreading their cost out of sight. The system paid for those attempts even though the user received no usable result.

Keep manual review effort visible. It may be difficult to assign an exact currency value, but hours spent correcting output belong in the comparison.

Work through a simple example

Suppose two hypothetical configurations cost the same amount for an initial request. One succeeds on the first attempt more often, while the other routinely needs a second generation and review. Their listed request price is identical, but their completed-task cost is not.

Conversely, a more expensive model can be wasteful if a simpler configuration already meets the task's quality and authority requirements. The decision depends on measured workload results, not an assumption that higher capability always pays for itself.

Use actual provider prices only after verifying the current rates and the relevant service configuration. Keep this calculation separate from technical acceptance so a cost saving does not conceal a prohibited effect.

Measure concurrency and tail behaviour

A candidate with longer outputs or more tool turns can occupy resources for longer and increase concurrent work. Test under comparable load, including retry behaviour when limits are reached.

Record end-to-end latency and the slow tail. Users may abandon or repeat a task when the final result takes too long, creating extra load that a quiet benchmark does not capture.

Bound repair loops. Repeatedly asking the model to fix an invalid response can consume budget without addressing a permanent schema or prompt mismatch. Return a clear limited result or escalation when the budget is exhausted.

Compare by task category

Some tasks may justify the candidate while others do not. If routing different tasks to different configurations, include the routing logic and fallback behaviour in evaluation. A classification error can send a difficult task to an unsuitable model.

Review total cost, accepted quality and operational effort together. The useful choice is the configuration that performs the required work dependably at an understood cost, with explicit limits on retries and recovery.