An average score can hide the expensive failure
Keep task coverage and failure consequence visible. A candidate that improves easy answers while mishandling a critical exception may be a worse release.
Read articleCobnex blogs / Australia
Cobnex blog
AI implementation, software architecture, cloud operations and Australian technology policy.
150 articles in AI engineering
Page 4 of 9
Compare candidate and baseline on identical tasks
A fair release comparison holds the task and evidence steady, then examines changed outcomes. Separate quality, useful completion and operational cost.
Read articleFalse acceptance matters more than coverage alone
Processing more documents automatically is useful only if the accepted records are reliable. Measure incorrect acceptance separately from the proportion sent to review.
Read articleMeasure coverage of the data-flow inventory
A completed architecture diagram is not evidence that every payload path is understood. Track which paths have been observed, configured and tested against their requirements.
Read articleReport the expensive tail of AI task costs
Typical cost is useful for planning, but rare long runs can dominate expenditure. Show the distribution, unsuccessful work and the task types behind the tail.
Read articleCount unnecessary refusals as well as unsafe answers
A stricter assistant can look more accurate by answering fewer questions. Measure the quality of answered cases and the useful work lost through abstention.
Read articleInvestigating a suspected retrieval leak
When an assistant appears to reveal a restricted document, contain the affected path and preserve enough evidence to find where access was lost.
Read articleDiagnose a relevance regression from both candidate lists
When search gets worse, identify the stage that changed. Preserve the question, source versions and branch results before tuning the final answer.
Read articleTrace a misleading answer back to its chunk
Before changing the prompt, inspect the evidence the model received. A missing heading or broken table can turn accurate source text into a misleading answer.
Read articleHandle a citation that staff can no longer open
A failed source link may indicate a moved document, changed access or missing historical evidence. Establish which one happened before changing the answer.
Read articleFind the layer still serving the old policy
Diagnose a stale answer by following its source revision through the serving path. Rebuilding the whole index should not be the first response.
Read articlePreserve the document that changed agent behaviour
When an assistant attempts an unexpected action, retain the relevant evidence and contain the capability. Editing the prompt immediately can make the incident harder to understand.
Read articleExplain why an agent run stopped
A stopped run may be waiting for approval, out of budget or uncertain about a completed write. Give operators a state they can act on.
Read articleRecover an approved proposal with an uncertain submission
Approval confirms permission to act. It does not tell you whether a timed-out execution already happened. Resolve that uncertainty before submitting again.
Read articleResume the run whose approval expired overnight
A saved checkpoint preserves progress, but it does not freeze authority or business state. Inspect completed work before deciding what can continue.
Read articleInvestigate disagreement between human evaluators
Reviewer disagreement may reveal an unclear question, missing evidence or an ambiguous scoring rule. Resolve the cause before treating one label as ground truth.
Read articleRestore the previous behaviour after a model regression
Confirm the affected configuration and contain new failures before changing prompts in place. A known prior release is easier to assess than an unrecorded emergency variation.
Read articleGive the reviewer the contradiction, not the whole queue
A useful exception record says which fields disagree and shows their evidence. That lets reviewers resolve the problem without repeating the extraction process manually.
Read article