Measure whether people recover from a form error
Completion rate is useful, but it hides where a form excludes people. Observe correction, retained work and confirmed outcomes across the supported interaction methods.
Read articleCobnex blogs / Australia
Cobnex blog
AI implementation, software architecture, cloud operations and Australian technology policy.
512 articles
Page 12 of 29
Measure the job people finish from the table
Row-rendering speed is only one part of quality. Evaluate finding, comparing and acting on the right records across desktop and mobile.
Read articleCount recovered work as well as successful sign-ins
Authentication metrics do not show whether users returned to their task. Measure draft recovery, duplicate avoidance and context correctness after expiry.
Read articleOne average relevance score is not a release decision
Compare query groups and individual regressions. Match the metric to whether the user needs one known item or several useful results.
Read articleCount unresolved dependency exposure, not just open alerts
Alert totals need context about execution, reachability and ownership. A smaller number can reflect better risk treatment or merely broader suppression.
Read articleCount unexplained permissions, not just wildcard characters
Permission quality depends on purpose, effective reach and tested boundaries. A policy can avoid an asterisk and still grant the wrong service too much authority.
Read articleMeasure how quickly the next responder can act safely
Handover quality appears in continuity of decisions, not document length. Track missing state, repeated experiments and unowned follow-up work.
Read articleMeasure eligible data left behind after the retention run
A job-success counter can hide failed destinations and overdue records. Track complete outcomes, authorised holds and unresolved copies separately.
Read articleInvestigating a suspected retrieval leak
When an assistant appears to reveal a restricted document, contain the affected path and preserve enough evidence to find where access was lost.
Read articleDiagnose a relevance regression from both candidate lists
When search gets worse, identify the stage that changed. Preserve the question, source versions and branch results before tuning the final answer.
Read articleTrace a misleading answer back to its chunk
Before changing the prompt, inspect the evidence the model received. A missing heading or broken table can turn accurate source text into a misleading answer.
Read articleHandle a citation that staff can no longer open
A failed source link may indicate a moved document, changed access or missing historical evidence. Establish which one happened before changing the answer.
Read articleFind the layer still serving the old policy
Diagnose a stale answer by following its source revision through the serving path. Rebuilding the whole index should not be the first response.
Read articlePreserve the document that changed agent behaviour
When an assistant attempts an unexpected action, retain the relevant evidence and contain the capability. Editing the prompt immediately can make the incident harder to understand.
Read articleExplain why an agent run stopped
A stopped run may be waiting for approval, out of budget or uncertain about a completed write. Give operators a state they can act on.
Read articleRecover an approved proposal with an uncertain submission
Approval confirms permission to act. It does not tell you whether a timed-out execution already happened. Resolve that uncertainty before submitting again.
Read articleResume the run whose approval expired overnight
A saved checkpoint preserves progress, but it does not freeze authority or business state. Inspect completed work before deciding what can continue.
Read articleInvestigate disagreement between human evaluators
Reviewer disagreement may reveal an unclear question, missing evidence or an ambiguous scoring rule. Resolve the cause before treating one label as ground truth.
Read article