Measure approvals that actually change the outcome
A high approval rate does not show that review is useful. Look at corrections, missed errors and the time people need to understand a proposal.
Read articleCobnex blogs / Australia
Cobnex blog
AI implementation, software architecture, cloud operations and Australian technology policy.
512 articles
Page 10 of 29
Measure recovery by the final business state
A restarted worker is not a recovered task until its effects are correct and its status is understandable. Measure duplication, unresolved work and operator effort together.
Read articleAn average score can hide the expensive failure
Keep task coverage and failure consequence visible. A candidate that improves easy answers while mishandling a critical exception may be a worse release.
Read articleCompare candidate and baseline on identical tasks
A fair release comparison holds the task and evidence steady, then examines changed outcomes. Separate quality, useful completion and operational cost.
Read articleFalse acceptance matters more than coverage alone
Processing more documents automatically is useful only if the accepted records are reliable. Measure incorrect acceptance separately from the proportion sent to review.
Read articleMeasure coverage of the data-flow inventory
A completed architecture diagram is not evidence that every payload path is understood. Track which paths have been observed, configured and tested against their requirements.
Read articleReport the expensive tail of AI task costs
Typical cost is useful for planning, but rare long runs can dominate expenditure. Show the distribution, unsuccessful work and the task types behind the tail.
Read articleCount unnecessary refusals as well as unsafe answers
A stricter assistant can look more accurate by answering fewer questions. Measure the quality of answered cases and the useful work lost through abstention.
Read articleMeasure the changes that cross module boundaries
Dependency counts are useful signals, but the practical question is whether ordinary business changes remain understandable and local. Review actual change patterns.
Read articleMeasure the oldest unpublished business event
Queue depth shows volume, but age shows how long a business change has been waiting. Track delivery and consumer completion as separate stages.
Read articleCount duplicate effects, not duplicate requests
Repeated requests are normal in a retrying system. The important measure is whether they create additional business actions or leave the caller without a reliable outcome.
Read articleMeasure use of the contract you want to retire
Request counts are a starting point. Identify active callers, their business cycles and the operations still depending on the old contract before setting a retirement decision.
Read articleMeasure unresolved business states in the saga
A coordinator's success rate can hide workflows that remain partly committed. Track the business states and their age, including failed or uncertain compensation.
Read articleMeasure isolation failures outside the main API
Include jobs, exports, caches and support tools in the evidence. A clean set of controller tests does not establish isolation across the application.
Read articleMeasure consumers still using a retired event shape
A producer deployment does not finish an event migration. Track who still depends on the old contract and whether historical recovery remains possible.
Read articleA high cache hit rate can hide stale answers
Measure freshness and user-visible correctness alongside speed. A cache that repeatedly serves the wrong value can look excellent on a performance dashboard.
Read articleCount correct rows, not just migrated rows
Progress counters can report success while conversions are incomplete or wrong. Use coverage, correctness and production impact as separate acceptance measures.
Read articleCount conflicts separately from failures
A rejected competing write may show that protection worked. Measure its user impact and recovery outcome without treating every conflict as an infrastructure incident.
Read article