Delete a source that has already produced an answer
Test withdrawal after the system has created passages, caches and conversation history. An empty search result alone does not prove removal is complete.
Read articleCobnex blogs / Australia
Cobnex blog
AI implementation, software architecture, cloud operations and Australian technology policy.
512 articles
Page 7 of 29
Hide an export instruction inside a supplier file
Exercise the document-to-tool boundary with a synthetic file and a harmless test destination. Measure what the application does, not only what the assistant says.
Read articleTimeout after the write, before the tool response
The hardest retry case is a successful mutation whose response never reaches the agent. Inject that failure deliberately and inspect the business record.
Read articleChange one field after the reviewer approves
A small payload mutation is enough to test whether approval is bound to the action. Use a material field and verify that execution cannot reuse the earlier decision.
Read articleInterrupt the run on both sides of a commit
Recovery tests need precise fault locations. Stopping a process at a random time rarely proves how it handles the gap between an external effect and its local record.
Read articleFind the development example hiding in the test set
Near-duplicate cases can make an evaluation look independent when it is testing familiar material. Inspect relationships between examples, not just identical strings.
Read articleUse a fluent response with the wrong extracted field
Schema checks will accept many semantically wrong values. Test a document where nearby fields are easy to confuse and assert the business meaning of each output.
Read articlePut subtotal and payable total next to each other
Test field selection with several plausible amounts in the same region. A parser that reads the digits correctly can still choose the wrong business value.
Read articleTrigger an error and inspect where its payload goes
Error paths often log more than successful requests. Use a synthetic marker to find diagnostic copies and unexpected fallbacks without exposing real information.
Read articleMake the cheap model retry until it costs more
A lower initial request cost can disappear through repair loops. Test a persistent validation failure and verify that the task stops within its total allowance.
Read articleAsk a familiar question the corpus cannot answer
A model may know a plausible answer from general knowledge while the product requires approved evidence. Test that distinction with a question just outside the source collection.
Read articleTry to bypass a module's business rule
Test the paths most likely to avoid the public contract: direct repository imports, background jobs and administrative updates. The same invariant should hold for each supported caller.
Read articleCrash the publisher after delivery but before acknowledgement
The outbox's duplicate window is predictable. Stop the worker after broker acceptance and verify that replay does not repeat the consumer's business effect.
Read articleSend identical commands at exactly the same time
Sequential retries can pass while concurrent requests still duplicate an effect. Use a barrier to make two callers compete for the same operation identity.
Read articleReplay the error response an older client expects
Error contracts influence retries, forms and recovery. Test them with supported older clients, not only with a server-side schema validator.
Read articleFail the compensation as well as the original step
A recovery path that only works when every corrective call succeeds is incomplete. Test the state left behind when compensation is rejected or its response is lost.
Read articleCreate matching record identifiers in two tenants
Similar records expose missing scope in queries and caches. Use two isolated test tenants and exercise the same operation through every supported path.
Read articleReplay an older event into the new consumer
Historical messages exercise assumptions that fresh test fixtures miss. Test their final effects before relying on replay as a recovery tool.
Read article