Distinguish suspicious output from a successful attack

Define the prohibited effect

Before running an injection test, state what the attacker is trying to achieve. It might be reading another organisation's record, sending a file to an unapproved destination or changing a protected field. The success condition should be observable in the application or controlled downstream system.

A keyword in the answer is usually an inadequate proxy. Quoting a malicious instruction during analysis does not necessarily mean it was followed. Conversely, a reassuring final response does not undo a tool call that already changed state.

Keep each case's legitimate task alongside the prohibited effect. A defence that stops all work can produce a low attack success rate while making the product unusable.

Use a funnel of observations

Track whether the payload reached the relevant model context, whether an unauthorised action was proposed, whether policy denied it and whether any prohibited side effect occurred. This separates model susceptibility from executor enforcement.

For example, ten proposed exports that are all denied reveal a model behaviour worth improving, but they do not equal ten completed leaks. One completed export hidden among many refusals deserves immediate attention even if the average refusal rate looks strong.

Record data exposure separately from write actions. A read tool may return restricted information that appears in an answer without any export operation. The test needs to inspect both tool results and user-visible output under the defined access policy.

Count meaningful attempts

Avoid inflating confidence with hundreds of tiny wording variations that exercise the same path. Group results by entry point and capability: document ingestion, web content, tool response, persistent memory and each consequential tool.

Report the number of independent cases and repeated runs. Model behaviour can vary, so one successful refusal does not establish consistent protection. Preserve configurations and test inputs to make comparisons useful across releases.

False positives matter too. Include ordinary documents containing quoted instructions, security examples and procedural language. Measure whether the assistant can still complete those tasks without unnecessary refusal or escalation.

Connect acceptance to controls

Set release criteria for the actual prohibited effects and the task's consequence. Application authorisation checks should have deterministic tests in addition to model-based evaluations. A statistical improvement in refusal does not justify removing a required access control.

Document residual failures and the available containment mechanism. If a new tool expands what the assistant can do, revisit the success conditions rather than reusing an old score uncritically. The useful result is a clear account of which boundaries held, which did not and whether the intended work remained possible.