# Release a model change like a behaviour change

A replacement model can keep the same API shape while changing answers, tool choices and refusals. Assess those behaviours as part of the application release.

By Cobnex editorial. Published 2026-09-10. Updated 2026-09-11.

## The interface can stay stable while the product changes

An extraction service returns the same JSON fields after a model update. Its responses still pass schema validation, but it now selects the invoice date where the workflow expects the due date. Nothing in the HTTP contract reveals the regression.

A model change can affect interpretation, verbosity, tool selection and willingness to answer. These are product behaviours even when the endpoint and types remain unchanged. Treating the update as a routine dependency bump misses the work needed to establish whether users still receive the intended result.

Start with the tasks the application supports and the failures it must contain. Compare the candidate with the current system using the same inputs, source snapshots and application rules.

## Version the behaviour package

Record the model identifier together with prompt version, tool schemas, retrieval configuration and output validators. These components interact. A model that works well with one prompt or tool description may behave differently with another.

Keep the candidate configuration explicit and reproducible. If a provider offers a stable version identifier, record it. If the endpoint can change behind an alias, acknowledge that limitation and use monitoring to detect behaviour drift rather than claiming exact reproducibility.

### A model release has evidence and a return path

The candidate is evaluated as an application configuration before controlled traffic and observation.

1. **Freeze candidate**: Model, prompts, tools and retrieval settings
2. **Compare tasks**: Quality, authority, latency and useful completion
3. **Limit exposure**: Route a defined cohort with observable outcomes
4. **Expand or restore**: Use agreed criteria and a tested prior configuration

## Compare failures, not only average scores

Inspect cases that changed from correct to incorrect, including structured fields and tool targets. A more fluent answer can hide a worse extraction. A higher support score can result from refusing questions the previous model answered usefully.

Measure latency and total cost after retries and validation failures. A lower per-token price does not establish a lower cost per completed task. Keep results by task category so common easy work does not obscure consequential exceptions.

Use deterministic tests for application authority rules. A new model's better refusal behaviour does not justify weakening the executor's access or approval checks.

## Make restoration practical

Retain the previous configuration while observing the candidate, and test that it still works with current tool and state schemas. A rollback switch is less useful if the old model cannot interpret newly saved workflow records.

Define how in-flight tasks are handled. Some can finish on the configuration that started them, while others may need a controlled pause. Avoid moving a partially executed mutation workflow between behaviours without preserving its operation state.

The release is ready when the team can explain what changed, show evidence on relevant tasks and restore a known configuration if the candidate fails in use.

## Sources

- [Microsoft Learn: evaluation and observability](https://learn.microsoft.com/en-us/azure/foundry/concepts/observability)
- [Microsoft Learn: retrieval and answer evaluators](https://learn.microsoft.com/en-us/azure/foundry/concepts/evaluation-evaluators/rag-evaluators)