Agent recovery checkpoints

Upgrade the agent while old workflows are paused

A deployment can change tool schemas and state meaning while runs are waiting for approval. Plan compatibility for those saved records before switching new traffic.

In this article

Inventory the versions still in storage

Count active and paused runs by workflow version, current step and age. Identify which versions can still produce effects. Historical completed records need readable history, while paused records may need executable compatibility.

Inspect stored tool inputs and approval payloads. If a new release renames a field or changes its meaning, an old proposal cannot simply be interpreted under the new schema without a migration rule.

Keep the model version separate from the workflow version. A model update changes generation behaviour, while a state-machine change can alter recovery and authority. Combining them under one vague release label makes failures harder to diagnose.

Choose a compatibility strategy

One option is to keep old execution code available until existing runs finish. Another is to migrate saved state into a new version with explicit transformations. A third is to stop selected runs and request a new proposal where meaning cannot be preserved safely.

The right choice depends on workflow duration and change scope. A short-lived read task may be easy to restart, while an approved mutation with uncertain execution requires careful preservation of its operation identity.

Never manufacture a new approval during migration. If the transformed payload changes what was authorised, return it for review under the normal process.

Rehearse with paused fixtures

Save fixtures at approval waits, before dispatch and after an uncertain external result. Upgrade the application and resume each one. Verify target state, operation identifiers and the final message.

Test rollback too. New code may write fields or states that the previous version cannot understand. Either preserve backward compatibility for the rollback window or provide a controlled route that prevents incompatible old workers from claiming those runs.

Include concurrent workers from different releases if the deployment platform can overlap them. Version-aware claiming should keep a worker from executing state it does not support.

Retire old behaviour deliberately

Monitor remaining runs by version and investigate those that exceed the expected lifetime. Do not keep obsolete credentials or broad permissions indefinitely merely because one forgotten run remains paused.

Close, migrate or explicitly archive those cases through a supported process. Preserve readable history and source links needed for investigation. The upgrade is complete when new tasks use the intended version and old tasks have a known, tested fate rather than being left for the next incident.

Primary sources

AWS: workflow redriveOWASP: transaction authorisation

References checked 11 September 2026.