# Keep one worker on the old credential during the drill

A delayed consumer reveals whether rotation depends on every process updating at once. Test its recovery after the previous credential is no longer usable.

By Cobnex editorial. Published 2026-09-10. Updated 2026-09-11.

## Prepare a safe target and two consumers

Use a test service with the same relevant credential behaviour as production. Start two workers and confirm both can perform the intended operation.

Allow one worker to refresh normally and deliberately hold the other on its current configuration. Record version references without printing credential values.

Choose a synthetic operation whose retry does not create a harmful external effect.

## Rotate through the normal process

Run the actual rotation workflow and verify the new credential at the target. Confirm the first worker adopts it.

Retire the old credential according to the supported test strategy. Do not assume that moving a secret-store label revokes the value at the target service.

Attempt a new connection from the delayed worker. This is more revealing than reusing an established session that may remain authenticated.

## Observe recovery and limits

The delayed worker should follow its documented refresh, restart or hold procedure. Confirm that it does not retry indefinitely with the same stale value.

Inspect authentication failures, secret-service calls and queued work. A poorly coordinated refresh can create a burst of requests that affects other consumers.

Check operation outcomes after recovery. A failed connection before execution is different from an interrupted request whose effect is unknown, so the retry path must preserve the appropriate operation identity.

## Interrupt rotation at a partial state

In a separate safe run, stop the workflow between updating the target and publishing or selecting the new secret version, where the chosen mechanism permits such a test.

Follow the documented repair path and confirm which credential is actually valid. Avoid repeatedly restarting the process without inspecting its current state.

Save the delayed-consumer and partial-state results as regression evidence. The drill passes when the system recovers through a known path and the old credential's continued validity is not the hidden reason everything appears healthy.

## Sources

- [AWS Secrets Manager: rotation overview](https://docs.aws.amazon.com/secretsmanager/latest/userguide/rotating-secrets.html)
- [OWASP: secrets management](https://cheatsheetseries.owasp.org/cheatsheets/Secrets_Management_Cheat_Sheet.html)