Track three clocks in the ingestion pipeline

Record the business event first

A document might change at 09:00, be detected at 09:08 and become searchable at 09:12. If the application records only the last worker completion time, it cannot explain the twelve-minute delay. It may even report excellent processing performance while the change detector is falling behind.

Store the source revision and the time the source says it changed. Keep the observation time separately, especially when polling a system that does not provide a reliable event timestamp. An observation timestamp is useful evidence, but it should not be presented as the actual modification time.

Use stable source identifiers. Names and paths can change without creating a different business document. Where the connector cannot distinguish a rename from deletion and creation, document that limitation and make the resulting behaviour testable.

Record progress without declaring victory early

Capture processing milestones such as extraction completed, passages produced and index write accepted. An accepted write is not necessarily query-visible immediately. Verify the search platform's semantics and add a read-side observation where freshness matters.

{
  "sourceId": "policy-42",
  "sourceRevision": "r18",
  "sourceChangedAt": "2026-09-01T23:00:00Z",
  "observedAt": "2026-09-01T23:08:00Z",
  "indexedAt": "2026-09-01T23:11:40Z",
  "queryVerifiedAt": "2026-09-01T23:12:00Z"
}

This example illustrates a progress record. Production code also needs explicit error states and a rule for unknown times. Avoid using a successful timestamp from an earlier revision when the current revision failed. That can make a broken document appear healthy.

Make repeated work harmless

Use source identity and revision to recognise duplicate events. Before publishing a processed result, check that it has not been superseded. A slow worker handling revision 17 must not overwrite revision 18 simply because it finishes later.

The exact enforcement mechanism depends on the storage system. A serialised per-document worker, conditional write or separate active-revision record can establish the ordering. Do not assume that sending operations in order guarantees their completion order across independent workers.

Keep deletion events in the same version discipline. Otherwise an old update can recreate a document after its withdrawal. A retained deletion marker can prevent this, provided its retention covers the possible replay period.

Verify the serving result

After publication, query for a fixture that identifies the new revision and confirm the old one is no longer eligible. If answers are cached, include cache invalidation or source-version checking in this completion path.

Expose the three delays separately: detection, processing and serving visibility. Alerting can then point to a connector backlog, a parser failure or delayed index publication instead of treating every stale answer as the same incident. A useful progress record tells an operator where time was spent and which revision a reader can actually receive.