START WITH THE CONTRACT
Write down how this sender actually delivers events.
Retry, retention, redelivery, ordering, signature, and delivery-log behavior are provider-specific. Do not recover from memory or from a different vendor's rules. Record the exact source, event types, endpoint version, stable delivery identity, signing method, retry policy, manual-redelivery path, and evidence-retention window.
Stripe documents automatic retries and possible duplicate delivery. GitHub documents that failed deliveries are not automatically redelivered and that events can arrive out of order. A recovery runbook must name the real sender's behavior rather than assume one universal webhook model.
Primary references: Stripe webhook delivery behavior, GitHub failed-delivery handling, and GitHub delivery evidence.
01 / CAPTURE
Preserve six evidence groups before replay.
- 01Expected source event
Business object, event type, source timestamp, and why the event should exist.
- 02Provider delivery
Delivery or event ID, attempt times, response status, endpoint, and provider-visible error.
- 03Receiver decision
Signature result, schema result, canonical event hash, duplicate/conflict decision, and committed state.
- 04Downstream effect
Stable destination ID, write status, response or lookup evidence, and whether the outcome is certain.
- 05Retry history
Attempt number, idempotency key, failure class, next decision, and terminal limit.
- 06Reconciliation set
Expected source IDs versus committed receiver IDs for the bounded incident window.
Minimize retained data. Store identifiers, hashes, states, and timestamps when they are enough; do not copy credentials, full payloads, or personal fields into an incident report by default.
02 / CLASSIFY
Choose the recovery path from the last trusted state.
| Observed state | What it proves | Safe next question |
|---|---|---|
| Not present in sender log | No delivery attempt is evidenced. | Was the expected source event created and subscribed? |
| Sent and rejected | The receiver returned a failure or could not be reached. | Is correction or bounded redelivery supported? |
| Accepted, not committed | Transport succeeded but durable processing is absent. | Can the event be re-enqueued with the same identity? |
| Committed, downstream uncertain | The receiver intended a write but lacks a trusted acknowledgement. | Can the destination be queried before any repeat? |
| Exact duplicate | The same event identity and canonical content already exist. | Can it return the recorded outcome without another effect? |
| Identity conflict | The same event ID carries different canonical content. | Who owns the conflict decision and source correction? |
| Missing in reconciliation | The expected source set exceeds the committed receiver set. | Which exact missing IDs are authorized for recovery? |
03 / RECOVER
Make replay use the same controls as first delivery.
- Pause unbounded retries and preserve the incident window.
- Verify the sender signature against the exact received bytes before parsing.
- Use one stable event identity and canonical content hash across every attempt.
- Check the destination by stable business ID when a prior effect is uncertain.
- Commit durable processing intent before dispatching the downstream action.
- Replay only the exact authorized missing or failed events.
- Keep conflict and invalid-input paths out of automatic retry.
- Cap attempts and move exhausted work to a visible terminal state.
- Reconcile the source and receiver sets again after recovery.
04 / ACCEPTANCE GATE
Twelve checks before closing the incident.
- The accountable system owner confirms the recovery scope.
- The sender, subscription, event type, and endpoint version are identified.
- The signature-verification rule and raw-byte boundary are documented.
- Every expected event has a stable source identity.
- Exact replay and conflicting replay produce different controlled outcomes.
- Retryable, invalid, conflict, review, and terminal states are distinguishable.
- The attempt budget is finite and visible.
- An ambiguous downstream write is queried before it is repeated.
- Recovery uses the same idempotency key as the original event.
- Reconciliation proves no expected event remains missing.
- Logs and handoff evidence exclude unnecessary secrets and personal data.
- The operator has a tested runbook for the next occurrence.
HONEST LIMITS
A checklist cannot supply authority or production evidence.
This resource is general engineering guidance, not client work, a vendor-specific implementation, a security assessment, legal advice, or proof that a production incident is resolved. Real recovery still requires authorized access, the actual provider contract, client-owned data and retention decisions, an accountable release owner, backups where relevant, and environment-specific tests.