Webhook Recovery Lab

THE DELIVERY RISK

A successful webhook response does not prove that every event arrived once.

Cross-system contact workflows fail quietly when signatures are ignored, source events disappear, retries repeat downstream writes, invalid records bypass review, or a destination accepts a write just before the worker loses its local acknowledgement. The lab makes each of those states explicit and compares the accepted ledger with an expected source manifest.

CONTROL PATH

Commit the recovery evidence before calling the delivery complete.

1. 01
   Verify

   Check HMAC-SHA256 over the exact request bytes and enforce a 16 KiB body ceiling before parsing or persistence.

2. 02
   Validate

   Require one exact six-field contract, a timezone-aware timestamp, bounded identifiers, contact permission, and deterministic fixture validation.

3. 03
   Minimize

   Canonicalize the event, bind its ID to a payload hash, store only the contact hash, and quarantine conflicting replay evidence without raw payloads.

4. 04
   Dispatch

   Commit a transactional outbox before delivery, record one attempt at a time, and stop after the fixed three-attempt ceiling.

5. 05
   Recover

   Reuse one stable sink idempotency key when delivery succeeded but local acknowledgement was deliberately lost.

6. 06
   Reconcile

   Compare seven expected source IDs with the event ledger, detect one withheld event, ingest it, and prove the final set is complete.

VERIFICATION EVIDENCE

What was actually executed.

• Native local runtimeThe committed fixtures ran on Python 3.12.13 and SQLite 3.50.4 with zero third-party runtime dependencies.
• Seven accounted eventsFour finished delivered, two entered review after invalid or provider-error fixture outcomes, and one became terminal after the three-attempt ceiling.
• Replay and conflict policyAn exact semantic replay added no event, outbox, attempt, audit, quarantine, or sink call. A changed payload under the same event ID preserved the original and added one hash-only quarantine record.
• Transient and terminal behaviorOne event delivered on its second attempt; another remained visibly terminal after three synthetic transient failures.
• Ambiguous delivery recoveryThe synthetic sink accepted one event before an injected local acknowledgement loss. Retry returned one duplicate acknowledgement while the sink retained exactly one unique delivery.
• Source reconciliationThe first comparison reported missing evt-007. After bounded recovery, all seven expected IDs were present with zero unexpected IDs.
• Database and privacy boundarySQLite integrity was ok with zero foreign-key violations, raw payloads, raw emails, network actions, client-system actions, or external actions.
• Sixteen automated testsThe suite covers signature, size, contract, replay, conflict, review, retry, terminal, ambiguous-delivery, reconciliation, rollback, integrity, and raw-address absence. It also caught and fixed an open SQLite handle on Windows.