It is “What happened, what changed, what is safe to repeat, and who owns the next decision?”
01 / FAILURE MODEL
Give each failure a name before giving it a retry.
“The automation failed” is too broad to operate. A useful design separates where trust broke and what the system knows at that point.
| Failure class | Example | Control question |
|---|---|---|
| Input | A required field is absent or malformed. | Can the item be rejected without changing another system? |
| Dependency | An API times out, rate-limits, or returns a temporary server error. | Is a later retry safe, bounded, and observable? |
| Logic | A mapping or business rule cannot produce a valid result. | Should code stop, quarantine the item, or request judgment? |
| State | The same event arrives twice or arrives out of order. | Which stable identifier and state transition prevent duplication? |
| Side effect | The remote write may have succeeded before the response was lost. | Can the system confirm the outcome before repeating the action? |
| Visibility | The run stops without an actionable record or alert. | Who will notice, what evidence will they receive, and what can they do? |
The classification does not need to be elaborate. It needs to change behaviour. If two failures have different safe next actions, they should not collapse into the same generic error.
02 / RETRY BOUNDARY
A retry is a controlled decision—not optimism in a loop.
Retry only when the failure is plausibly temporary and the action can be repeated safely. Use a finite budget, record each attempt, and promote exhausted work to an explicit terminal state.
Stable run and item identifiers, attempt number, next eligible time, failure class, dependency response, and the rule that made another attempt safe.
Invalid input, rejected authorization, broken schemas, and deterministic logic errors usually require correction or review—not another identical attempt. A retry budget that cannot end is an outage generator.
03 / DUPLICATE EFFECTS
Separate “repeat the computation” from “repeat the side effect.”
Recomputing a transformation may be harmless. Sending a second invoice, creating a second CRM record, publishing twice, or charging twice is not. The workflow needs a stable identity for the business event and a record of the intended transition.
- Choose an idempotency key derived from the business event, not a random retry.
- Record the requested action before or with the side effect when the target contract permits it.
- Check the destination by stable identifier when a response is ambiguous.
- Do not mark a run successful until the expected state can be verified.
- Make manual replay follow the same duplicate controls as automatic retry.
04 / TERMINAL STATES
Some work must stop and wait for a person.
A terminal failure is not the same as disappearing. Preserve the last trusted state, the failed step, the evidence, the action already taken, and the choices still available. Then route the item to an accountable queue.
The operator should be able to distinguish “safe to retry,” “input must be corrected,” “dependency contract changed,” “side effect is uncertain,” and “human judgment required.” A single red badge without those distinctions creates more investigation work.
05 / RUN EVIDENCE
Logs are useful only when they answer an operating question.
For each run, retain a stable identifier, workflow version, start and end times, step transitions, sanitized input references, validation outcome, external request identifiers, retry history, terminal state, and accountable decision. Avoid copying credentials or unnecessary personal data into logs.
Observable failure is a product feature. It turns an invisible exception into work the team can own.
06 / REVIEW CHECKLIST
Before calling a workflow reliable.
- Can every run be found by a stable identifier?
- Are retryable and terminal errors distinguished?
- Is every retry finite, delayed appropriately, and visible?
- Can a repeated event avoid duplicate external effects?
- Can an ambiguous side effect be reconciled before replay?
- Does terminal work enter an accountable queue?
- Does the operator receive enough evidence to choose the next action?
- Are recovery instructions tested and documented?
INSPECTABLE REFERENCES
Two failure-control views.
Loopwatch shows step history, recovered retry, permanent failure, human review, five passing API tests, and zero silent demo failures. The n8n Reliability Lab adds five fixtures, seven fail-closed invariants, and zero external actions in a native workflow. Each is not client work or a production-readiness claim.