ZZZ Digital Lab

n8n workflow troubleshooting and hardening: synthetic demo

Synthetic data only. Self-initiated demonstration; no client engagement or independent human certification is claimed.

I use AI coding tools; every change ships with automated tests and a written handoff.

A deliberately broken lead-event workflow has no retries, duplicate processing, no error workflow, an unbounded back-edge and a fake secret marker inline. The hardened export adds bounded retry/backoff, stable idempotency keys, a failure branch, a dead-letter sink, an error workflow and credential references.

Before and after

Run offline

Requires Node 20+. No npm dependencies, accounts, requests or installation are needed:

node tests/run.cjs
node lint.cjs after/main.json after/error-workflow.json
node demo.cjs

The deliberately broken export should fail the linter with exit 1:

node lint.cjs before/broken.json

Linter exit 0: no findings under this project's policy. Exit 1: policy findings. Exit 2: invalid/unreadable/non-JSON input. All examples remain inactive and all service destinations use .invalid placeholders. Do not activate the baseline. Its SYNTHETIC_NOT_A_SECRET marker is intentionally fake, included to reproduce the inline-value defect.

python -B build_workflows.py regenerates the JSON exports from the checked-in Code-node source files, using the Python standard library. Tests ensure embedded and standalone Code-node sources match.

Before / after evidence

Defect Hardened design Automated evidence
No retries or bounds HTTP nodes: max 3 attempts, fixed 1,000 ms delay, 5,000 ms header timeout; workflow 300-second limit Linter mutations; local transient-failure replay records attempts and waits
Duplicate processing Stable source + event ID key, same-batch deduplication, conflicting payload rejection, receiver-side atomic idempotent PUT contract Repeated execution, payload conflict and lost-acknowledgement replay
No error workflow Separate Error Trigger export, safe incident summary and incident sink Reference checks, actual incident Code execution, trigger-error input case
Unbounded loop Acyclic workflow and max 50 events per envelope Back-edge detection and oversized input rejection
Secrets inline Locally configured credential references; webhook header-auth reference Missing-reference and inline-header mutation checks; values redacted from findings
No dead-letter recovery Original linked event persisted using deterministic dead-letter key; then Stop and Error Actual dead-letter Code execution, permanent failure, storage failure and recovery replay

Recorded result: 29/29 tests pass, 14 baseline findings, 0 hardened findings. Local replay accepts 2 distinct events, then 0 new events on duplicate replay; transient failure needs 3 attempts; permanent failure creates 1 review record; recovery accepts 1 event. These are local policy/contract results.

flowchart LR
  W[Authenticated webhook] --> P[Validate and bound events]
  P --> D[Deliver with idempotency key]
  D -->|success| S[Success]
  D -->|exhausted retries| B[Build linked dead letter]
  B --> Q[Persist dead letter]
  Q --> F[Stop and Error]
  F -. execution failure .-> E[Separate error workflow]
  E --> I[Persist safe incident summary]

Receiver and operator contract

Idempotency-Key is lead-event:v1:<source>:<event_id>. The receiving service must atomically store one canonical payload per key and reject a different payload for the same key. A replay of the same payload returns the original outcome without a second business side effect. Merely adding the header does not make an arbitrary API idempotent. The supplied in-memory fake receiver demonstrates this contract; no durable receiver service is shipped or deployed.

The dead-letter service must provide durable upsert by dead_letter_key. The incident service must do the same by incident_key. These are integration contracts, not bundled external products. Successful recovery does not silently delete the dead-letter record; an operator reviews and closes it after reconciling the receiving system.

The webhook is configured to acknowledge after the last node. Proxy/client timeout and delivery replay policies must be agreed in staging. A workflow timeout may interrupt a large batch; replay the whole original envelope using the same keys. Validation errors retain evidence in failed executions subject to the instance's retention policy. Rejecting input is not equivalent to durable queuing of unaccepted input. This example's native retry setting can retry permanent errors as well as transient failures; a production-specific status-code/Retry-After policy is separate scope.

Limits, documentation and review

No callable n8n CLI was found on PATH, and n8n was not run in this job. No import, activated webhook, credential binding, native retry timing, item-linking runtime, error-trigger delivery or durable sink was tested. The exports use standard node fields and recorded type versions, but they still need import/compatibility validation in the client's installed n8n version. All placeholder IDs must be rebound locally before staging.

lib/replay.cjs runs the exact embedded Code-node JavaScript in a Node VM and models the exported HTTP retry policy with a fake receiver. It does not execute HTTP Request nodes or n8n scheduling. The static linter enforces a narrow policy for these workflow shapes. It is not a full schema validator, complete secret scanner, arbitrary JavaScript proof or general security audit. Source-text checks for bounds cannot prove all possible scripts terminate.

Review using AI coding tools covered stable identity, payload conflicts, credential references, graph termination, error routing, linked dead-letter payloads and failed persistence. The test harness normalizes VM results into host objects before comparing them. Client acceptance remains necessary.

The public HTTP Request documentation describes credential selection and request options. The node settings guide describes retry and error-output behavior. The Error Trigger guide explains automatic-error testing and publication requirements; manual execution alone does not verify that trigger. These public pages were read without sending any repository or fixture data.

Read DIAGNOSIS_REPORT.md and HANDOFF.md for scope, staging acceptance and rollback. License: MIT.

Verification and recovery

Regenerate portfolio images

After generating the evidence, optionally run python -B make_images.py. This requires locally installed Pillow; the PDF project also uses PyMuPDF for source-page thumbnails and openpyxl for stored-workbook previews. No font or image asset is downloaded. The PNGs are 1000x750 and derived from the recorded outputs. Native app screenshots are not claimed.