Manaz Afzal Book a consultation

Work snippet

Idempotent webhook intake and a versioned field map in n8n

Several vendors post webhooks into this service group, and each used to have its own flow and mapping. Now a single n8n entry point checks the signature, keeps the raw payload, maps fields from a versioned table, and writes to the job system or CRM keyed on the event, so a resent event updates the same record instead of adding one.

The problemEach webhook has its own fragile flow, and vendor retries create second records.

What changesOne intake, one versioned field map, and no duplicates from retries.

Area
Inbound integration architecture
Tools
n8n · ServiceM8 · HubSpot
Typical scope
US$1,600-3,000
Timeline
About 4 weeks · fixed milestones

The problem

Separate mappings per vendor, several with no unique key and nothing saved, meant a resent event created a duplicate and a failed run had nothing to replay from.

  • The same customer fields are spelled differently in each flow
  • A vendor retry has already created a second job
  • A failed mid-run cannot be replayed because the body is gone
  • A live edit to one mapping leaves no current version

Discovery findings

A review of eight inbound webhooks into n8n for a service group that already has several point-to-point zaps, plus two weeks of execution history.

Findings from the review
IDFindingEvidenceRisk if left
F1Each webhook has its own mapping, with the same customer fields spelled three waysEight workflows, three names for "mobile"A later change has to be made in several places, and some are missed
F2Two webhooks have no idempotency keyA vendor retry created a second job and a second contactDuplicates that operations then has to merge
F3Payloads are not stored, so a failed mid-run cannot be replayed safelyExecution log has the error, not the bodyThe vendor must be asked to resend, or the item is lost
F4A change to a downstream field was edited live in one flow, with no version noteThe mapping in n8n disagreed with the written sheetNobody could say which was current
F5Slow vendors time out, and the same event arrives again laterTimeouts followed by a second postWithout a key, the second post is treated as new work

Approach

  1. Put every event through one intake

    Verify, store, key.

  2. Hold one mapping table

    Versioned, dated, and used by every writer.

  3. Make every writer idempotent

    The same key cannot create a second record.

  4. Replay from what was stored

    After a fix or a mapping change, with the version recorded.

Current state and future state

Intake pattern, current and future

Current state

Vendor AOwn n8n flow
Vendor BOwn mapping
Vendor CNo key, no store
DestinationsJobs, CRM, sheets

Future state

Any signed webhookOne intake
Store and keyPayload kept
Shared mapVersioned fields
DestinationsIdempotent writers

Future-state modules

VendorsSigned webhooksIntakeVerify, store, keyMapping tableVersionedWritersJob, CRM, sheetReplayFrom stored payload

How it is built

Module-level build
  1. Module AIntake and verify

    One n8n entry workflow checks the signature, rejects stale events, and stores the raw payload.

  2. Module BIdempotency key

    A written key per vendor, usually event id plus vendor name. A known key is not written again.

  3. Module CMapping table

    A single table of source paths to destination fields, with a version and a date.

  4. Module DIdempotent writers

    Each destination uses the same key, so a replay cannot create a second record.

  5. Module EReplay and versioning

    A stored payload can be replayed after a mapping change. The version used is recorded on the run.

Field mapping

Shared mapping table, extract
Destination fieldSource path (typical)TransformIf missing
External event keyevent.id or delivery idPrefix with the vendor codeRefuse the event, alert
Customer emailcustomer.email or contact.emailLower-case, trimPark if no phone either
Mobilecustomer.mobile or phoneNormalise to a single formatPark if no email either
Site line 1, suburb, postcodesite.address_*Join only when all required parts existPark with the missing part named
Requested windowschedule.start / schedule.endStore as received, in the written timezoneJob is unscheduled
Source systemVendor name from the intakeNever taken from a free-text fieldAlways set by the intake, not the payload

Worked exhibits

Two weeks of intake, walkthrough figures
Events received3,420
Accepted and stored3,371
Refused, bad signature or stale18
Parked by mapping31missing required fields
Retries that did not create a duplicate142same event key
A stored event ready for replay
Intake record
vendorVendor A
event keyA-100883
received08:14:07
statusStored, written
map versionv3, dated 18 May
payloadKept in full
Writer result
ServiceM8 job17688
key writtenA-100883
replay at 10:02No second job
CRM contactMatched on email

Exceptions and runbook

Exception runbook
CaseRuleWho actsFirst action
Unknown field in a new payload versionPark the event, do not guess a mappingOperations and me, if monthly support is in placeAdd a versioned row to the map, then replay
Missing required fieldPark, do not create a half recordOperationsAsk the vendor or complete by hand, then replay
Vendor retry after a timeoutThe key already exists: do not write again, return success to the vendor if the first write completedFlowIf the first write did not complete, replay from the stored payload
Mapping changeA new version is dated; old stored events can be replayed under the new version only if the owner agreesOwnerThe run record shows which version was used

How to read the exhibits

  1. The two-week counts show intake, refusals and parked events across the flow.
  2. The stored event is what makes a replay safe after a mapping change.
  3. Source system is set by the intake, not by a free-text field in the payload.

Test plan

PASS / FAIL checks
IDCheckMethodPASS whenFAIL when
T1Happy pathPost a complete signed event for each vendorOne stored payload, one destination writeA missing store, or a write with no key
T2RetryPost the same key twiceOne destination recordA second job or contact
T3Bad signaturePost with a wrong secretRefused, not stored as a valid eventA record is created
T4Missing fieldOmit the email and the phoneParked, no destination writeA record with a blank customer
T5Replay after a map changeChange a mapping version, replay a stored eventThe run records the new version; still one destination recordA second record, or the version is not recorded

Scope

AreaIncluded
DiscoveryExisting webhook flows, keys, and two weeks of history, with written findings
Shared intakeSignature checks, payload store and event keys
Versioned mapping tableSource paths, transforms and missing-field rules
Idempotent writersTo the agreed destinations
ReplayFrom stored payloads, with the mapping version recorded
Test planThe PASS and FAIL checks in this write-up
Handover and supportSOPs, walkthrough, 14 days of fixes, optional monthly hour block
Not includedRebuilding the vendors themselves, or adding a new destination that is not agreed in the scope

Rollout, handover and monthly support

MilestoneDeliverableTimingShare of fee
M1 Discovery and keysFindings, the key design and the first mapping versionWeek 125%
M2 Intake and writersStore, map and write in a copy, replay safeWeek 235%
M3 Remaining vendors and testsAll agreed vendors on the shared intakeWeek 325%
M4 HandoverSOPs, versioning rules, 14 days of fixesWeek 415%
Handover and monthly support

Handover

  • Written operating procedures for the happy path and each exception in the runbook
  • A recorded walkthrough of the architecture, the queue and how to replay an item
  • A credentials and ownership list, with every connection in your name
  • Fourteen days of fixes after go-live, for defects in the agreed scope

Monthly support

A set block of hours each month, at US$35 per hour, agreed in writing before it starts. Typical use of the block: new vendor fields, mapping versions, and parked events that need a person

  • Review of the exception queue and the weekly digest
  • Small mapping or rule changes, agreed before they are made
  • A short written note on what was done and what is still open

Typical scope for this kind of project: US$1,600-3,000

About 4 weeks · fixed milestones. The systems health check (US$150, fixed) comes first and confirms the scope. Extra hourly work is US$35 per hour, agreed before it starts. After handover, a set block of hours each month is available at the same rate, agreed in writing, to watch the flows, handle changes and work the exception queue.

Discuss an engagement.

Outline the operational problem through the contact form. I respond with a recommended starting point, including whether the US$150 systems health check is the appropriate first step.

Related work snippets

Next step

Bring me the work that repeats.

I scope it in writing, test it against your current process on your own data and hand over everything I build.

Book a consultation