The non-negotiables
1. Verify the signature first
Before parsing, before logging, before anything. Untrusted input is a vulnerability. See Verifying signatures.
2. Deduplicate by messageId
Persist every
messageId you have processed. On receipt, check first — if already seen, ack 200 without reprocessing.3. Respond within 10 seconds
Hard timeout. Do real work asynchronously; ack first.
4. Use data.status / data.ordVersion as truth
Events may arrive out of order. Never trust the event name alone to infer current state.
The idempotent consumer pattern
Two patterns to internalize:Pattern A — synchronous dedupe
If your work is fast (well under the 10-second budget), you can do it inline:messageId arrives twice concurrently, only one passes the insert; the other returns a duplicate-key error and acknowledges without processing.
Pattern B — async dedupe (recommended for any non-trivial work)
For anything that touches the database, calls an external service, or might be slow:Handling out-of-order events
Events on the same order can arrive out of order. The defensive pattern uses the monotonic version field (data.ordVersion):
ordVersion increments on every mutation of the order’s content. Tracking the highest version you have seen makes your local view monotonically consistent, even when delivery order is scrambled.
When to GET vs. trust the payload
The webhook payload reflects the resource’s state at the moment of emission. For most uses, that is fresh enough — trust it. You shouldGET the resource fresh when:
- You need fields that are not in the webhook payload (rare —
datais the full resource). - A long delay elapsed between the event’s
occurredAtand your processing time. - Your business logic depends on the absolute current state, not the state-at-emission.
GET is also useful for reconciliation: at daily intervals, list orders by updatedAt >= yesterday and verify each matches your local record.
Fast-ack patterns
The hard limit is 10 seconds. Strategies to stay well under:
Avoid:
- HTTP calls to slow third parties (especially payment networks) inside the handler.
- Synchronous email sends.
- Database transactions that may deadlock.
Replay-safety per event class
A retry can deliver the same event multiple times. Your business logic must be safe to apply more than once.
The general rule: state changes are safe to retry because state is overwritten. Side effects (email, log entry, counter increment) must be gated behind the dedupe insert.
Observability
Track these per-event and aggregate:
Alert when:
- Verification failure rate > 0 sustained for 5 minutes (key issue or attack).
- Ack p99 > 3 seconds (heading toward timeout).
- Queue depth grows monotonically for more than 10 minutes.
Common bugs (and their symptoms)
Endpoint hardening
Next
Verifying signatures
Verify every delivery.
Retry & delivery
The delivery contract.
Payment events
Event catalog.