> ## Documentation Index
> Fetch the complete documentation index at: https://developers.bloobank.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Best practices

> Build a robust webhook consumer — idempotency, ordering, deduplication, fast acks, and observability.

A webhook consumer that "works on the happy path" is easy. A consumer that survives retries, duplicates, out-of-order events, partial failures, and timeouts is the actual goal. This page is a checklist of the patterns that get you there.

For the delivery contract, see [Retry & delivery](/get-started/webhooks/retry-and-delivery). For the event catalog, see [Payment events](/get-started/webhooks/payment-events).

## The non-negotiables

<CardGroup cols={2}>
  <Card title="1. Verify the signature first" icon="signature">
    Before parsing, before logging, before anything. Untrusted input is a vulnerability. See [Verifying signatures](/get-started/webhooks/signature-verification).
  </Card>

  <Card title="2. Deduplicate by messageId" icon="copy">
    Persist every `messageId` you have processed. On receipt, check first — if already seen, ack `200` without reprocessing.
  </Card>

  <Card title="3. Respond within 10 seconds" icon="bolt">
    Hard timeout. Do real work asynchronously; ack first.
  </Card>

  <Card title="4. Use data.status / data.ordVersion as truth" icon="route">
    Events may arrive out of order. Never trust the event name alone to infer current state.
  </Card>
</CardGroup>

## The idempotent consumer pattern

```mermaid theme={"dark"}
flowchart TD
    A[Webhook arrives] --> B[Verify signature]
    B -->|fail| Z1[401, no body]
    B -->|ok| C[Read messageId]
    C --> D{Already<br/>processed?}
    D -->|Yes| E[ACK 200<br/>do nothing]
    D -->|No| F[Insert messageId<br/>into processed table]
    F --> G[Enqueue async job<br/>with full payload]
    G --> H[ACK 200]
```

Two patterns to internalize:

### Pattern A — synchronous dedupe

If your work is fast (well under the 10-second budget), you can do it inline:

```ts theme={"dark"}
async function handleWebhook(req, res) {
  const event = verifySignature(req.rawBody, req.headers, SECRET);
  if (!event) return res.status(401).end();

  // Atomic insert — if it returns conflict, this is a duplicate
  try {
    await db.processedMessages.insertOne({ messageId: event.messageId, at: new Date() });
  } catch (e) {
    if (isDuplicateKey(e)) return res.status(200).end();   // already processed
    throw e;
  }

  await applyEvent(event);            // your business logic
  res.status(200).end();
}
```

The atomic insert is the dedupe gate. If the same `messageId` arrives twice concurrently, only one passes the insert; the other returns a duplicate-key error and acknowledges without processing.

### Pattern B — async dedupe (recommended for any non-trivial work)

For anything that touches the database, calls an external service, or might be slow:

```ts theme={"dark"}
async function handleWebhook(req, res) {
  const event = verifySignature(req.rawBody, req.headers, SECRET);
  if (!event) return res.status(401).end();

  // Cheap dedupe check first (best-effort, race-tolerant)
  if (await alreadyProcessed(event.messageId)) {
    return res.status(200).end();
  }

  // Enqueue and ack immediately
  await queue.publish({ event });
  res.status(200).end();
}

// Worker:
async function processQueueItem({ event }) {
  // Authoritative dedupe via atomic insert
  try {
    await db.processedMessages.insertOne({ messageId: event.messageId, at: new Date() });
  } catch (e) {
    if (isDuplicateKey(e)) return;
    throw e;
  }
  await applyEvent(event);
}
```

The HTTP handler stays under 10 seconds even if downstream work is slow.

## Handling out-of-order events

Events on the same order can arrive out of order. The defensive pattern uses the monotonic version field (`data.ordVersion`):

```ts theme={"dark"}
async function applyEvent(event) {
  const order = await db.orders.findOne({ id: event.data.id });
  if (order && order.lastSeenVersion >= event.data.ordVersion) {
    // Late event — older snapshot than we have. Ignore.
    return;
  }
  await db.orders.update(
    { id: event.data.id },
    {
      $set: {
        status:           event.data.status,
        lastSeenVersion:  event.data.ordVersion,
        lastUpdated:      event.occurredAt,
        // ... other fields you care about
      },
    },
    { upsert: true }
  );
  await fireBusinessLogic(event.data);
}
```

`ordVersion` increments on every mutation of the order's content. Tracking the highest version you have seen makes your local view monotonically consistent, even when delivery order is scrambled.

## When to GET vs. trust the payload

The webhook payload reflects the resource's state **at the moment of emission**. For most uses, that is fresh enough — trust it.

You should `GET` the resource fresh when:

* You need fields that are not in the webhook payload (rare — `data` is the full resource).
* A long delay elapsed between the event's `occurredAt` and your processing time.
* Your business logic depends on the **absolute current** state, not the state-at-emission.

`GET` is also useful for **reconciliation**: at daily intervals, list orders by `updatedAt >= yesterday` and verify each matches your local record.

## Fast-ack patterns

The hard limit is 10 seconds. Strategies to stay well under:

| Strategy                            | When                                                                                                  |
| ----------------------------------- | ----------------------------------------------------------------------------------------------------- |
| **Enqueue + ack** (Pattern B above) | Always preferable. Queue can be SQS, Kafka, Redis Streams, or in-process.                             |
| **In-memory async**                 | Lightweight tasks (e.g., update a single row). Risk: process crash loses the event before processing. |
| **Inline processing**               | Only when work is provably bounded (e.g., a CPU-only computation taking \< 100 ms).                   |

Avoid:

* HTTP calls to slow third parties (especially payment networks) inside the handler.
* Synchronous email sends.
* Database transactions that may deadlock.

Move all of these to a worker.

## Replay-safety per event class

A retry can deliver the same event multiple times. Your business logic must be safe to apply more than once.

| Event                   | Replay-safe pattern                                                                                                       |
| ----------------------- | ------------------------------------------------------------------------------------------------------------------------- |
| `payment_order.created` | Upsert into local order table. Second time updates nothing.                                                               |
| `payment_order.success` | Mark order as paid. Increment a settled-counter only on **first** observation; the dedupe table prevents double-counting. |
| `payment_order.failed`  | Mark order as failed; alert operator on **first** observation.                                                            |
| `payment_order.expired` | Mark order as expired; release any local hold.                                                                            |

The general rule: **state changes** are safe to retry because state is overwritten. **Side effects** (email, log entry, counter increment) must be gated behind the dedupe insert.

## Observability

Track these per-event and aggregate:

| Metric                                         | Why                                                                                                                                                         |
| ---------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Webhook ack latency (p50, p99)                 | Detect when you are approaching the 10-second timeout.                                                                                                      |
| Webhook ack status code distribution           | Identify endpoints returning 4xx/5xx.                                                                                                                       |
| Signature verification failures (count)        | Should be near zero; non-zero is a key drift or attempted abuse.                                                                                            |
| Duplicate `messageId` rate                     | Some duplicates are expected (retries on transient receiver issues). A spike indicates BlooBank-side retries — usually correlates with downstream slowness. |
| Event-name distribution                        | Anomalous spikes in `payment_order.failed` are operationally significant.                                                                                   |
| Worker queue depth                             | Backlog growth means you cannot keep up — scale workers.                                                                                                    |
| End-to-end lag (`occurredAt` → processed time) | The user-visible "did my system see the event yet" metric.                                                                                                  |

Alert when:

* Verification failure rate > 0 sustained for 5 minutes (key issue or attack).
* Ack p99 > 3 seconds (heading toward timeout).
* Queue depth grows monotonically for more than 10 minutes.

## Common bugs (and their symptoms)

| Bug                                                                     | Symptom                                                                                                                                                |
| ----------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Parsing the body before verifying                                       | Signature mismatch on every delivery. Verify first, parse second.                                                                                      |
| Logging the raw signature header                                        | Secret-adjacent material in logs. Log only the verification result.                                                                                    |
| Returning `400` on JSON parse error                                     | Permanently failed delivery in BlooBank's queue. Use `500` for any error you cannot classify.                                                          |
| Storing `messageId` in memory only                                      | After process restart, you reprocess everything. Persist to durable storage.                                                                           |
| Slow inline work                                                        | Intermittent timeouts and BlooBank retries. Move to a worker.                                                                                          |
| No timeout in your HTTP client (when you `GET` the resource in handler) | Cascading slowness — your `GET` hangs, your ack misses 10 s, BlooBank retries, repeat. Bounded timeout (≤ 3 s) is mandatory.                           |
| Trusting event-name ordering                                            | Race conditions when events arrive out of order. Use `ordVersion`.                                                                                     |
| Reusing `messageId` for business id                                     | If two unrelated events accidentally collide on a key your code derived from `messageId`, you may skip processing. `messageId` is for dedupe **only**. |

## Endpoint hardening

| Hardening                    | Why                                                                                                                                               |
| ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- |
| HTTPS only                   | Mandatory. The platform refuses HTTP.                                                                                                             |
| Path obfuscation             | A receiver at `/webhooks/bloobank-7a8b9c` is harder to discover. Not a substitute for signature verification.                                     |
| Rate limit your own endpoint | A misconfigured retry-storm should not overwhelm downstream. Bound concurrency to a known-safe level.                                             |
| Restrict by IP if available  | If BlooBank publishes egress IP ranges (check the Dashboard), allowlist them at the edge. Compose with signature verification, do not replace it. |

## Next

<CardGroup cols={3}>
  <Card title="Verifying signatures" icon="signature" href="/get-started/webhooks/signature-verification">
    Verify every delivery.
  </Card>

  <Card title="Retry & delivery" icon="paper-plane" href="/get-started/webhooks/retry-and-delivery">
    The delivery contract.
  </Card>

  <Card title="Payment events" icon="credit-card" href="/get-started/webhooks/payment-events">
    Event catalog.
  </Card>
</CardGroup>