Webhooks and Idempotency: A Handler That Never Double-Fulfills

A merchant's server sends POST /v1/payments. Suward creates the payment, starts writing the response, and the connection drops before a single byte reaches the client. From the merchant's side, the call looks like it failed. It didn't. That sequence isn't a malfunction - it's what a network does under load, and any integration that assumes otherwise will eventually create a second payment for an order that already has one.

The same shape of problem shows up on the other side of the wire. Suward delivers webhooks at-least-once, which is a precise phrase, not a hedge: the same eventId can and will reach your endpoint more than once. A handler that isn't correct under duplication isn't reliable, it's untested against the one condition guaranteed to occur at volume.

Retries are not a bug in the protocol

Picture the exact failure from the intro played out on the receiving end instead. Suward POSTs payment.accepted to your webhook URL. Your server runs the handler, commits the work, then starts writing back 200. The response never arrives - a load balancer restarts mid-request, a proxy times out, whatever. Suward's delivery worker sees a failed attempt and does the only sane thing: it retries.

That retry schedule is not aggressive. The first retry lands 30 seconds later, then it doubles - 1 minute, 2 minutes, 4 minutes - capping at 24 hours between attempts, for up to 20 attempts total. The last attempt lands roughly eight and a half days after the first. Every response outside the 2xx range counts as a failure worth retrying, and so does a timeout. X-Suward-Attempt tells you which attempt you're looking at, counting from 1.

None of that is generous by accident. Your endpoint will occasionally be unavailable exactly when an event arrives: maybe down, maybe just slow. A gateway that fired once and gave up would silently lose money-moving events, so the cost of that design lands on you instead - your handler has to produce the same result whether an event arrives once or five times.

Idempotent creation stops the duplicate invoice before it starts

Retries aren't only a webhook problem. The same physics apply to the request that creates a payment in the first place, and Suward closes that gap with externalId rather than a separate idempotency-key header you'd have to invent yourself.

Send POST /v1/payments with externalId: "order-42", lose the response to a network blip, and retry with the identical externalId. Suward returns 409 instead of a second payment - fetch the existing resource instead of creating another. That single field turns a network failure from a decision your code has to make ("did that actually go through?") into a lookup it can perform safely every time. Any create call carrying externalId is safe to retry on a timeout or a 5xx; a 400 is not, because the payload itself is wrong and retrying just repeats the mistake.

Every Payment Is a State Machine covers the other half of this guarantee - the atomic write that keeps payment status and merchant balance from ever disagreeing.

Verify the raw bytes, not the object your framework builds

Suward signs every webhook body with Ed25519 and sends the signature as X-Suward-Signature, formatted ed25519=<hex>. Verification runs against the exact bytes that left Suward's server - not a JSON object your framework parsed, then handed to your route.

That distinction breaks integrations constantly, and it's almost always Express. express.json() parses the body before your handler sees it, and when your code later calls JSON.stringify to re-derive "the bytes," the result has different whitespace and a different key order than the original. Ed25519 is exact - one byte off anywhere in the message and the signature simply fails. The fix is to capture the raw body on this route specifically, before any JSON middleware touches it:

app.post(
  "/webhooks/suward",
  express.raw({ type: "application/json" }),
  webhookHandler
);

Verify against those raw bytes, and only parse JSON afterward, once you've decided the request is genuine. webhookPublicKey - available from your project settings in the dashboard - is a PKIX/SPKI PEM block starting with -----BEGIN PUBLIC KEY-----, not a raw hex string. Parse the PEM first, then verify with the Ed25519 key it holds.

The replay window closes the gap signing alone leaves open

A valid signature proves the body came from Suward. It says nothing about when. Without a freshness check, a captured request - logged by a proxy, replayed by an attacker, replayed by a bug - would verify successfully forever.

X-Suward-Timestamp carries the send time in Unix milliseconds, and you reject anything more than 300 seconds away from your own clock. Keep your server clock in sync (NTP, not "close enough") or you'll reject legitimate deliveries for the wrong reason.

Deduplicate on eventId, in the same transaction as the work

Every event carries a unique eventId. Delivery being at-least-once means your handler will see that id again - after a retry you triggered by returning a slow 5xx, after your own process crashed between "processed" and "acknowledged," after nothing went wrong at all and Suward's queue simply redelivered.

The fix people reach for first - a Set or a map kept in your process - looks like it works in a quick test and then fails in exactly the two ways that matter. It empties every time your service restarts, and it isn't shared across replicas, so the second instance behind your load balancer has never heard of an event the first one already handled. Both gaps let a redelivery run your fulfillment logic twice.

What actually works: a table with a unique constraint on eventId, and the insert into that table committing in the same database transaction as the business logic it guards - crediting a balance, activating a license, marking an order shipped. Either both write, or neither does. There's no window where the mark exists but the work didn't happen, or the reverse.

Delivery order is not event order

Don't infer state from the sequence deliveries arrive in. Suward doesn't guarantee it, and under retries it can't - a payment.success delivery can reach your endpoint and get processed before your handler finishes with the payment.accepted delivery for the same payment, particularly if the first attempt was slow and triggered a race with a fast retry of something else.

Compare updatedAt from the payload against whatever you last stored for that resource, and skip the write if the incoming event is not newer. That one comparison is what makes "arrived out of order" harmless instead of a source of corrupted state.

Answer with the status code that matches what actually happened

Suward's retry logic treats every non-2xx response the same way: a 4xx is retried exactly like a 5xx or a plain timeout, going straight back into the queue for another attempt. That single fact should decide your status codes.

Send 500 when something on your side is genuinely recoverable and you want the event back - your database connection is down, a downstream call timed out. Reserve 400 strictly for the case where the signature doesn't verify; nothing else earns a 4xx, because it retries exactly the same as a 5xx here, so spending it on "duplicate" or "an old state I don't need" just burns an attempt for no reason. Everything else you've already handled - a duplicate you processed before, a state older than what you have stored, an event type you don't act on - gets a plain 200.

Send that 200 only once the transaction doing the real work has committed, though. Sending it earlier - before the write lands - reintroduces the exact race idempotency is supposed to close: your process crashes after the 200 leaves but before the commit, and the event is gone for good, because as far as Suward knows, you already handled it.

A handler that holds up under all of this

import { webhooks } from "@crylabsorg/suward-sdk";
 
app.post("/webhooks/suward", express.raw({ type: "application/json" }), async (req, res) => {
  const body = req.body.toString("utf8");
  const verified = await webhooks.WebhooksHelper.verifySignature(
    body,
    req.header("X-Suward-Signature"),
    process.env.SUWARD_WEBHOOK_PUBLIC_KEY,
    req.header("X-Suward-Timestamp")
  );
  if (!verified) return res.status(400).send("bad signature");
 
  const event = JSON.parse(body);
  const resource = event.payment ?? event.staticDeposit;
 
  try {
    await db.transaction(async (tx) => {
      const claimed = await tx.query(
        `INSERT INTO webhook_events (event_id) VALUES ($1) ON CONFLICT DO NOTHING`,
        [event.eventId]
      );
      if (claimed.rowCount === 0) return;
 
      const stored = await tx.query(
        `SELECT updated_at FROM payments WHERE id = $1`,
        [resource.id]
      );
      if (stored.rows[0] && stored.rows[0].updated_at >= resource.updatedAt) return;
 
      if (event.type === "payment.accepted") await fulfil(tx, event.payment);
      if (event.type === "payment.success") await settle(tx, event.payment);
 
      await tx.query(
        `UPDATE payments SET status = $2, updated_at = $3 WHERE id = $1`,
        [resource.id, resource.status, resource.updatedAt]
      );
    });
  } catch {
    return res.sendStatus(500);
  }
 
  res.sendStatus(200);
});

Every piece above earns its place. The unique constraint on event_id is what makes a redelivered event a no-op instead of a repeated side effect, and the updated_at comparison is what makes out-of-order delivery harmless on top of that. One transaction ties the claim and the fulfillment together so both land or neither does, and res.sendStatus(200) only runs once that transaction has already committed.

When retries run out, reconcile through the API instead of trusting silence

Twenty attempts sounds like a lot until you consider what it's protecting against: an endpoint down for more than eight and a half days straight. Past that point, Suward marks the event failed and stops delivering it. Your system never hears about that state change again, and nothing recovers on its own.

That's the case a well-built integration doesn't leave to hope. Treat webhooks as the fast path rather than the only one, and add a periodic reconciliation job that reads the payments and static-wallet deposits you expect to be resolved and compares their state against what your database recorded. GET /v1/payments/{paymentId} and GET /v1/static-wallets/{staticWalletId}/deposits return the identical object shape the webhook payload carries, so the same handler logic - check updatedAt, apply the same transaction - covers a polled result exactly as well as a pushed one. Run it nightly, or hourly if your volume justifies it, against anything still pending past a reasonable window. A missed webhook becomes a delayed correction instead of a payment your business never finds out was paid.

The pattern, not just the code

Every rule above reduces to one habit: never let a webhook handler assume it's the first, or the last, time it will see a given event. Verify the bytes that were actually signed. Check the timestamp before you check anything else. Claim the eventId and do the work in one transaction. Read updatedAt instead of trusting arrival order. Answer with the status code that matches whether the event should come back. When the retries genuinely run out, go get the answer yourself instead of waiting for one that isn't coming.

None of that is exotic engineering. It's the same discipline any system applies once it stops assuming the network is reliable and starts building for the network it actually has.