Designing Webhooks That Don't Break at 2 AM

Webhooks are the simplest possible integration surface: an HTTP POST to a URL you control. They are also one of the most common sources of silent production failures. The server was down for 30 seconds during a deploy. The endpoint returned a 200 but threw before writing to the database. A race condition meant two deliveries of the same event both committed. Debugging any of these at 2 AM, against an event that happened six hours ago, is not a good time.

Here is how MailFrame plans to approach webhook delivery and the design decisions behind each piece.

Roadmap note: This post describes the planned design for MailFrame webhook delivery, which is on the roadmap and not yet available. Today, results come back synchronously in the POST /v1/parse response. The signing scheme, headers, retry schedule, dead-letter queue, replay endpoints, dashboard screens, and endpoint health monitoring described below are all planned and not yet available; treat names, paths, and payloads as the planned shape, subject to change. The receiver-side advice (verifying HMAC signatures, idempotency, fast acknowledgement) is general practice for any webhook you consume.

Signing Every Payload

Each planned webhook delivery will include a MailFrame-Signature header. The value is planned to be an HMAC-SHA256 digest of the raw request body, keyed with a secret that is unique per webhook endpoint configuration and visible only at creation time.

MailFrame-Signature: sha256=b94d27b9934d3e08a52e52d7da7dabfac484efe04294e576c9b3b1c1e4a07b9e

On your end, verification is a few lines:

import { createHmac, timingSafeEqual } from "node:crypto";

export function verifySignature(
  rawBody: Buffer,
  signatureHeader: string | undefined,
  secret: string
): boolean {
  if (!signatureHeader) return false;

  const digest = createHmac("sha256", secret)
    .update(rawBody)
    .digest("hex");

  const received = signatureHeader.startsWith("sha256=")
    ? signatureHeader.slice(7)
    : signatureHeader;

  try {
    return timingSafeEqual(
      Buffer.from(digest, "hex"),
      Buffer.from(received, "hex")
    );
  } catch {
    // Buffer lengths differ — invalid signature format
    return false;
  }
}

Two details matter here. First, compute the digest over the raw body bytes before any JSON parsing. Parsers can normalise whitespace or reorder keys, producing a different byte sequence that breaks verification. Buffer your framework’s raw body; in Express that means express.raw({ type: 'application/json' }) on the route. Second, always use timingSafeEqual — === on hex strings is vulnerable to timing side-channel attacks that can let an attacker brute-force the correct signature byte by byte.

Signing-secret rotation is planned as well (not yet available): during a rotation, the planned design delivers with both the old and new secrets for an overlap window so you can cut over without a gap.

Idempotency Keys

In the planned design, every delivery attempt will carry a stable X-MailFrame-Delivery-Id header. The value would be the same across all retry attempts for a given event — changing only for distinct parse completions.

X-MailFrame-Delivery-Id: dlv_01jx9m4kz3p8q2n7
X-MailFrame-Event-Id:    evt_01jx9m4kz3p8q2n7
X-MailFrame-Attempt:     2

Store the X-MailFrame-Delivery-Id in your database as a unique constraint on receipt. If you receive the same delivery ID twice — which happens legitimately when your endpoint returned a non-2xx on attempt one and the sender retried — your insert will fail the unique constraint and you can return 200 safely without double-processing. This is the simplest reliable idempotency pattern and it requires nothing more complex than a unique index.

Retry Schedule and Exponential Backoff (planned)

Automatic retries are on the roadmap, not yet available. The planned design retries failed deliveries (any non-2xx response, or a connection timeout) on an exponential backoff schedule with jitter:

After attempt 7 the delivery would be marked exhausted and moved to the planned dead-letter queue. The planned schedule spans roughly 35 hours, which covers most transient infrastructure incidents without holding open a job indefinitely.

Jitter is planned at ±15% of the computed delay. Without jitter, a wave of events that all failed at the same time (e.g. your endpoint was down for a deploy) would all retry in lockstep, creating a thundering-herd spike. Jitter spreads the load.

The planned design expects endpoints to respond within 10 seconds. Long-running processing should be offloaded asynchronously: receive the webhook, write the payload to a queue, return 200 immediately, process in the background.

Dead-Letter Queue (planned)

The dead-letter queue, its API, and the replay endpoints below are on the roadmap, not yet available — treat the paths and payloads here as the planned shape, not a guarantee. In the planned design, deliveries that exhaust all retry attempts are not dropped — they would be written to your dead-letter queue (DLQ), planned to be accessible via the API and a future dashboard:

# List DLQ entries
curl https://api.mailframe.ai/v1/webhooks/{endpoint_id}/dlq \
  -H "Authorization: Bearer $MAILFRAME_API_KEY"

# Replay a single entry
curl -X POST https://api.mailframe.ai/v1/webhooks/{endpoint_id}/dlq/{dlq_id}/replay \
  -H "Authorization: Bearer $MAILFRAME_API_KEY"

A planned DLQ entry would carry the full original payload, the delivery timeline, and the response received (or error) for each attempt:

{
  "dlq_id": "dlq_01jxb2p9k4m7q3n8",
  "delivery_id": "dlv_01jx9m4kz3p8q2n7",
  "event_id": "evt_01jx9m4kz3p8q2n7",
  "schema_id": "inv_schema_01jx...",
  "created_at": "2026-05-14T03:22:11Z",
  "exhausted_at": "2026-05-15T14:47:03Z",
  "attempts": [
    {
      "attempt": 1,
      "sent_at": "2026-05-14T03:22:11Z",
      "status": "timeout",
      "duration_ms": 10003
    },
    {
      "attempt": 2,
      "sent_at": "2026-05-14T03:22:41Z",
      "status": "error",
      "http_status": 503,
      "duration_ms": 412
    }
  ],
  "payload": {
    "job_id": "job_01jx9kz4m3p8q2n7",
    "confidence": 0.97,
    "data": { "...": "..." }
  }
}

Planned replays would go through the same delivery pipeline as fresh events, which means they would be signed with the current signing secret and carry a fresh X-MailFrame-Attempt counter starting from 1. Your idempotency check (on delivery_id) will short-circuit if the original event was already processed successfully despite the failed delivery acknowledgement.

Endpoint Health Monitoring (planned)

Endpoint health monitoring is a planned capability rather than a current guarantee — here is the behavior we are building toward. MailFrame will track a rolling success rate per endpoint. If an endpoint drops below 50% success over a 1-hour window, we plan to send an alert to the email address associated with the account. If it drops to zero over 15 minutes (your endpoint is completely unreachable), the intent is to pause automatic delivery to that endpoint and drain into the DLQ, rather than burning through all retry slots while you are mid-incident, with re-enable planned from a future dashboard once the endpoint recovers.

Checklist for a Production Webhook Endpoint

General receiver-side practice, plus the items to adopt when MailFrame’s planned webhook delivery launches:

• Verify the MailFrame-Signature header on every request before processing anything
• Parse from the raw body buffer, not a re-serialised JSON string
• Store X-MailFrame-Delivery-Id as a unique key; return 200 on duplicate receipt
• Return 200 within 10 seconds; offload processing to a queue
• Set up DLQ monitoring once the planned DLQ API is available — a webhook alert or a scheduled sweep
• Test your endpoint with a delivery test before going live

Webhooks done right are fire-and-forget from the sender’s side and reliably exactly-once from the receiver’s side. The primitives — signing, idempotency keys, backoff retries, dead-letter queues — are not novel, but every implementation detail matters. We are designing each of those details carefully so you will not have to reconstruct them from scratch.

If you want to discuss endpoint design for your integration, reach out during early access — we are actively supporting every team through their initial setup.