Webhooks are the duct tape of modern integrations — cheap, fast, and everywhere. They're also the first thing that breaks when a downstream service hiccups, a deploy goes sideways, or a client's firewall silently drops POST requests at 2 a.m. Most teams discover this the hard way: a payment processed twice, an appointment never created, a chart update that vanished into the void.
The good news is that webhook reliability is a solved problem. The patterns aren't exotic. They just require discipline to implement up front rather than bolted on after the first incident.
Why Webhooks Fail in Ways You Won't Immediately SeeHTTP is stateless and fire-and-forget by default. When you POST an event to a consumer endpoint, you get a response code back — or you don't. A 200 OK means the endpoint received the request, not that it processed it correctly. A timeout means nothing conclusive. A 500 means something broke, but not what or whether it's recoverable.
The failure modes compound: transient network errors, cold-start latency on serverless receivers, database deadlocks mid-handler, deployments that briefly take endpoints offline. Any of these can cause an event to be lost or duplicated depending on how your sender handles them.
Idempotency Keys: The Non-Negotiable FoundationEvery webhook payload you emit should carry a stable, unique idempotency key — typically a UUID tied to the event, not the delivery attempt. The receiver stores processed keys in a fast lookup (Redis works well; a dedupe table in Postgres works fine at lower volume) and short-circuits duplicate deliveries before they touch business logic.
The key insight: idempotency is the receiver's responsibility, but the sender must provide a stable identifier. If your sender generates a new ID on every retry, you've already lost. Tie the key to the source event: payment.completed:{payment_id} or appointment.updated:{appointment_id}:{updated_at_unix}.
Retention window matters too. Keep processed keys for at least as long as your maximum retry window — typically 72 hours covers most real-world scenarios.
Retry Logic: Exponential Backoff With JitterRetrying immediately after a failure is almost always wrong. If the consumer is overloaded or mid-deploy, hammering it compounds the problem. The standard pattern:
Add jitter — a random offset of ±20% — to each interval. Without it, a batch of events that all fail simultaneously will retry in synchronized waves and create artificial traffic spikes. Jitter spreads the load.
Define what counts as a retryable failure. 5xx responses and timeouts are retryable. 4xx responses (except 429) are not — a 400 or 404 means the payload is malformed or the resource doesn't exist; retrying won't help. Route non-retryable failures directly to your dead-letter queue rather than burning retry attempts.
Dead-Letter Queues: Failure Is a First-Class CitizenA dead-letter queue (DLQ) is where events go after exhausting retries. It is not a graveyard — it's an audit trail and a recovery mechanism. Every event in your DLQ should be replayable once the underlying issue is resolved.
Operationally, your DLQ needs three things: the original payload, the delivery attempt history (timestamps, response codes, error messages), and a manual or automated replay trigger. Without the attempt history, you're debugging blind.
In practice, this often means a simple webhook_events table with status transitions: pending → delivered | failed → dead_lettered. Add an index on (status, next_attempt_at) and a background worker polling for due retries. You don't need Kafka for this unless you're processing millions of events per hour.
Reliability isn't just about delivery — it's about trust. Any endpoint that accepts webhooks should verify the sender's HMAC signature before processing. Stripe, GitHub, and most mature platforms include a signature header. Compute the expected HMAC over the raw request body using your shared secret, compare in constant time, and reject anything that doesn't match.
This prevents replay attacks and spoofed payloads. It also means your idempotency store isn't polluted with garbage from scanners probing your endpoint.
Practical TakeawayBefore your next webhook integration goes to production, confirm four things: every outbound event has a stable idempotency key; your retry schedule uses exponential backoff with jitter; failed events land in a queryable dead-letter queue with full attempt history; and your receiver verifies signatures before touching business logic. These four controls eliminate the vast majority of webhook-related incidents — and they take a day to implement correctly, not a week.