Pub/Sub, From First Principles: Part 4 — Production Resilience and Guarantees

Part 4 of 4 in the Pub/Sub, From First Principles series.

Everything so far assumes messages always arrive and subscribers always succeed — and, just as importantly, that everything lives in one process's memory. Real systems cross that boundary: publishers and subscribers run on different machines, connected by a network that drops packets, and processes that crash and restart. That single step — leaving one memory space — is what creates a whole layer of concerns the in-memory version never had to face: persistence (a message must survive a crash, so it's written to durable storage before it counts as published), acknowledgements (a broker can't know a subscriber finished unless the subscriber explicitly says so), replication (the broker itself can't be a single point of failure), and network failures (any message, or any acknowledgement, can simply not arrive). None of these need solving here in depth — but every concept below exists because of them. These four — delivery guarantees, idempotency and retries, dead-letter queues, and backpressure — are also the vocabulary of nearly every serious conversation about event-driven systems, whether that's a design review or a systems interview.

Delivery guarantees

There are three worth knowing: at-most-once (a message is delivered zero or one times — it can be lost, never duplicated), at-least-once (delivered one or more times — never silently lost, but sometimes processed twice), and exactly-once (delivered precisely once — the one that's genuinely hard to guarantee across a network, and not something retries alone get you to). Most systems, including Kafka and SQS by default, settle for at-least-once: the broker redelivers until it receives an acknowledgement, and a lost acknowledgement looks identical to a lost message — so duplicates aren't a rare accident, they're the mechanism working as designed.

Idempotency

At-least-once delivery pushes the duplicate problem onto the consumer, which means idempotency isn't an optimization — it's the other half of the delivery guarantee. A consumer is idempotent when processing the same message a second time produces the same outcome as the first: a safe no-op, not a duplicate charge or a duplicate row. The standard mechanism is a unique ID per logical event and an atomic "claim" against a ledger of processed IDs (a database unique constraint doing the check-and-record in one step) — checked before the real work runs, so a redelivered message gets recognized and skipped no matter which consumer instance it lands on.

Fault tolerance

When a subscriber fails, retrying immediately is tempting and usually wrong — if the failure was caused by an overloaded downstream dependency, hammering it with instant retries just adds to the overload. The standard fix is exponential backoff with jitter: each retry waits roughly twice as long as the last, with some randomness added so a batch of consumers that failed at the same moment don't all retry in lockstep and recreate the exact spike they're recovering from.

Error handling

A message can't retry forever. After some capped number of attempts, it should stop competing for the consumer's attention and move to a dead-letter queue (DLQ) instead — a place to inspect what failed and why, rather than a place where messages just quietly vanish. A DLQ doesn't fix anything by itself; its value is observability (a growing DLQ is a leading indicator of a bug worth alerting on) and recoverability (once the underlying issue is fixed, those messages can be replayed back through the system deliberately).

Flow control

A queue doesn't create capacity — it just delays the moment you run out of it. If subscribers can sustainably handle less than what publishers are producing, an unbounded queue just grows until something else breaks. Backpressure means giving each queue a real limit and deciding, deliberately, what happens once it's hit: block the publisher until there's room (no data loss, but a slow subscriber now stalls everyone upstream of it), drop the newest or oldest message (bounded memory, but silent, potentially costly data loss), or reject the new message with an error (no silent loss, but publishing is no longer fire-and-forget — something upstream now has to handle the rejection). For anything where losing or duplicating an event has a real cost, rejecting loudly is usually the safer default than failing silently.

Put together, these four are what separate a pub/sub implementation that works in a demo from one that survives contact with real traffic: it degrades predictably under failure, instead of losing data, duplicating it, or falling over.