← All articles
Engineering · 6 min read

Event-Driven Data Pipelines: Kafka vs. Postgres LISTEN/NOTIFY

By Romanov Solutions · June 4, 2026

Every team building a real-time feature eventually hits the same fork in the road: stand up a Kafka cluster, or lean on the Postgres instance you already have? Both can move events. Both have failure modes. The wrong choice costs you six months of operational regret.

What Postgres LISTEN/NOTIFY Actually Does

Postgres ships a lightweight pub/sub mechanism baked into the database engine. A producer calls NOTIFY channel, payload; any connected session that has issued LISTEN channel receives the payload asynchronously. No broker, no ZooKeeper, no separate deployment — just your existing Postgres connection pool.

The ceiling is real, though. Notifications are not persisted. If your consumer is offline when the NOTIFY fires, the event is gone. There is no replay, no consumer group offset, no dead-letter queue. Payload size is capped at 8 KB. And because delivery rides on open connections, you are effectively limited to the number of backends Postgres can sustain — typically a few hundred before connection overhead becomes a bottleneck.

Where it shines: internal tooling, cache invalidation signals, low-volume workflow triggers, and any pattern where losing an occasional event is acceptable or where the source-of-truth row is always queryable as a fallback. A dental practice management integration that needs to push appointment-status changes to a waiting-room display? LISTEN/NOTIFY is more than sufficient and adds zero infrastructure.

What Kafka Actually Does

Kafka is a distributed, durable, ordered log. Producers append records to partitioned topics; consumers read at their own pace and commit offsets. Records are retained on disk for a configurable window — days, weeks, or indefinitely. Multiple independent consumer groups can replay the same topic from the beginning without affecting each other.

That durability and replay capability is the entire value proposition. A billing pipeline that must guarantee every charge event is processed exactly once, an audit log that regulators can subpoena, a CDC stream feeding three downstream services simultaneously — these are Kafka's native habitat.

The cost is real too. A production-grade Kafka deployment (or a managed equivalent like Confluent Cloud or Amazon MSK) adds schema registry overhead, monitoring surface area, consumer lag alerting, and partition-rebalance incidents to your ops calendar. A three-broker cluster on MSK runs roughly $300–$500/month before egress. That is not expensive at scale; it is expensive when your throughput is 200 events per day.

The Decision Matrix The Transactional Outbox: The Honest Middle Ground

If you are not ready for Kafka but cannot accept lost events, the outbox pattern is the pragmatic answer. Write events to an outbox table inside the same database transaction as your domain mutation. A separate poller — or a tool like Debezium reading the Postgres WAL — picks up unprocessed rows and forwards them downstream, then marks them delivered. You get at-least-once delivery, a full audit trail, and zero dependency on a broker being available at write time.

Debezium deserves a callout here: it tails the WAL and emits change events to Kafka topics without any application-level code changes. It is how teams retrofit event-driven behavior onto legacy schemas without touching the application. We have used this pattern on modernization engagements where the source system was a 15-year-old monolith — WAL-based CDC let us stream changes to a new microservice without a single stored procedure modification.

Latency Characteristics in Practice

Postgres LISTEN/NOTIFY delivers in under 5 ms on a local network — faster than most Kafka configurations because there is no broker hop. Kafka's end-to-end latency with default settings sits around 5–15 ms, tunable down to under 5 ms with aggressive linger.ms and acks=1 settings, but at the cost of durability guarantees. For most business applications, both are effectively real-time. Latency is rarely the deciding factor; durability and fan-out are.

Practical Takeaway

Default to Postgres LISTEN/NOTIFY for anything internal and low-volume — you will ship faster and operate less. Reach for Kafka when you need durable replay, multiple independent consumers, or throughput that strains a single Postgres backend. If you are unsure, implement the transactional outbox pattern first: it keeps your options open and gives you a migration path to Kafka without a rewrite. Pick the boring tool that fits the load, not the impressive tool that fits the conference talk.

event-driven architectureapache kafkapostgres listen notifydata pipelinesreal-time data processingbackend engineeringdatabase change events
Was this useful?
Ask AI