Requirements
- Channels: email, SMS, push, in-app, webhooks. Triggers come from domain events or scheduled jobs.
- Respect user preferences and quiet hours. Templates and localisation. At-least-once delivery without spamming duplicates.
Design
diagram
Key decisions
- Queue per channel so a slow SMS provider doesn't block email (a bulkhead).
- Dedupe key =
eventId + userId + channel, checked before sending. - Retries with backoff: for webhooks, retry for hours to days with exponential backoff, then disable the endpoint and notify the owner.
- Sign webhooks (HMAC with a timestamp) so receivers can verify them and reject replays.
- Provider failover behind a circuit breaker for critical channels (OTP SMS).
- Batching and digests to avoid a flood when 10k entries publish in one release.
Cheatsheet
The whole topic on one page. Click to open full screen.
Sources & further learning
Videos, courses, docs and books I recommend for this topic.
Related topics
Architecture PatternsApplied in production
Event-Driven Architecture
Services announce facts ("EntryPublished") and others react asynchronously — decoupling producers from consumers in time, space and knowledge.
Intermediate5 sources
Design PrinciplesApplied in production
Idempotency
Doing an operation twice has the same effect as doing it once. The foundation of safe retries, at-least-once messaging and reliable APIs.
Intermediate4 sources
Distributed Systems & DataUnderstood
Message Queues vs Event Streams
RabbitMQ/SQS-style queues distribute work; Kafka-style logs retain ordered events for many consumers and replay. Know which one your problem needs.
Intermediate5 sources
Architecture PatternsUnderstood
Circuit Breaker, Retry, Timeout & Bulkhead
The resilience toolkit — fail fast, retry safely, bound waiting, and isolate resources so one slow dependency can't take down the whole system.
Intermediate6 sources