Event-Driven Architecture Basics: When to Use Queues vs Streams

A synchronous application often looks like:

Client
   ↓
API
   ↓
Service
   ↓
Database
   ↓
Response

Every step waits for the next one.

Event-driven systems introduce asynchronous communication:

Producer
   ↓
Broker
   ↓
Consumer

The producer can publish a message and continue without waiting for every downstream action to complete immediately.

This can improve:

decoupling

burst handling

background processing

independent scaling

integration flexibility

but it introduces new responsibilities:

duplicates

retries

ordering

eventual consistency

schema evolution

lag

observability

1. Understand the event-driven model first

Event-driven architecture overview (diagram)

Producer

A producer creates a message.

Examples:

API service

checkout service

scheduler

IoT device

database change publisher

Broker

The broker accepts and distributes messages.

Depending on the technology, it may provide:

durability

routing

retention

acknowledgements

partitions

consumer groups

retries

Consumer

A consumer processes the message.

For example:

send email

update search index

calculate analytics

resize image

charge payment

update projection

Asynchronous does not mean unreliable

A good architecture explicitly defines:

what happens if consumer crashes

what happens if broker is unavailable

how messages are retried

how duplicates are handled

how failed messages are inspected

Eventual consistency becomes normal

Suppose:

order created
     ↓
event published
     ↓
search index updated

there may be a brief period where:

database contains order

but

search index does not yet contain order

The application must decide whether that delay is acceptable.

2. Separate events from commands and jobs

Event

An event describes something that already happened.

OrderCreated

PaymentCaptured

UserRegistered

FileUploaded

It is usually written in the past tense because the producer is announcing a fact.

Command

A command asks something to happen.

SendWelcomeEmail

GenerateInvoice

ResizeImage

RecalculateReport

Job

A job is a concrete unit of background work.

resize image 123

generate report 456

send email 789

Why the distinction matters

An event:

OrderCreated

may interest:

email service

analytics service

inventory service

fraud service

while:

SendOrderConfirmationEmail

normally has one intended responsibility.

Avoid disguised remote procedure calls

If every message is:

ServiceA asks ServiceB
to execute exact internal operation

the system may still be tightly coupled even though a broker sits between the services.

3. Use queues for competing workers and units of work

A queue typically models:

one message
     ↓
one successful worker

Example

Image uploaded
     ↓
resize job queued
     ↓
Worker A
or
Worker B
or
Worker C

The workers compete for jobs.

Good queue workloads

email delivery

image processing

report generation

background imports

webhook delivery

asynchronous payment work

document conversion

Queue depth acts as a buffer

If producers generate:

10,000 jobs

faster than workers can process them, the queue can hold the backlog while workers continue at their available rate.

Queues decouple request latency from processing latency

Instead of:

HTTP request
     ↓
generate PDF for 40 seconds
     ↓
response

the application can:

HTTP request
     ↓
enqueue job
     ↓
return job ID

worker processes later

Acknowledgement completes the work item

Typically:

receive message

perform work

acknowledge success

If the worker crashes before acknowledgement, the message may become available again.

4. Use streams for retained event history and replay

A stream behaves more like:

append-only event log

Events remain available according to a retention policy.

Example

OrderCreated
PaymentCaptured
OrderPacked
OrderShipped

Consumers track position

Rather than deleting the event after one consumer reads it, a stream consumer typically advances an:

offset
or
position

Independent consumer groups

Order stream
   ├── Analytics group
   ├── Search group
   ├── Fraud group
   └── Notification group

Every group can process the same events independently.

Replay is a major capability

A new analytics consumer may begin from:

today

or:

three months ago

if retention allows it.

Good stream workloads

business event feeds

analytics pipelines

change propagation

search indexing

audit-style processing

materialized views

telemetry pipelines

5. Understand consumer groups and fan-out

Queues and streams delivery model (diagram)

Competing consumers

Queue:

Job 1 → Worker A

Job 2 → Worker B

Job 3 → Worker C

The workers share the workload.

Fan-out consumers

Stream:

Event 1
  ↓
Analytics
  ↓
Search
  ↓
Notifications

Each logical consumer group receives the event independently.

Streams can also scale inside a consumer group

Analytics consumer group

Consumer A
Consumer B
Consumer C

partitions are divided among consumers so the group can process in parallel.

Consumer groups provide independent progress

Search may be at:

offset 80,000

while analytics is at:

offset 78,000

without one blocking the other.

Do not accidentally create multiple groups when you want load sharing

If two workers belong to separate logical groups, both may process every event instead of splitting the work.

6. Treat ordering as a scoped guarantee

Global ordering is expensive and often unnecessary.

Typical requirement

Events for the same order
must be processed in order.

You may not need:

every order in the entire system
to share one global sequence

Partition by entity key

partition key:
orderId

can route:

OrderCreated

PaymentCaptured

OrderShipped

for the same order to the same partition.

Order is usually guaranteed inside a partition

Partition 1:
A1 → A2 → A3

Partition 2:
B1 → B2 → B3

but:

A2 vs B2

may have no meaningful global ordering.

Retries can complicate ordering

Suppose:

Event 10 fails

Event 11 succeeds

If Event 11 depends on Event 10, blindly continuing can violate business state.

Ask whether ordering is actually required

Some events can be designed to tolerate reordering using:

version numbers

timestamps

state checks

idempotent updates

7. Design for delivery semantics and duplicates

At-most-once

message delivered
zero or one times

Advantages:

no duplicate processing

tradeoff:

message may be lost

At-least-once

message delivered
one or more times

This reduces loss risk but means:

duplicates must be expected

Why duplicates happen

consumer performs database write
     ↓
consumer crashes
     ↓
acknowledgement never reaches broker
     ↓
message delivered again

Make consumers idempotent

Idempotent processing means:

same message processed twice

does not create
two business side effects

Use message IDs

eventId:
evt_123

A consumer may record:

evt_123 already processed

and skip duplicate effects.

Prefer naturally idempotent updates

Safer:

set order status = paid

than:

increment paid counter blindly

Be cautious with exactly-once claims

Exactly-once may be achievable inside carefully controlled broker or storage boundaries, but external side effects such as:

email

payment provider call

third-party HTTP request

still require explicit idempotency and recovery design.

8. Build retries, backoff, and dead-letter handling

Transient failure

dependency timeout

temporary 503

network interruption

retrying can help.

Permanent failure

invalid schema

deleted customer

unsupported state

malformed payload

retrying forever does not help.

Use bounded retries

attempt 1

wait

attempt 2

wait longer

attempt 3

quarantine

Use backoff

Instead of:

retry every 10 milliseconds

consider:

1 second

5 seconds

30 seconds

2 minutes

depending on the workload.

Add jitter where many consumers retry together

Randomized delay helps prevent:

10,000 failed jobs
retrying simultaneously

Dead-letter queue

After bounded failures:

message
   ↓
dead-letter queue

where operators can:

inspect

alert

fix

replay

discard deliberately

Do not make dead-letter storage a graveyard

Track:

dead-letter count

message age

failure reason

owner

recovery procedure

9. Decide whether replay is a requirement

Replay means processing historical events again.

Example

You create a new fraud model today but want to analyze:

the last 30 days
of payment events

A retained stream can support that naturally.

Rebuild derived state

event stream
     ↓
replay
     ↓
new search index

Recompute analytics

historical events
     ↓
new calculation logic
     ↓
new projection

Replay changes schema requirements

Consumers may encounter:

old event version

missing newer fields

legacy enum value

long after the producer has changed.

Replay also replays mistakes

If a consumer performs:

send email

replaying a year of historical events should not accidentally send:

a year of duplicate emails

Separate side effects from rebuildable projections

Replaying:

search index updates

is different from replaying:

customer notifications

10. Evolve event schemas carefully

Event data may survive longer than the code that produced it.

Initial event

{
  "eventType": "OrderCreated",
  "orderId": "ord_123",
  "total": 120
}

Additive evolution

{
  "eventType": "OrderCreated",
  "orderId": "ord_123",
  "total": 120,
  "currency": "EUR"
}

can be safe when:

currency is optional
for older consumers

Removing a field can break replay

A consumer rebuilding historical state may still need to understand older payloads.

Include event identity

eventId

eventType

occurredAt

schemaVersion

can make processing and diagnostics easier.

Include entity identity

orderId

userId

deviceId

helps with:

partitioning

deduplication

traceability

Avoid giant event envelopes

Publish the data consumers genuinely need rather than copying the entire internal database row automatically.

11. Keep database changes and event publication aligned

A classic problem:

save order
     ↓
publish OrderCreated

Failure case 1

database commit succeeds

broker publish fails

Result:

order exists

but event missing

Failure case 2

broker publish succeeds

database transaction fails

Result:

event says order exists

but order does not exist

Transactional outbox pattern

database transaction:

insert order

insert outbox event

commit together

Then a publisher reads the outbox and sends the event to the broker.

Outbox publisher

outbox row
     ↓
publish
     ↓
mark published

Duplicates are still possible

If publish succeeds but:

mark published

fails, the publisher may send the event again.

Consumers still benefit from:

idempotency

12. Design for backpressure, lag, and scaling

If producers generate messages faster than consumers process them:

backlog grows

Queue metric

queue depth

shows waiting work.

Stream metric

consumer lag

shows how far a consumer group is behind the newest event.

Message age matters

A queue containing:

100 messages

may be fine if the oldest is:

2 seconds old

but serious if the oldest is:

2 hours old

Scale consumers

1 worker
   ↓
4 workers
   ↓
8 workers

when work can be processed in parallel.

Streams are limited by partitions

If a consumer group has:

4 partitions

adding:

20 active consumers

does not necessarily produce 20-way parallel processing.

Watch for hot partitions

partition key:
tenantId

can produce imbalance if:

one tenant generates
80% of all traffic

Backpressure should influence producers when necessary

Unlimited message production can turn:

temporary slowdown

into:

storage exhaustion
or
multi-hour recovery backlog

13. Monitor asynchronous systems differently

A synchronous request exposes:

response time
status code

quickly.

An asynchronous workflow may fail:

30 seconds later

in another process

after several retries

Queue metrics

depth

oldest message age

processing rate

retry count

dead-letter count

Stream metrics

consumer lag

partition lag

processing throughput

offset progress

rebalance frequency

Consumer metrics

processing latency

success rate

failure rate

duplicate detection

dependency latency

Use correlation identifiers

requestId

traceId

eventId

orderId

help connect:

API request

database write

published event

consumer processing

downstream effect

Alert on user impact, not only broker health

A broker can be:

healthy

while:

email queue is 4 hours behind

14. Choose queues or streams from semantics

Queue vs stream decision tree (diagram)

Choose a queue when

message represents work

one worker should complete it

successful acknowledgement
finishes the message

replay is not a core requirement

Examples

send email

resize image

generate invoice PDF

process import

deliver webhook

Choose a stream when

message represents a retained event

multiple independent consumers
need the same event

consumer progress is independent

replay is valuable

event history matters

Examples

order event feed

analytics events

change data propagation

search indexing

telemetry

Use both when semantics differ

OrderCreated stream
     ↓
Notification service
     ↓
SendEmail queue
     ↓
email workers

The event remains replayable while the concrete email work is distributed through a queue.

Prefer the simpler model when requirements are modest

A small application with:

one producer

one background worker

no replay

no fan-out

may not need a complex streaming platform.

15. Copy/paste event-driven architecture checklist

Event-driven architecture checklist

Before introducing messaging
- What problem does asynchronous communication solve?
- Is synchronous request-response actually insufficient?
- Is burst buffering required?
- Is background processing required?
- Is independent scaling required?
- Is fan-out required?
- Is replay required?
- Is event history valuable?
- Is added operational complexity justified?

Message semantics
- Is this an event?
- Is this a command?
- Is this a job?
- Does it describe something that happened?
- Does it ask one component to perform work?
- Is there one intended consumer responsibility?
- Could several consumers legitimately react independently?

Events
- Use business-meaningful names.
- Prefer past-tense facts.
- Event occurred before publication.
- Event should not command hidden implementation behavior.
- Event identity included.
- Entity identity included.
- Occurrence time included where useful.
- Schema version included where useful.

Commands
- Intent is explicit.
- Target responsibility clear.
- Failure behavior clear.
- Retry behavior clear.
- Idempotency requirements clear.
- Avoid pretending commands are facts.

Jobs
- Work item explicit.
- Input bounded.
- Retry safe.
- Completion detectable.
- Result stored or observable.
- Timeout defined.
- Ownership defined.

Queue signals
- One work item should be completed by one worker.
- Competing consumers useful.
- Message can leave active queue after acknowledgement.
- Replay not a core requirement.
- Task backlog useful.
- Worker autoscaling useful.

Queue examples
- Send email.
- Resize image.
- Generate report.
- Convert document.
- Process import.
- Deliver webhook.
- Run background synchronization.
- Execute retryable task.

Stream signals
- Event history matters.
- Multiple independent consumer groups.
- Replay required.
- Retention required.
- Derived views.
- Analytics.
- Integration feed.
- Change propagation.
- Audit-style processing.

Stream examples
- Order events.
- Payment events.
- User activity.
- Telemetry.
- Change-data feed.
- Search-index feed.
- Analytics events.
- Domain-event integration.

Queue vs stream
- One worker handles work -> queue.
- Many independent consumers need same event -> stream.
- Replay required -> stream.
- Temporary work backlog -> queue.
- Historical log valuable -> stream.
- Concrete side-effect task -> often queue.
- Retained domain fact -> often stream.
- Use both when semantics differ.

Producer
- Message publication errors handled.
- Broker unavailable scenario understood.
- Timeout defined.
- Retry policy defined.
- Duplicate publication possible?
- Message ID generated.
- Correlation ID propagated.
- Payload validation performed.

Consumer
- Consumer can restart safely.
- Duplicate delivery safe.
- Timeout defined.
- External dependencies bounded.
- Failure categorized.
- Retry policy bounded.
- Dead-letter behavior defined.
- Metrics emitted.

Delivery semantics
- At-most-once acceptable?
- At-least-once required?
- Duplicate delivery expected?
- Exactly-once boundary clearly defined?
- External side effects protected?
- Acknowledgement timing understood?

At-most-once
- Loss acceptable?
- No retry needed?
- Duplicate avoidance more important than guaranteed processing?
- Appropriate only for non-critical workloads?

At-least-once
- Consumer idempotent.
- Duplicate message IDs handled.
- Side effects protected.
- Database writes deduplicated.
- Retries bounded.
- Acknowledgement after successful processing.

Idempotency
- Message has stable ID.
- Processed IDs stored if needed.
- Natural idempotent operation preferred.
- Duplicate payment prevented.
- Duplicate email considered.
- Duplicate webhook considered.
- Duplicate counters protected.
- Retry produces same safe outcome.

Acknowledgements
- Ack only after required work succeeds.
- Understand visibility timeout or lease.
- Understand redelivery behavior.
- Long-running job extends lease if needed.
- Consumer crash before ack tested.
- Consumer crash after side effect tested.

Retries
- Transient errors retry.
- Permanent errors do not retry forever.
- Attempt limit defined.
- Backoff defined.
- Jitter considered.
- Retry metrics tracked.
- Retry storm prevented.
- External rate limits respected.

Dead-letter handling
- Dead-letter destination exists.
- Reason recorded.
- Original message retained.
- Attempt count retained.
- Alert threshold defined.
- Owner defined.
- Inspection process defined.
- Replay process defined.
- Discard process defined.
- Dead-letter queue reviewed regularly.

Poison messages
- Invalid schema detected.
- Unsupported event version handled.
- Invalid business state handled.
- Infinite retry prevented.
- Quarantine supported.
- Root cause visible.

Ordering
- Is ordering required?
- Global ordering actually necessary?
- Per-entity ordering enough?
- Partition key selected.
- Same entity routed consistently.
- Retry behavior preserves required ordering.
- Out-of-order handling defined.
- Version check available where useful.

Partitioning
- Partition key stable.
- Distribution balanced.
- Hot-key risk evaluated.
- Ordering scope documented.
- Consumer parallelism tied to partitions.
- Repartitioning impact understood.
- Partition count sized with growth in mind.

Consumer groups
- One group per independent logical consumer.
- Workers inside group share partitions or work.
- Different groups intentionally receive same events.
- Group naming consistent.
- Ownership known.
- Lag monitored per group.

Fan-out
- Which consumers need same event?
- Can consumer be added without producer change?
- Does producer avoid knowledge of all downstream consumers?
- Failure of one consumer isolated from others?

Retention
- How long are events retained?
- Time-based retention?
- Size-based retention?
- Compliance restrictions?
- Storage cost?
- Replay horizon?
- Event deletion policy?

Replay
- Who can replay?
- From what offset or timestamp?
- Can replay trigger external side effects?
- Consumer can distinguish rebuild mode?
- Historical schemas supported?
- Replay tested.
- Replay rate limited if needed.
- Production load impact understood.

Schema
- Event envelope defined.
- Event type defined.
- Event ID defined.
- Entity ID defined.
- Timestamp defined.
- Schema version strategy.
- Required fields limited.
- Optional additive evolution preferred.
- Unknown fields tolerated where appropriate.

Schema evolution
- New optional fields safe.
- Old consumers tested.
- Historical events readable.
- Removed fields avoided.
- Type changes avoided.
- Enum growth handled.
- Version compatibility documented.
- Schema validation automated where useful.

Event payload
- Include data consumers need.
- Avoid copying entire database rows blindly.
- Avoid secrets.
- Avoid sensitive data unless necessary.
- Avoid huge payloads.
- Link to object storage for large binary data.
- Define source of truth.
- Define snapshot versus reference semantics.

Security
- Broker authentication.
- Consumer authentication.
- Producer authorization.
- Topic / queue permissions.
- Encryption in transit.
- Encryption at rest where needed.
- Secrets protected.
- Sensitive payloads minimized.
- Audit access to critical streams.

Privacy
- Personal data identified.
- Retention compatible with privacy requirements.
- Replay does not violate deletion requirements.
- Sensitive fields minimized.
- Access scoped.
- Dead-letter storage protected.

Transactional publication
- Database write and event publication can diverge?
- Outbox pattern considered.
- Distributed transaction avoided unless justified.
- Outbox publisher idempotent.
- Duplicate publication expected.
- Consumer deduplication preserved.

Transactional outbox
- Business row and outbox row in same transaction.
- Outbox publication retried.
- Publication status tracked.
- Duplicate send safe.
- Cleanup policy defined.
- Outbox growth monitored.

Inbox pattern
- Consumer stores received message ID.
- Business update and processed marker can be committed together.
- Duplicate deliveries detected.
- Inbox retention defined.
- Storage growth managed.

Eventual consistency
- User-visible delay acceptable?
- Maximum acceptable delay known?
- UI communicates pending state where needed?
- Read-after-write requirements identified?
- Compensating behavior available?

Backpressure
- Producer rate can exceed consumer rate?
- Queue or stream can absorb burst?
- Storage capacity known?
- Producer throttling possible?
- Consumer autoscaling possible?
- Backlog recovery rate known?
- Maximum acceptable lag known?

Queue depth
- Normal depth known.
- Warning threshold.
- Critical threshold.
- Oldest message age.
- Arrival rate.
- Completion rate.
- Retry rate.

Stream lag
- Lag per consumer group.
- Lag per partition.
- Oldest unprocessed event.
- Processing throughput.
- Catch-up time estimate.
- Alert threshold.
- Partition imbalance.

Scaling
- Consumer work parallelizable?
- Concurrency safe?
- Database can handle added workers?
- External APIs can handle added workers?
- Queue worker count scalable?
- Stream partition count supports parallelism?
- Autoscaling signal appropriate?

Hot partitions
- High-volume key identified.
- Tenant imbalance considered.
- Celebrity-object problem considered.
- Key strategy tested.
- Partition metrics visible.

Timeouts
- Message processing timeout.
- Dependency timeout.
- Visibility timeout.
- Long-running job strategy.
- Heartbeat if needed.
- Cancellation behavior.

Long-running jobs
- Progress stored.
- Job ownership durable.
- Lease renewal.
- Retry from safe checkpoint.
- Duplicate work safe.
- Maximum runtime.
- Cancellation.
- Result storage.

Observability
- Message ID logged.
- Trace ID propagated.
- Entity ID logged.
- Queue name or stream identified.
- Consumer group logged.
- Processing duration measured.
- Retries measured.
- Dead-letter count measured.
- Lag measured.

Tracing
- Producer span.
- Broker context propagated.
- Consumer span.
- Downstream calls linked.
- Async trace boundaries understood.
- Sampling suitable for high volume.

Alerts
- Oldest queue message too old.
- Consumer lag too high.
- Dead-letter spike.
- Processing failure rate.
- Broker unavailable.
- Partition imbalance.
- Retry storm.
- Publication failures.

Deployment
- New producer compatible with old consumers.
- New consumer compatible with old events.
- Rolling deployment safe.
- Schema rollout order defined.
- Feature flags considered.
- Replay after deployment safe.

Testing
- Happy-path processing.
- Duplicate delivery.
- Consumer crash before ack.
- Consumer crash after side effect.
- Broker unavailable.
- Dependency timeout.
- Retry exhaustion.
- Dead-letter handling.
- Out-of-order messages.
- Old schema.
- New optional fields.
- Replay.
- High backlog.

Local development
- Broker easy to start.
- Topics or queues provisioned.
- Sample messages available.
- Consumer logs understandable.
- Dead-letter inspection possible.
- Tests do not require fragile shared infrastructure where avoidable.

Operations
- Broker backups or durability understood.
- Availability model understood.
- Capacity monitored.
- Partition growth understood.
- Retention monitored.
- Upgrade process documented.
- Incident runbook available.
- On-call ownership clear.

Cost
- Broker compute.
- Message volume.
- Retained storage.
- Network transfer.
- Consumer compute.
- Cross-region traffic.
- Dead-letter storage.
- Replay cost.

Avoid event-driven architecture when
- Simple synchronous call is enough.
- There is only one small component.
- Eventual consistency is unacceptable.
- Operational maturity is insufficient.
- Messaging adds more complexity than value.
- There is no clear asynchronous requirement.

Common queue anti-patterns
- Infinite retry.
- No dead-letter handling.
- Acknowledge before side effect.
- No idempotency.
- One giant queue for unrelated workloads.
- No message age monitoring.
- Unbounded job duration.
- Hidden priority requirements.

Common stream anti-patterns
- Treating stream like temporary queue without reason.
- Replaying unsafe side effects.
- Assuming global ordering.
- Bad partition key.
- Too few partitions for required parallelism.
- Too many tiny consumer groups.
- No retention strategy.
- Breaking historical schemas.

Common event anti-patterns
- Event contains implementation command.
- Generic event named DataChanged.
- Huge payload.
- Sensitive information copied unnecessarily.
- No event ID.
- No entity ID.
- Producer expects exact downstream consumer behavior.
- Event schema changes destructively.

Event naming
- Use domain language.
- Prefer concrete business facts.
- OrderCreated.
- PaymentCaptured.
- UserRegistered.
- Avoid vague names.
- Avoid implementation-specific names where possible.

Queue naming
- Reflect work type.
- send-email.
- resize-image.
- generate-report.
- Avoid generic jobs queue when ownership differs.

Stream naming
- Reflect event domain.
- orders.
- payments.
- user-activity.
- telemetry.
- Keep naming conventions consistent.

Queue use case: email
- Application creates email job.
- Queue buffers jobs.
- Worker sends email.
- Temporary provider failure retries.
- Permanent failure dead-letters.
- Duplicate sending controlled.

Queue use case: image processing
- Upload stored.
- Resize job queued.
- Worker downloads source.
- Generates sizes.
- Stores outputs.
- Acknowledges only after success.

Stream use case: order events
- Order service publishes OrderCreated.
- Analytics consumes.
- Search consumes.
- Notification consumes.
- Each consumer tracks its own progress.
- Events retained for replay.

Stream use case: telemetry
- Devices publish readings.
- Stream partitions data.
- Real-time alert consumer.
- Analytics consumer.
- Archive consumer.
- Each runs independently.

Hybrid example
- OrderCreated published to stream.
- Notification consumer receives event.
- Creates SendEmail job.
- Email queue distributes work.
- Email workers retry independently.
- Business event remains replayable.

Final review
- Is this message work or a fact?
- Should one worker process it?
- Should several independent consumers process it?
- Must the message remain available after processing?
- Is replay required?
- Is retention required?
- Is ordering required?
- What is the ordering key?
- Are duplicates acceptable and handled?
- Is consumer idempotency implemented?
- Are retries bounded?
- Is backoff used?
- Is dead-letter handling operational?
- Are poison messages quarantined?
- Are schemas backward compatible?
- Can historical events still be processed?
- Is database-to-broker publication reliable?
- Is the outbox pattern needed?
- Is queue depth monitored?
- Is stream lag monitored?
- Is oldest-message age monitored?
- Can consumers scale safely?
- Can downstream dependencies handle that scale?
- Is eventual consistency acceptable?
- Are security and privacy requirements satisfied?
- Is replay safe?
- Is the added broker complexity justified?
- Would a simpler synchronous design work?
- Does the chosen queue or stream model match the actual message semantics?

16. FAQ

What is the simplest difference between a queue and a stream?

A queue usually represents units of work that should be processed by one worker from a competing set. A stream usually represents retained events that several independent consumer groups may read at their own pace.

When should I use a message queue?

Use a queue for background jobs such as sending email, resizing images, generating reports, processing imports, delivering webhooks, and other work where one successful worker should complete each item.

When should I use an event stream?

Use a stream when event retention, replay, multiple independent consumers, event history, analytics, change propagation, or rebuilding derived state are important requirements.

Why can the same event be delivered more than once?

A consumer can complete its side effect and then fail before the broker records the acknowledgement. The broker may therefore redeliver the message. This is why at-least-once systems should use idempotent consumers.

What is a dead-letter queue?

It is a destination for messages that cannot be processed successfully after the configured retry policy. It allows teams to inspect, alert on, fix, replay, or deliberately discard failed messages without blocking the main workload indefinitely.

Do event streams guarantee ordering?

Ordering is commonly scoped to a partition rather than an entire stream. Applications that need ordering for one entity should choose a partition key that keeps related events together and should still consider retries and out-of-order behavior.

Can one application use both queues and streams?

Yes. A common design is to publish durable domain events to a stream and let consumers create specific background jobs in queues. The stream preserves history and fan-out while queues distribute concrete work among workers.

Key terms (quick glossary)

Event-driven architecture

An architectural style where components communicate by producing and consuming asynchronous messages about events, commands, or work.

Producer

A component that creates and publishes a message to a broker, queue, or stream.

Consumer

A component that receives and processes messages from a broker.

Message broker

Infrastructure that receives, stores, routes, and delivers asynchronous messages between producers and consumers.

Queue

A messaging structure commonly used to distribute individual units of work among competing consumers.

Stream

A retained ordered sequence of events that consumers can process using independent positions or offsets.

Consumer group

A logical collection of consumers that share processing responsibility while maintaining one group-level position in a stream.

Offset

A position representing how far a consumer or consumer group has progressed through a stream.

Partition

A subdivision of a stream used to distribute storage and processing while commonly preserving order within that partition.

At-least-once delivery

A delivery model where a message is expected to be processed but may be delivered more than once.

At-most-once delivery

A delivery model where duplicate delivery is avoided but some messages may be lost.

Idempotency

The property that repeating the same operation does not create unintended additional effects.

Dead-letter queue

A destination used to isolate messages that repeatedly fail processing so they can be investigated or recovered separately.

Replay

Processing previously stored events again from an earlier stream position or point in time.

Consumer lag

The distance between the newest available stream data and the position a consumer group has processed.

Transactional outbox

A pattern that writes an outgoing event record in the same database transaction as the business state change and publishes it asynchronously afterward.

Backpressure

The condition where producers create work faster than consumers can process it, requiring buffering, throttling, or additional processing capacity.

Eventual consistency

A consistency model where different parts of a distributed system may temporarily observe different state but converge after asynchronous processing completes.