ADR-0001: Use Amazon SNS + SQS for Inter-Service Communication (Claude)

Status

Accepted

Date

2025-11-15

Context

Our platform has grown from 3 microservices to 12 over the past year. Services currently communicate synchronously via HTTP REST calls. This creates several problems:

Cascade failures: when the inventory service is down, the order service can’t process orders, even though inventory checks could be eventual.
Tight coupling: services need to know each other’s API contracts and endpoints.
Performance bottlenecks: some operations trigger chains of 4-5 synchronous calls, adding latency.

We process approximately 100K events per day. We expect this to grow to 500K within 12 months. That is roughly 1-6 messages per second on average, with bursts well inside the limits of any managed broker.

Deal Engine already runs almost entirely on AWS: compute, storage, networking, IAM, and observability are all AWS-native, and our infrastructure is defined in Terraform against AWS providers. The team has 8 backend engineers. All 8 work with AWS services daily; 2 have prior Kafka experience.

Decision

We will adopt Amazon SNS + SQS as the primary mechanism for asynchronous inter-service communication, using the standard fan-out topology:

Producers publish domain events to an SNS topic per event family (e.g. orders, inventory, payments).
Each consuming service owns a dedicated SQS queue subscribed to the relevant topic(s), with a filter policy so it only receives the message types it cares about.
Every queue gets a dead-letter queue with a redrive policy (default maxReceiveCount: 5).

Where ordering matters — financial events in particular — we will use FIFO topics and FIFO queues, with the message group ID set to the entity being mutated (e.g. deal_id, account_id). This gives ordering per entity plus deduplication, while still allowing parallel processing across entities.

Synchronous HTTP remains for request/response patterns where the caller needs an immediate result (e.g. authentication checks).

Topics, queues, subscriptions, filter policies, and DLQs are all defined in Terraform alongside the rest of our infrastructure. Access is granted through IAM roles, not shared credentials.

Alternatives Considered

Apache Kafka (Confluent Cloud or MSK)

Pros: Log-based retention makes event replay and rewind trivial. Strong ecosystem for stream processing (Kafka Streams, ksqlDB). Partition-level ordering with high throughput. Vendor-neutral in principle.
Cons: Substantial operational and conceptual surface area — brokers or a managed vendor, schema registry, consumer groups, partition and rebalance tuning, offset management. Only 2 of 8 engineers have worked with it. Introduces a second infrastructure vendor with its own billing, networking (PrivateLink/VPC peering), IAM model, and Terraform provider, none of which composes with our existing AWS setup. Baseline cost is a fixed monthly cluster charge that dwarfs our actual usage at 100-500K events/day.
Why rejected: We would be paying the full price of Kafka’s complexity to serve a workload two to three orders of magnitude below where that complexity starts to pay for itself. “Vendor neutrality” is largely theoretical for us — we are already on AWS for everything else, so adding Confluent does not reduce lock-in, it adds a second lock-in.

RabbitMQ

Pros: Simpler than Kafka, flexible routing (topic, direct, fanout exchanges), mature and well understood.
Cons: We would either self-host (brokers, clustering, quorum queues, upgrades, on-call) or run Amazon MQ, which costs more than SNS+SQS for the same job. Weak replay. No advantage over SQS for our access patterns.
Why rejected: All of the operational burden of running a broker, none of the benefits over a fully managed AWS-native option.

Keep Synchronous HTTP (with circuit breakers)

Pros: No new infrastructure. Team already familiar. Circuit breakers address cascade failures.
Cons: Doesn’t solve tight coupling. Latency still accumulates across call chains. Circuit breakers are a band-aid, not a solution to the fundamental coupling problem.
Why rejected: Addresses symptoms, not root cause.

EventBridge

Pros: Also AWS-native. Richer content-based routing than SNS filter policies, schema registry, built-in archive and replay, and native third-party SaaS integrations.
Cons: Higher per-event cost, no FIFO ordering guarantees, and at-least-once delivery with weaker throughput ceilings for high-fan-in workloads.
Why rejected (for now): Not rejected outright. SNS+SQS covers our service-to-service backbone more cheaply and with the ordering guarantees we need. We expect to use EventBridge alongside it for cross-account, SaaS, and scheduled/rule-driven integrations, and potentially for the event archive (see Risks).

Consequences

Positive

Services become decoupled: producers publish to a topic and don’t need to know who consumes it. New consumers are added by creating a queue and a subscription, with no producer change.
Cascade failures eliminated for async workflows. A down consumer means a growing queue, not a failed order.
No infrastructure to operate: no brokers, no clusters, no patching, no capacity planning, no 3am rebalance debugging.
Native fit with what we already run: IAM for authz, CloudWatch for metrics and alarms, X-Ray for tracing, Terraform for provisioning, Lambda and ECS for consumers with SQS-driven autoscaling.
Dead-letter queues and redrive are built in. Poison messages are isolated and replayable without custom tooling.
Costs scale to zero and stay small. At 500K events/day (~15M/month) fanned out to a handful of consumers, this is on the order of tens of dollars a month, versus a fixed several-hundred-dollar floor for a managed Kafka cluster. (Approximate — SQS ~$0.40/M requests standard, ~$0.50/M FIFO; SNS ~$0.50/M publishes with SQS deliveries free. Verify current pricing for our region before budgeting.)
Every engineer on the team can already reason about it.

Negative

No event log, no free replay. This is the real trade-off against Kafka. SQS is a queue, not a log: once a message is consumed and deleted it’s gone, and maximum retention is 14 days. Replaying last quarter’s events is not something the transport gives us. See Risks for mitigation.
No built-in stream processing. There is no equivalent of Kafka Streams. Aggregations, joins, and windowing must be done in consumer code, or by moving that specific workload to Kinesis Data Streams / Firehose if one ever justifies it.
Full event-sourcing patterns are off the table with this transport alone. If a future service genuinely needs an immutable ordered log as its source of truth, it should use DynamoDB Streams, Kinesis, or a dedicated event store — and that will need its own ADR rather than being bolted onto SQS.
Deeper AWS lock-in. We accept this consciously. Mitigation: keep a thin internal publish/subscribe interface in our shared library so that SNS/SQS SDK calls are not scattered through domain code.
256 KB message size limit. Large payloads need the S3 extended client (claim check pattern) or, better, slimmer events that carry IDs rather than full entity snapshots.
Eventual consistency replaces strong consistency for async flows. Some workflows need redesign.
Delivery is at-least-once on standard queues, so consumers must be idempotent. FIFO queues offer deduplication within a 5-minute window, which helps but is not a substitute for idempotent handlers.
Debugging distributed async flows is harder than tracing synchronous HTTP calls. We’ll need distributed tracing (see ADR-0024).

Risks

Loss of audit/debug replay. Mitigation: subscribe an archival consumer to each topic that writes raw events to S3 (via Firehose or a small Lambda), partitioned by date. This gives us durable, queryable history through Athena at negligible cost, and a path to re-publish if needed. This should be set up as part of the initial rollout, not deferred.
FIFO throughput ceilings. Default FIFO queues handle 300 messages/sec (3,000 with batching); high-throughput mode raises this substantially. Our projected peak is far below the default, but message group ID choice matters — grouping too coarsely (e.g. one group for all events) serialises everything. Group by entity.
SNS FIFO topics can only deliver to SQS FIFO queues. Any consumer of an ordered topic must use a FIFO queue; Lambda and HTTP subscriptions won’t work directly on those topics.
Filter policy sprawl. As topics grow, subscription filters can become an undocumented routing layer. Keep them in Terraform, keep them simple, and prefer more topics over more elaborate filters.
Schema evolution across services still requires discipline — we lose Confluent’s schema registry. Plan: versioned event contracts in a shared package, additive-only changes, consumers ignore unknown fields. Consider EventBridge Schema Registry if this becomes painful.

ADR-0024: Adopt OpenTelemetry for Distributed Tracing
ADR-0018: Service Communication Contracts (superseded by this ADR)