ADR-0023: Use AWS SQS and SNS for Asynchronous Inter-Service Communication
Status
Accepted
Date
2026-08-31
Context
Deal Engine is progressively moving functionality from the existing monolithic platform into independently deployable services.
As part of this migration, some workflows — such as refund processing, ticket operations, notifications, and external GDS interactions — do not require all processing to complete within a single synchronous request.
The platform currently relies heavily on synchronous service-to-service communication. This creates several problems:
- Cascade failures: if a downstream service or external GDS is unavailable, upstream services may also fail or remain blocked waiting for a response.
- Tight runtime coupling: the calling service requires the downstream service to be available at the same time.
- Latency amplification: workflows involving several synchronous calls accumulate latency across the complete request chain.
- External-system instability: GDS and airline APIs may timeout, respond slowly, or become temporarily unavailable.
- Retry complexity: retrying synchronous operations can create duplicate processing unless idempotency is carefully implemented.
- Independent scaling: some workloads, particularly refund and ticket-processing jobs, may need to scale independently from the services producing the work.
Deal Engine already operates its infrastructure primarily within AWS.
We therefore need a managed asynchronous messaging mechanism that integrates naturally with the existing AWS environment while introducing minimal additional operational complexity.
Not all communication should become asynchronous. Operations requiring an immediate response will continue to use synchronous APIs.
Decision
We will use Amazon SQS as the default mechanism for asynchronous work distribution between services.
Amazon SNS will be used where a single business event needs to be delivered to multiple independent consumers.
The general distinction will be:
SQS
────
"Someone needs to process this work."
Producer
│
▼
SQS Queue
│
▼
Consumer
SNS + SQS
─────────
"Something happened and several systems
may independently care about it."
Producer
│
▼
SNS Topic
│
├──────────────┐
▼ ▼
SQS Queue A SQS Queue B
│ │
▼ ▼
Consumer A Consumer B
Synchronous HTTP APIs will remain appropriate when the caller requires an immediate result.
For example:
Authentication request
Service
│
│ HTTP
▼
Auth Service
│
▼
Immediate response
while long-running refund processing could use:
Refund API
│
│ create refund
▼
Refund Service
│
│ persist
▼
Refund DB
│
│ enqueue work
▼
SQS Refund Queue
│
▼
Refund Worker
│
▼
GDS / Airline API
The API may therefore return:
202 Accepted
refundId = REF-123
status = PENDING
rather than keeping the original request open while the complete external refund operation is performed.
Message Processing Semantics
SQS Standard queues provide at-least-once delivery.
Consumers must therefore assume that the same message can be delivered more than once.
For example:
SQS
│
│ RefundRequested(REF-123)
▼
Worker
│
│ calls GDS
▼
GDS
│
│ REFUND SUCCESSFUL
▼
Worker
│
💥 crashes before
deleting message
SQS may later redeliver the message:
SQS
│
│ RefundRequested(REF-123)
▼
Worker
The consumer must not issue a second refund simply because the message was delivered again.
All consumers processing business-critical operations must therefore be idempotent.
A stable business identifier such as refundId will be used to detect previously processed operations.
Retry Strategy
Transient failures will be retried through SQS.
The SQS visibility timeout will prevent another worker from immediately processing a message while it is already being handled.
If processing succeeds, the consumer deletes the message.
If processing fails, the message becomes visible again after the visibility timeout.
Conceptually:
SQS
│
▼
Worker
│
┌─────┴─────┐
│ │
SUCCESS FAILURE
│ │
▼ ▼
Delete message Message becomes
visible again
│
▼
Retry
Retry policies must use bounded retries and backoff where appropriate.
Messages that repeatedly fail will be moved to a Dead-Letter Queue (DLQ).
SQS Queue
│
│ retry
▼
Consumer
│
│ retry
▼
Consumer
│
│ maxReceiveCount exceeded
▼
Dead-Letter Queue
DLQs will be monitored and must have an operational recovery procedure.
External-System Timeouts
A timeout when communicating with a GDS or airline system must not automatically be interpreted as a failed business operation.
For example:
Refund Worker
│
│ refund REF-123
▼
GDS
│
│ refund succeeds
│
X response lost
Worker sees:
TIMEOUT
The actual business state may now be:
Deal Engine → UNKNOWN
GDS → REFUNDED
Blindly retrying the refund could therefore create a duplicate operation.
For operations where the external system supports status queries or stable external references, the service will reconcile the operation before retrying.
TIMEOUT
│
▼
UNKNOWN
│
▼
Query GDS using
stable reference
│
├──────────────┐
▼ ▼
REFUNDED NOT FOUND
│ │
▼ ▼
SUCCEEDED Retry
Idempotency and reconciliation requirements will be documented separately for each business-critical workflow.
SNS Usage
SNS will not automatically be placed in front of every SQS queue.
For point-to-point asynchronous work:
Refund Service
│
▼
SQS
│
▼
Refund Worker
is sufficient.
SNS will be introduced when an event has multiple independent consumers.
For example:
RefundCompleted
│
▼
SNS
/ | \
/ | \
▼ ▼ ▼
SQS Audit SQS Email SQS Analytics
│ │ │
▼ ▼ ▼
Audit Notify Analytics
Service Service Service
This prevents the producer from needing to know which systems consume the event.
Each consumer receives its own SQS queue so that:
- consumers process events independently;
- one slow consumer does not block another;
- each consumer can define its own retry policy;
- each consumer can have its own DLQ;
- services can scale independently.
Alternatives Considered
Apache Kafka
Pros
- Excellent support for high-throughput event streaming.
- Strong event replay capabilities.
- Consumer groups allow multiple processing models.
- Long-lived event logs are useful for audit and event-driven architectures.
- Strong ecosystem for stream processing.
Cons
- Introduces additional concepts and operational complexity.
- Requires Kafka-specific monitoring, partition management, consumer offset management, and schema governance.
- A managed Kafka platform would introduce another significant infrastructure dependency.
- Kafka’s strengths are not required for the initial asynchronous workloads identified.
Why rejected
Deal Engine already relies heavily on AWS, and the immediate requirement is reliable asynchronous job and event processing rather than large-scale stream processing.
SQS and SNS provide the required capabilities while integrating directly with the existing AWS environment and reducing operational overhead.
Kafka may be reconsidered if future requirements include:
- high-volume event streaming;
- long-term event replay;
- event sourcing;
- stream processing;
- multiple consumers independently replaying historical events.
RabbitMQ
Pros
- Mature messaging system.
- Flexible routing capabilities.
- Supports queues and publish/subscribe patterns.
- Familiar messaging semantics.
Cons
- Introduces additional infrastructure to operate or another managed service dependency.
- Does not provide a significant advantage over AWS-native messaging for the identified use cases.
- Requires additional operational knowledge and monitoring.
Why rejected
The platform is already AWS-centric. SQS and SNS provide the required messaging capabilities without introducing another messaging technology.
Continue Using Synchronous HTTP
Pros
- Simple programming model.
- Existing team knowledge.
- Immediate responses.
- Straightforward debugging for simple request/response interactions.
Cons
- Services remain runtime-coupled.
- Downstream failures can propagate upstream.
- Long-running GDS operations occupy request resources.
- Retry handling becomes difficult.
- Scaling producers and consumers independently is harder.
Why rejected as the default for asynchronous workflows
Synchronous communication remains appropriate where an immediate result is required, but it should not be required for long-running or independently recoverable workflows.
The architecture will therefore intentionally support both:
Need immediate answer?
│
┌───┴───┐
YES NO
│ │
HTTP SQS
│
│
Multiple consumers?
┌──┴──┐
NO YES
│ │
SQS SNS
│
SQS queues
Consequences
Positive
- Reduced runtime coupling between services.
- Downstream outages do not necessarily cause upstream failures.
- Work can remain queued while consumers are temporarily unavailable.
- Consumers can scale independently.
- AWS manages the messaging infrastructure.
- Native integration with existing AWS infrastructure and monitoring.
- Built-in retry and dead-letter queue capabilities.
- SNS enables event fan-out without coupling producers to consumers.
- Supports gradual extraction of capabilities from the monolith.
Negative
- Asynchronous workflows introduce eventual consistency.
- Message processing becomes more difficult to trace than synchronous calls.
- Consumers must be idempotent.
- Duplicate message delivery must be expected.
- Business workflows require explicit state management.
- Engineers must understand visibility timeout, retention, retry, DLQ, and delivery semantics.
- Debugging requires correlation IDs and distributed observability.
Risks
Duplicate Processing
SQS Standard provides at-least-once delivery.
Mitigation: all business-critical consumers must implement idempotent processing using stable business identifiers.
Poison Messages
A malformed or permanently failing message could otherwise be retried repeatedly.
Mitigation: configure bounded retries and DLQs.
Lost Database-to-Queue Events
A service could persist a business operation and crash before publishing the corresponding SQS message:
BEGIN TRANSACTION
INSERT refund
COMMIT
💥 crash
Send SQS message ← never happens
The database and message broker do not participate in the same transaction.
Mitigation: use the Transactional Outbox Pattern for workflows where loss of the message would violate business guarantees.
SAME DB TRANSACTION
┌──────────────────────────────┐
│ │
│ INSERT refund │
│ │
│ INSERT outbox_event │
│ │
└──────────────┬───────────────┘
│ COMMIT
▼
Outbox Publisher
│
▼
SQS
External Operation Succeeds but Response Is Lost
Blind retrying could create duplicate external operations.
Mitigation: use stable external references, explicit UNKNOWN states, provider-side status queries where available, and reconciliation before retrying uncertain operations.
Poor Observability
Asynchronous processing makes request chains less obvious.
Mitigation: propagate correlation IDs and trace context through messages and instrument producers and consumers using the platform’s standard observability tooling.
Operational Requirements
Each production queue must define:
- queue owner;
- expected processing latency;
- visibility timeout;
- message retention period;
- retry policy;
- maximum receive count;
- DLQ;
- DLQ alarms;
- consumer concurrency;
- idempotency strategy;
- correlation/trace identifiers;
- recovery procedure.
CloudWatch alarms should detect conditions such as:
ApproximateAgeOfOldestMessage ↑
│
▼
Consumers may not be keeping up
DLQ Message Count > 0
│
▼
Processing requires investigation
Migration Strategy
SQS/SNS adoption will be incremental.
Existing synchronous workflows will not be converted solely for architectural consistency.
A workflow should move to asynchronous processing when there is a concrete benefit such as:
- long-running processing;
- external-system latency;
- retry requirements;
- temporary downstream unavailability;
- independent scaling;
- fan-out to multiple consumers;
- eventual consistency being acceptable.
For the monolith-to-services migration, a typical extraction can therefore evolve as:
PHASE 1
Monolith
│
│ HTTP
▼
Extracted Service
PHASE 2
Monolith
│
▼
SQS
│
▼
Extracted Service
PHASE 3 — when fan-out is required
Extracted Service
│
▼
SNS
/ \
▼ ▼
SQS SQS
│ │
▼ ▼
Service Service
A B
Success Criteria
The decision will be considered successful when:
- asynchronous workflows continue processing after temporary consumer outages;
- duplicate message delivery does not produce duplicate business operations;
- failed messages are recoverable through DLQs;
- queue backlogs are observable and alertable;
- services can scale independently;
- GDS/external-system failures do not unnecessarily propagate to upstream services;
- new services can be extracted from the monolith without introducing a new messaging platform for each workflow.
Related Decisions
- ADR-0024: Idempotency Strategy for Asynchronous Consumers
- ADR-0025: Transactional Outbox for Reliable Event Publication
- ADR-0026: Retry and Dead-Letter Queue Strategy
- ADR-0027: Distributed Tracing for Asynchronous Workflows
- ADR-0028: GDS Timeout and Reconciliation Strategy