ADR-0023: Use Apache Kafka for Inter-Service Communication

Status

Accepted

Date

2025-11-15

Context

Our platform has grown from 3 microservices to 12 over the past year. Services currently communicate synchronously via HTTP REST calls. This creates several problems:

Cascade failures: when the inventory service is down, the order service can’t process orders, even though inventory checks could be eventual.
Tight coupling: services need to know each other’s API contracts and endpoints.
Performance bottlenecks: some operations trigger chains of 4-5 synchronous calls, adding latency.

We process approximately 100K events per day. We expect this to grow to 500K within 12 months. The team has 8 backend engineers, 2 of whom have prior Kafka experience.

Decision

We will adopt Apache Kafka as the primary mechanism for asynchronous inter-service communication. Synchronous HTTP will remain for request/response patterns where the caller needs an immediate result (e.g., authentication checks).

We will use Confluent Cloud as the managed Kafka provider to minimize operational overhead.

Alternatives Considered

RabbitMQ

Pros: Simpler to operate, lower learning curve, supports multiple messaging patterns (pub/sub, point-to-point, routing).
Cons: Less suitable for event sourcing patterns we plan to adopt. Weaker replay/rewind capabilities. Community momentum has shifted toward Kafka for event-driven architectures.
Why rejected: We anticipate needing event replay for audit and debugging. Kafka’s log-based architecture is better suited.

AWS SQS + SNS

Pros: Fully managed, no infrastructure to maintain, tight AWS integration.
Cons: Vendor lock-in to AWS. Limited message ordering guarantees. No built-in stream processing (would need Kinesis or Lambda). Higher per-message cost at our projected volume.
Why rejected: We want to avoid deepening AWS lock-in, and we need ordered message delivery for financial events.

Keep Synchronous HTTP (with circuit breakers)

Pros: No new infrastructure. Team already familiar. Circuit breakers address cascade failures.
Cons: Doesn’t solve tight coupling. Latency still accumulates across call chains. Circuit breakers are a band-aid, not a solution to the fundamental coupling problem.
Why rejected: Addresses symptoms, not root cause.

Consequences

Positive

Services become decoupled: producers don’t need to know consumers.
Cascade failures eliminated for async workflows.
Event replay enables powerful debugging and audit capabilities.
Foundation for event sourcing patterns in future services.

Negative

Operational complexity increases (Kafka cluster, schema registry, consumer group management). Mitigated by using Confluent Cloud.
Team needs training on Kafka concepts and patterns.
Eventual consistency replaces strong consistency for async flows. Some workflows need redesign.
Debugging distributed async flows is harder than tracing synchronous HTTP calls. We’ll need distributed tracing (see ADR-0024).

Risks

If message volume exceeds Confluent Cloud pricing tiers, costs could increase significantly. Monitor and set alerts.
Schema evolution across services requires discipline. Plan to adopt Avro with schema registry.

ADR-0024: Adopt OpenTelemetry for Distributed Tracing
ADR-0018: Service Communication Contracts (superseded by this ADR)