Deal-Engines on Corebaseit — Design System Engineering · POS · Payments · Security

contact@corebaseit.com (Vincent Bevia) — Mon, 01 Jan 0001 00:00:00 +0000

ADR-0001: Use Apache Kafka for Inter-Service Communication

Status

Accepted

Date

2025-11-15

Context

Our platform has grown from 3 microservices to 12 over the past year. Services currently communicate synchronously via HTTP REST calls. This creates several problems:

Cascade failures: when the inventory service is down, the order service can’t process orders, even though inventory checks could be eventual.
Tight coupling: services need to know each other’s API contracts and endpoints.
Performance bottlenecks: some operations trigger chains of 4-5 synchronous calls, adding latency.

We process approximately 100K events per day. We expect this to grow to 500K within 12 months. The team has 8 backend engineers, 2 of whom have prior Kafka experience.

Decision

We will adopt Apache Kafka as the primary mechanism for asynchronous inter-service communication. Synchronous HTTP will remain for request/response patterns where the caller needs an immediate result (e.g., authentication checks).

We will use Confluent Cloud as the managed Kafka provider to minimize operational overhead.

Alternatives Considered

RabbitMQ

Pros: Simpler to operate, lower learning curve, supports multiple messaging patterns (pub/sub, point-to-point, routing).
Cons: Less suitable for event sourcing patterns we plan to adopt. Weaker replay/rewind capabilities. Community momentum has shifted toward Kafka for event-driven architectures.
Why rejected: We anticipate needing event replay for audit and debugging. Kafka’s log-based architecture is better suited.

AWS SQS + SNS

Pros: Fully managed, no infrastructure to maintain, tight AWS integration.
Cons: Vendor lock-in to AWS. Limited message ordering guarantees. No built-in stream processing (would need Kinesis or Lambda). Higher per-message cost at our projected volume.
Why rejected: We want to avoid deepening AWS lock-in, and we need ordered message delivery for financial events.

Keep Synchronous HTTP (with circuit breakers)

Pros: No new infrastructure. Team already familiar. Circuit breakers address cascade failures.
Cons: Doesn’t solve tight coupling. Latency still accumulates across call chains. Circuit breakers are a band-aid, not a solution to the fundamental coupling problem.
Why rejected: Addresses symptoms, not root cause.

Consequences

Positive

Services become decoupled: producers don’t need to know consumers.
Cascade failures eliminated for async workflows.
Event replay enables powerful debugging and audit capabilities.
Foundation for event sourcing patterns in future services.

Negative

Operational complexity increases (Kafka cluster, schema registry, consumer group management). Mitigated by using Confluent Cloud.
Team needs training on Kafka concepts and patterns.
Eventual consistency replaces strong consistency for async flows. Some workflows need redesign.
Debugging distributed async flows is harder than tracing synchronous HTTP calls. We’ll need distributed tracing (see ADR-0024).

Risks

If message volume exceeds Confluent Cloud pricing tiers, costs could increase significantly. Monitor and set alerts.
Schema evolution across services requires discipline. Plan to adopt Avro with schema registry.

ADR-0024: Adopt OpenTelemetry for Distributed Tracing
ADR-0018: Service Communication Contracts (superseded by this ADR)

contact@corebaseit.com (Vincent Bevia) — Mon, 01 Jan 0001 00:00:00 +0000

ADR-0002: Use AWS SQS for Asynchronous Inter-Service Communication. (from ChatGPT)

Status

Accepted

Date

2026-08-31

Context

As Deal Engine extracts capabilities from the existing monolith into independent services, some workflows no longer require synchronous request/response communication.

Operations such as refund processing may involve slow or temporarily unavailable external systems such as GDS and airline APIs. Using synchronous HTTP for these workflows creates runtime coupling: if the downstream service is unavailable or slow, the upstream service is also affected.

Deal Engine already operates its infrastructure primarily on AWS. We need a reliable asynchronous communication mechanism that integrates with the existing platform without introducing unnecessary operational complexity.

Decision

We will use Amazon SQS as the default mechanism for asynchronous point-to-point communication between services.

Synchronous HTTP will remain for operations where the caller requires an immediate response.

Consumers must assume at-least-once delivery and therefore be designed to handle duplicate messages safely.

Alternatives Considered

Apache Kafka

Pros: High throughput, durable event log, strong replay capabilities, stream processing ecosystem.
Cons: Higher operational and conceptual complexity than required for our current workloads.
Why rejected: We primarily need reliable asynchronous work distribution, not event streaming or long-term event replay.

RabbitMQ

Pros: Mature messaging platform with flexible routing capabilities.
Cons: Introduces another infrastructure technology and operational model.
Why rejected: SQS provides the required queueing capabilities and integrates naturally with our existing AWS infrastructure.

Synchronous HTTP

Pros: Simple request/response model and immediate results.
Cons: Runtime coupling, failure propagation, and poor fit for long-running external operations.
Why rejected: It remains appropriate for synchronous workflows but not as the default for asynchronous processing.

Consequences

Positive

Services are less runtime-coupled.
Temporary consumer outages do not necessarily cause upstream failures.
Consumers can process and scale independently.
Fully managed AWS service with minimal operational overhead.
Natural fit with Deal Engine’s existing AWS infrastructure.

Negative

Asynchronous workflows introduce eventual consistency.
SQS provides at-least-once delivery, so consumers must handle duplicates.
Distributed asynchronous workflows require stronger observability and correlation.
Engineers must account for queue-specific concepts such as visibility timeout and message retention.

ADR-0024: Idempotency Strategy for SQS Consumers
ADR-0025: Transactional Outbox for Reliable Message Publication
ADR-0026: Dead-Letter Queue and Retry Strategy
ADR-0027: Use SNS for Event Fan-Out

contact@corebaseit.com (Vincent Bevia) — Mon, 01 Jan 0001 00:00:00 +0000

ADR-0001: Use Amazon SNS + SQS for Inter-Service Communication (Claude)

Status

Accepted

Date

2025-11-15

Context

Our platform has grown from 3 microservices to 12 over the past year. Services currently communicate synchronously via HTTP REST calls. This creates several problems:

Cascade failures: when the inventory service is down, the order service can’t process orders, even though inventory checks could be eventual.
Tight coupling: services need to know each other’s API contracts and endpoints.
Performance bottlenecks: some operations trigger chains of 4-5 synchronous calls, adding latency.

We process approximately 100K events per day. We expect this to grow to 500K within 12 months. That is roughly 1-6 messages per second on average, with bursts well inside the limits of any managed broker.

Deal Engine already runs almost entirely on AWS: compute, storage, networking, IAM, and observability are all AWS-native, and our infrastructure is defined in Terraform against AWS providers. The team has 8 backend engineers. All 8 work with AWS services daily; 2 have prior Kafka experience.

Decision

We will adopt Amazon SNS + SQS as the primary mechanism for asynchronous inter-service communication, using the standard fan-out topology:

Producers publish domain events to an SNS topic per event family (e.g. orders, inventory, payments).
Each consuming service owns a dedicated SQS queue subscribed to the relevant topic(s), with a filter policy so it only receives the message types it cares about.
Every queue gets a dead-letter queue with a redrive policy (default maxReceiveCount: 5).

Where ordering matters — financial events in particular — we will use FIFO topics and FIFO queues, with the message group ID set to the entity being mutated (e.g. deal_id, account_id). This gives ordering per entity plus deduplication, while still allowing parallel processing across entities.

Synchronous HTTP remains for request/response patterns where the caller needs an immediate result (e.g. authentication checks).

Topics, queues, subscriptions, filter policies, and DLQs are all defined in Terraform alongside the rest of our infrastructure. Access is granted through IAM roles, not shared credentials.

Alternatives Considered

Apache Kafka (Confluent Cloud or MSK)

Pros: Log-based retention makes event replay and rewind trivial. Strong ecosystem for stream processing (Kafka Streams, ksqlDB). Partition-level ordering with high throughput. Vendor-neutral in principle.
Cons: Substantial operational and conceptual surface area — brokers or a managed vendor, schema registry, consumer groups, partition and rebalance tuning, offset management. Only 2 of 8 engineers have worked with it. Introduces a second infrastructure vendor with its own billing, networking (PrivateLink/VPC peering), IAM model, and Terraform provider, none of which composes with our existing AWS setup. Baseline cost is a fixed monthly cluster charge that dwarfs our actual usage at 100-500K events/day.
Why rejected: We would be paying the full price of Kafka’s complexity to serve a workload two to three orders of magnitude below where that complexity starts to pay for itself. “Vendor neutrality” is largely theoretical for us — we are already on AWS for everything else, so adding Confluent does not reduce lock-in, it adds a second lock-in.

RabbitMQ

Pros: Simpler than Kafka, flexible routing (topic, direct, fanout exchanges), mature and well understood.
Cons: We would either self-host (brokers, clustering, quorum queues, upgrades, on-call) or run Amazon MQ, which costs more than SNS+SQS for the same job. Weak replay. No advantage over SQS for our access patterns.
Why rejected: All of the operational burden of running a broker, none of the benefits over a fully managed AWS-native option.

Keep Synchronous HTTP (with circuit breakers)

Pros: No new infrastructure. Team already familiar. Circuit breakers address cascade failures.
Cons: Doesn’t solve tight coupling. Latency still accumulates across call chains. Circuit breakers are a band-aid, not a solution to the fundamental coupling problem.
Why rejected: Addresses symptoms, not root cause.

EventBridge

Pros: Also AWS-native. Richer content-based routing than SNS filter policies, schema registry, built-in archive and replay, and native third-party SaaS integrations.
Cons: Higher per-event cost, no FIFO ordering guarantees, and at-least-once delivery with weaker throughput ceilings for high-fan-in workloads.
Why rejected (for now): Not rejected outright. SNS+SQS covers our service-to-service backbone more cheaply and with the ordering guarantees we need. We expect to use EventBridge alongside it for cross-account, SaaS, and scheduled/rule-driven integrations, and potentially for the event archive (see Risks).

Consequences

Positive

Services become decoupled: producers publish to a topic and don’t need to know who consumes it. New consumers are added by creating a queue and a subscription, with no producer change.
Cascade failures eliminated for async workflows. A down consumer means a growing queue, not a failed order.
No infrastructure to operate: no brokers, no clusters, no patching, no capacity planning, no 3am rebalance debugging.
Native fit with what we already run: IAM for authz, CloudWatch for metrics and alarms, X-Ray for tracing, Terraform for provisioning, Lambda and ECS for consumers with SQS-driven autoscaling.
Dead-letter queues and redrive are built in. Poison messages are isolated and replayable without custom tooling.
Costs scale to zero and stay small. At 500K events/day (~15M/month) fanned out to a handful of consumers, this is on the order of tens of dollars a month, versus a fixed several-hundred-dollar floor for a managed Kafka cluster. (Approximate — SQS ~$0.40/M requests standard, ~$0.50/M FIFO; SNS ~$0.50/M publishes with SQS deliveries free. Verify current pricing for our region before budgeting.)
Every engineer on the team can already reason about it.

Negative

No event log, no free replay. This is the real trade-off against Kafka. SQS is a queue, not a log: once a message is consumed and deleted it’s gone, and maximum retention is 14 days. Replaying last quarter’s events is not something the transport gives us. See Risks for mitigation.
No built-in stream processing. There is no equivalent of Kafka Streams. Aggregations, joins, and windowing must be done in consumer code, or by moving that specific workload to Kinesis Data Streams / Firehose if one ever justifies it.
Full event-sourcing patterns are off the table with this transport alone. If a future service genuinely needs an immutable ordered log as its source of truth, it should use DynamoDB Streams, Kinesis, or a dedicated event store — and that will need its own ADR rather than being bolted onto SQS.
Deeper AWS lock-in. We accept this consciously. Mitigation: keep a thin internal publish/subscribe interface in our shared library so that SNS/SQS SDK calls are not scattered through domain code.
256 KB message size limit. Large payloads need the S3 extended client (claim check pattern) or, better, slimmer events that carry IDs rather than full entity snapshots.
Eventual consistency replaces strong consistency for async flows. Some workflows need redesign.
Delivery is at-least-once on standard queues, so consumers must be idempotent. FIFO queues offer deduplication within a 5-minute window, which helps but is not a substitute for idempotent handlers.
Debugging distributed async flows is harder than tracing synchronous HTTP calls. We’ll need distributed tracing (see ADR-0024).

Risks

Loss of audit/debug replay. Mitigation: subscribe an archival consumer to each topic that writes raw events to S3 (via Firehose or a small Lambda), partitioned by date. This gives us durable, queryable history through Athena at negligible cost, and a path to re-publish if needed. This should be set up as part of the initial rollout, not deferred.
FIFO throughput ceilings. Default FIFO queues handle 300 messages/sec (3,000 with batching); high-throughput mode raises this substantially. Our projected peak is far below the default, but message group ID choice matters — grouping too coarsely (e.g. one group for all events) serialises everything. Group by entity.
SNS FIFO topics can only deliver to SQS FIFO queues. Any consumer of an ordered topic must use a FIFO queue; Lambda and HTTP subscriptions won’t work directly on those topics.
Filter policy sprawl. As topics grow, subscription filters can become an undocumented routing layer. Keep them in Terraform, keep them simple, and prefer more topics over more elaborate filters.
Schema evolution across services still requires discipline — we lose Confluent’s schema registry. Plan: versioned event contracts in a shared package, additive-only changes, consumers ignore unknown fields. Consider EventBridge Schema Registry if this becomes painful.

ADR-0024: Adopt OpenTelemetry for Distributed Tracing
ADR-0018: Service Communication Contracts (superseded by this ADR)

contact@corebaseit.com (Vincent Bevia) — Mon, 01 Jan 0001 00:00:00 +0000

ADR 003: Choice of Message Broker for Asynchronous Job Orchestration

Google Cloud Tasks vs. Google Cloud Pub/Sub

This record documents the architectural reasoning for transitioning the Deal Engine platform away from its current database-polling model toward a modern, event-driven task queue. It summarises the comparison between the two candidate Google Cloud messaging technologies and the rationale for the selected option.

1. Summary

Field	Value
Status	Proposed
Scope	Deal Engine — asynchronous job orchestration
Decision	Adopt Google Cloud Tasks; deprecate NoSQL polling
Affected Services	Radar, Papi Regional Worker, PAPI Scheduler, NoSQL task store

2. Context: The Database-as-a-Queue Bottleneck

Currently, the regional communication services (Radar and Papi) actively poll the NoSQL database to claim and execute ticket processing tasks. [6, 8, 12] This database-as-a-queue model suffers from several distributed systems pain points:

Severe concurrency risks: Multiple workers can claim the same ticket and trigger double-refunds. [4, 6]
Continuous read overhead: High-frequency polling imposes constant load on the NoSQL instance irrespective of task volume. [6]
No native backpressure: There is no queue-level mechanism to protect fragile third-party airline APIs from traffic spikes. [12]

3. Options Evaluated

The architectural evaluation compared two primary GCP messaging technologies to resolve these bottlenecks:

Evaluation Criteria	Option 1: Google Cloud Tasks (Recommended)	Option 2: Google Cloud Pub/Sub
Messaging Model	Target-Specific Task Queue (Message-to-Endpoint) — the broker manages queues and actively pushes tasks to specific worker HTTP endpoints.	Publish/Subscribe (Message-to-Subscription) — high-throughput topic-based broadcasting where multiple subscribers pull or receive pushes.
Rate-Limiting & Backpressure	Excellent (Built-in) — allows native configuration of maximum dispatch rates (tasks/second) and concurrent dispatches per queue. [5]	Poor (Client-Side) — throttling and flow control must be manually programmed and managed at the individual subscriber/worker level. [5]
Deduplication	Native & Long-Lived — automatically rejects duplicate tasks matching a custom Task Name within a 20-hour window. [5]	No Native Deduplication — requires manual implementation of lock tables or state validation inside the worker code or database layer. [5]
Workload Alignment	Perfect Fit — ticket processing is naturally a point-to-point task (Scheduler → Worker) rather than a multi-system fan-out. [5]	Overkill — high-throughput, horizontal event streaming is designed for massive event fan-outs rather than single-worker task execution.
Infrastructure Overhead	Zero — already a listed, pre-existing asset in Deal Engine’s core GCP infrastructure inventory. [10]	Medium — requires provisioning new Pub/Sub topics, subscriptions, and access permissions.

4. Decision & Justification

We select Google Cloud Tasks to replace the NoSQL polling mechanism.

The driving factor is that rate-limiting and strict deduplication are paramount when dealing with fragile GDS partners and high-stakes financial operations. [5, 12] Cloud Tasks allows us to:

Enforce hard rate limits at the queue level, protecting third-party airline endpoints from being overwhelmed during traffic spikes. [5]
Utilize native deduplication by naming the task using our deterministic idempotency_key. If a duplicate execution is triggered, Cloud Tasks drops it at the broker level before it can ever reach Radar, entirely preventing accidental duplicate ticket transactions. [5]
Minimize infrastructure procurement delays by leveraging a resource already approved and listed in our cloud stack. [10]

5. Consequences

Positive: The NoSQL database is immediately freed from polling CPU and IOPS overhead [6], backpressure is enforced at the queue level, and duplicate executions are programmatically prevented. [5]
Neutral: Regional workers (Radar and Papi) must be refactored to act as HTTP target endpoints, secured via private VPC networks and IAM authentication. [8]

6. Migration Strategy

A four-phase migration path will be executed to ensure zero downtime for active transactional traffic:

Phase	Stage	Description
Phase 1	Shadowing	Tasks are dispatched to Cloud Tasks alongside the existing polling flow, without executing the final GDS action, to validate connectivity and payload handling.
Phase 2	Canary Routing	A small share of live transactional traffic is routed through Cloud Tasks while the legacy polling mechanism remains available as a fallback.
Phase 3	Scale-Up	Traffic through Cloud Tasks is increased incrementally while dispatch stability, retry rates, and success rates are monitored.
Phase 4	Polling Deprecation	Once stability is confirmed, the legacy polling logic is decommissioned and the associated NoSQL schemas are retired.

contact@corebaseit.com (Vincent Bevia) — Mon, 01 Jan 0001 00:00:00 +0000

ADR 003: Message Broker Selection for Asynchronous Job Orchestration

Replacing NoSQL Polling

The current reliance on NoSQL polling for core transactional workflows has reached its scaling limit, necessitating a shift to a push-based orchestration model. This ADR outlines the transition to a managed message broker to support Deal Engine’s multi-cloud evolution, ensuring regional reliability while safely decomposing the Aleron monolith.

1. Status and Meta-Data

Field	Value
Status	Proposed
Date	October 24, 2023
Deciders	Principal Engineering Team
Consulted	Infrastructure, DevOps, and Backend Engineering (PAPI and Radar teams)

2. Context and Problem Statement

The Deal Engine architecture currently utilizes a “Database-as-a-Queue” anti-pattern. In this model, the Aleron core and the PAPI Coordinator (the logic engine) write task states to a NoSQL database. Regional workers—specifically Radar and the Papi Regional Worker—continuously poll this database to identify pending tasks. Once a task is retrieved, the workers execute the action against external Global Distribution Systems (GDS) and airline APIs via the Twingate Client.

Operational Pain Points

This “pull-based” model introduces significant systemic risks:

Concurrency and Race Conditions: The lack of atomic message visibility leads to multiple workers claiming the same task. In the travel domain, this results in duplicate ticket transactions and double-refunds.
Resource Inefficiency: High-frequency polling against the NoSQL instance creates constant, unnecessary CPU and I/O load, regardless of actual task volume.
Backpressure Deficits: There is no native mechanism to throttle outgoing requests. This leaves external GDS endpoints vulnerable to traffic spikes, risking IP blocking and rate-limit penalties.
Non-Deterministic Latency for Rudder: The polling delay negatively impacts Rudder (the Tax Calculator), creating inconsistent timing for critical tax and fee calculations that must precede ticket finalization.
Fragile Error Recovery: Retries and crash recovery are managed manually via database state. If a worker fails mid-execution, tasks often hang in a “Processing” state without native visibility timeouts.

3. Decision Drivers

The selection of a replacement broker is governed by the specialized constraints of the travel industry rather than raw throughput. Our primary requirement is the protection of fragile external endpoints.

Key Differentiators

Granular Rate Limiting: We must strictly enforce Transactions Per Second (TPS) limits at the queue level to honor GDS constraints.
Guaranteed Idempotency: The system must support deterministic keys to prevent duplicate execution of any ticket action.
Scheduled Execution: Many Deal Engine jobs require delayed execution (e.g., “process refund in 2 hours”), a feature currently managed poorly by database timestamps.
Operational Simplicity: We will prioritize existing Google Cloud Platform (GCP) assets to minimize security review friction and leverage our established GKE and Cloud Run footprints.

4. Options Evaluated

Option 1: Google Cloud Tasks (The Targeted Queue)

Cloud Tasks is a distributed queue service designed for “message-to-endpoint” push communication.

Push Model: Cloud Tasks explicitly targets HTTP endpoints, allowing it to trigger workers residing on our existing Google Kubernetes Engine (GKE) and Cloud Run clusters.
Native Scheduling: It supports a schedule_time parameter, allowing the Scheduler to offload future-dated jobs directly to the broker.
Deduplication: It provides deterministic deduplication via the “Task Name” feature.
Rate Control: Offers native max_dispatch_rate and max_concurrent_dispatches to protect downstream APIs and the Twingate egress point.

Option 2: Google Cloud Pub/Sub (The Event-Driven Broker)

Pub/Sub is a high-scale asynchronous messaging service designed for fan-out patterns.

Limitations: Pub/Sub lacks native, broker-side rate limiting and does not support scheduled message delivery (delayed messages).
Delivery Model: It follows an “at-least-once” delivery model without the deterministic deduplication required to prevent double-refunds.
Contextual Fit: While excellent for high-throughput analytics ingestion, it is ill-suited for the point-to-point, task-oriented coordination required between the PAPI Coordinator and regional workers.

5. Decision Outcome & Justification

The engineering team will implement Google Cloud Tasks as the primary orchestration engine for core transactional operations.

Strategic Alignment

TPS Protection: The max-dispatch-rate feature in Cloud Tasks acts as a native “Control Tower,” ensuring we never exceed GDS TPS limits regardless of internal traffic spikes.
Deterministic Idempotency: We will implement a standardized naming convention for tasks: [service]-[ticketID]-[transactionID]-[attemptCount]. Because Cloud Tasks prevents the creation of two tasks with the same name within a queue, this effectively eliminates the “double-worker” race condition at the infrastructure level.
Virtual Control Tower: While the legacy Control Tower UI remains for business visibility, Cloud Tasks provides an operational Control Tower via the GCP Console, offering native visibility into queue depth, retry rates, and execution latency.
Infrastructure Synergy: Cloud Tasks is already an approved asset within the Deal Engine Primary Infrastructure list. It integrates seamlessly with our current GKE/Cloud Run execution environments without requiring new vendor procurement.

6. Consequences and Impact Analysis

Positive Impacts

Reduced Database Pressure: Removing polling will significantly lower NoSQL I/O, allowing that resource to scale more effectively for state persistence.
Stabilized Rudder Calculations: Transitioning to a push model ensures that tax calculations are triggered immediately upon task creation, reducing non-deterministic wait times.
Operational Reliability: Native dead-letter queues and configurable retry backoffs replace fragile, manual database state management.

Negative / Neutral Impacts

Worker Refactoring: The Radar and Papi workers must be updated to serve as secured HTTP targets. This requires configuring Identity-Aware Proxy (IAP) or service-to-service OIDC authentication.
Retention Management: Cloud Tasks has a 31-day task retention limit. Long-term auditing of task outcomes must still be persisted to Cloud SQL.

7. Migration Roadmap

To adhere to our Zero-Interruption mandate, the transition will follow a four-phase approach:

Phase	Activity	Description
Phase 1	Shadowing / Dry-run	The PAPI Coordinator will begin dispatching tasks to Cloud Tasks in a “shadow” mode. Workers will receive the payload but will not execute the final GDS action, allowing us to validate regional connectivity through the Twingate Client.
Phase 2	Canary Hybrid Routing	We will divert a small percentage (e.g., 5%) of transactional traffic through Cloud Tasks. The remaining 95% will continue via the legacy NoSQL polling mechanism as a fallback.
Phase 3	Scale-up	We will gradually increase the traffic percentage while monitoring the GCP Console and internal observability tools for dispatch stability and success rates.
Phase 4	Polling Deprecation	Once 100% stability is achieved over a full business cycle, we will decommission the legacy polling logic in the PAPI Communication Service and clean up the associated NoSQL schemas.

contact@corebaseit.com (Vincent Bevia) — Mon, 01 Jan 0001 00:00:00 +0000

ADR-0023: Use Apache Kafka for Inter-Service Communication

Status

Accepted

Date

2025-11-15

Context

Our platform has grown from 3 microservices to 12 over the past year. Services currently communicate synchronously via HTTP REST calls. This creates several problems:

Cascade failures: when the inventory service is down, the order service can’t process orders, even though inventory checks could be eventual.
Tight coupling: services need to know each other’s API contracts and endpoints.
Performance bottlenecks: some operations trigger chains of 4-5 synchronous calls, adding latency.

We process approximately 100K events per day. We expect this to grow to 500K within 12 months. The team has 8 backend engineers, 2 of whom have prior Kafka experience.

Decision

We will use Confluent Cloud as the managed Kafka provider to minimize operational overhead.

Alternatives Considered

RabbitMQ

Pros: Simpler to operate, lower learning curve, supports multiple messaging patterns (pub/sub, point-to-point, routing).
Cons: Less suitable for event sourcing patterns we plan to adopt. Weaker replay/rewind capabilities. Community momentum has shifted toward Kafka for event-driven architectures.
Why rejected: We anticipate needing event replay for audit and debugging. Kafka’s log-based architecture is better suited.

AWS SQS + SNS

Pros: Fully managed, no infrastructure to maintain, tight AWS integration.
Cons: Vendor lock-in to AWS. Limited message ordering guarantees. No built-in stream processing (would need Kinesis or Lambda). Higher per-message cost at our projected volume.
Why rejected: We want to avoid deepening AWS lock-in, and we need ordered message delivery for financial events.

Keep Synchronous HTTP (with circuit breakers)

Pros: No new infrastructure. Team already familiar. Circuit breakers address cascade failures.
Cons: Doesn’t solve tight coupling. Latency still accumulates across call chains. Circuit breakers are a band-aid, not a solution to the fundamental coupling problem.
Why rejected: Addresses symptoms, not root cause.

Consequences

Positive

Services become decoupled: producers don’t need to know consumers.
Cascade failures eliminated for async workflows.
Event replay enables powerful debugging and audit capabilities.
Foundation for event sourcing patterns in future services.

Negative

Operational complexity increases (Kafka cluster, schema registry, consumer group management). Mitigated by using Confluent Cloud.
Team needs training on Kafka concepts and patterns.
Eventual consistency replaces strong consistency for async flows. Some workflows need redesign.
Debugging distributed async flows is harder than tracing synchronous HTTP calls. We’ll need distributed tracing (see ADR-0024).

Risks

If message volume exceeds Confluent Cloud pricing tiers, costs could increase significantly. Monitor and set alerts.
Schema evolution across services requires discipline. Plan to adopt Avro with schema registry.

ADR-0024: Adopt OpenTelemetry for Distributed Tracing
ADR-0018: Service Communication Contracts (superseded by this ADR)

contact@corebaseit.com (Vincent Bevia) — Mon, 01 Jan 0001 00:00:00 +0000

ADR-NNNN: [Short Title Describing the Decision]

Status

[Proposed | Accepted | Deprecated | Superseded by ADR-XXXX]

Date

[YYYY-MM-DD]

Context

[Describe the situation. What problem are we solving? What constraints exist? What forces are at play? Be specific – include numbers, deadlines, team capabilities, and technical requirements where relevant.]

Decision

[State the decision clearly and concisely. Use active voice: “We will use X” not “X should be considered.”]

Alternatives Considered

[Alternative 1]

Pros: …
Cons: …
Why rejected: …

[Alternative 2]

Pros: …
Cons: …
Why rejected: …

Consequences

Positive

[What becomes easier or better]

Negative

[What becomes harder or worse]

Risks

[What could go wrong]

[Links to related ADRs]

contact@corebaseit.com (Vincent Bevia) — Mon, 01 Jan 0001 00:00:00 +0000

For Deal Engine, I’d make the ADR much more specific than “migrate the monolith to microservices.” An ADR should capture one architectural decision.

Given their domain and the migration context, a strong example would be extracting Refund Processing from the Scala monolith. It naturally lets you discuss migration strategy, GDS/external-provider failures, retries, idempotency, observability, and rollback.

Here’s a realistic ADR you could use for interview preparation:

ADR-0024: Extract Refund Processing from the Scala Monolith into a Dedicated Service

Status

Proposed

Date

2026-08-30

Context

Deal Engine currently handles refund processing as part of the existing Scala monolith.

Refund processing has become a good candidate for extraction because it has a relatively clear business boundary and interacts with external systems such as GDS and airline APIs.

The current implementation creates several challenges:

Refund logic is coupled to the deployment lifecycle of the monolith.
Failures or latency in external GDS/airline systems can consume resources inside the monolith.
Retry and recovery logic is difficult to evolve independently.
Refund operations may take significantly longer than normal request/response operations.
External systems may successfully process a refund even when Deal Engine does not receive the response.
Duplicate requests or worker retries must not result in duplicate refunds.
Observability of the complete refund lifecycle is difficult when processing is embedded inside the larger application.

Migrating the entire monolith at once would introduce significant technical and operational risk.

We therefore need a migration approach that allows functionality to be extracted incrementally while the existing platform continues operating.

Decision

We will extract refund processing from the Scala monolith into a dedicated Refund Service.

The migration will follow an incremental Strangler Fig approach rather than rewriting the existing platform.

The initial architecture will be:

Client / Internal System
 │
 ▼
 API Gateway
 │
 ▼
 Scala Monolith
 │
 │ refund request
 ▼
 Refund Service
 │
 ├── Refund Database
 │
 └── GDS / Airline APIs

During the first migration phase, the monolith will remain the public entry point for refund requests.

The monolith will delegate eligible refund operations to the Refund Service.

The Refund Service will own:

refund orchestration;
refund state transitions;
idempotency;
interaction with GDS and airline APIs;
retry policies;
timeout handling;
reconciliation;
refund-specific persistence;
refund-specific observability.

The service will maintain an explicit refund state machine such as:

REQUESTED
 │
 ▼
PROCESSING
 │
 ├───────────────┐
 ▼ ▼
SUCCEEDED FAILED
 │
 │
 └──── UNKNOWN
 │
 ▼
 RECONCILIATION

A refund request must contain an idempotency key or stable refund identifier.

Repeated processing of the same refund identifier must not create another external refund.

The first version will preserve synchronous communication between the monolith and Refund Service where an immediate response is available.

Long-running operations, retries, and reconciliation may later be moved to asynchronous processing through a queue.

This allows the service boundary to be established before introducing additional messaging infrastructure.

Alternatives Considered

Rewrite the Entire Monolith as Microservices

Pros

Clean architecture without transitional components.
Opportunity to redesign all service boundaries.
Removes legacy architecture in one migration.

Cons

Very large migration scope.
Long period before business value is delivered.
Difficult to validate behavioral equivalence.
High rollback risk.
Existing business rules may be poorly documented.

Why rejected

A full rewrite introduces unnecessary risk. Incremental extraction allows individual business capabilities to be migrated and validated independently.

Extract a Simpler CRUD Capability First

For example, configuration or reference-data management.

Pros

Easier first migration.
Lower operational risk.
Useful for validating deployment and infrastructure.

Cons

Does not exercise the difficult architectural problems Deal Engine needs to solve.
Provides limited evidence for handling concurrency, retries, external-system failures, and long-running workflows.

Why rejected

Refund processing provides a more meaningful vertical slice through the architecture and validates patterns required by future service extractions.

Keep Refund Processing Inside the Monolith

Pros

No migration effort.
No additional service infrastructure.
Existing transactional model remains unchanged.

Cons

Refund processing remains coupled to the monolith.
External-system latency continues affecting monolith resources.
Independent scaling and deployment are impossible.
Recovery and reconciliation logic remain mixed with unrelated functionality.

Why rejected

This preserves the architectural constraints that the migration is intended to remove.

Consequences

Positive

Refund processing can evolve independently from the monolith.
GDS and airline integration failures are isolated behind a clear service boundary.
Refund-specific retry and recovery policies become easier to implement.
Independent deployment becomes possible.
Independent scaling becomes possible.
Observability can be built around the complete refund lifecycle.
The migration pattern can later be reused for other capabilities.

The architecture can evolve incrementally:

 MONOLITH
 │
 │
 ┌────────┴────────┐
 │ │
 Existing Logic Refund Service
 │
 ▼
 GDS / Airline

followed eventually by:

 API Gateway
 │
 ┌──────────────┼──────────────┐
 ▼ ▼ ▼
 Monolith Refund Service Other
 Services

Negative

The platform temporarily operates as a distributed system plus a monolith.
Network failures must now be considered between the monolith and Refund Service.
Distributed tracing becomes necessary.
Data ownership boundaries must be clearly defined.
Some business transactions can no longer rely on a single database transaction.
Operational complexity increases.

Risks

Duplicate refund

A worker may retry after an external refund succeeded but before the response was persisted.

Mitigation: idempotency keys, persistent refund state, and reconciliation with the external provider.

Unknown external state

The GDS may timeout after receiving the request.

The system must not assume:

TIMEOUT == FAILED

Instead:

TIMEOUT
 ↓
UNKNOWN
 ↓
QUERY / RECONCILE
 ↓
SUCCEEDED | FAILED

Data ownership

Both the monolith and Refund Service could accidentally become authoritative for refund state.

Mitigation: once a refund is migrated, the Refund Service becomes the source of truth for refund execution state.

Migration regressions

Existing refund behavior may contain undocumented business rules.

Mitigation: migrate incrementally, compare old/new behavior, use feature flags, and initially route only selected refund traffic through the new service.

Rollout Strategy

The migration will be gradual:

Phase 1
Refund Service deployed
 ↓
No production traffic

Phase 2
 ↓
Shadow / test traffic
 ↓
Compare behaviour

Phase 3
 ↓
Small % production traffic
 ↓
Monitor

Phase 4
 ↓
Increase traffic

Phase 5
 ↓
100% refund processing

Phase 6
 ↓
Remove refund logic
from monolith

A feature flag will control whether refund requests are processed by the existing monolith implementation or delegated to the Refund Service.

Rollback therefore consists of routing new refund requests back to the monolith while investigating the service.

Success Criteria

The extraction will be considered successful when:

refund behavior remains functionally equivalent;
duplicate refunds are prevented;
external timeout scenarios are recoverable;
migrated traffic can be rolled back safely;
refund processing is independently deployable;
refund lifecycle metrics and traces are available;
the monolith no longer owns refund execution logic.

ADR-0025: Refund Service Data Ownership
ADR-0026: Idempotency Strategy for Refund Processing
ADR-0027: Asynchronous Refund Processing
ADR-0028: Distributed Tracing with OpenTelemetry
ADR-0029: Service-to-GDS Retry and Reconciliation Strategy

Notice something particularly useful here: we deliberately did not decide everything in ADR-0024.

For example:

ADR-0024
Extract Refund Service
 │
 ├── ADR-0025
 │ Who owns refund data?
 │
 ├── ADR-0026
 │ How do we guarantee idempotency?
 │
 ├── SPIKE
 │ Can GDS refund status be queried reliably?
 │ ↓
 │ ADR-0029
 │ Reconciliation strategy
 │
 └── SPIKE
 SQS vs Kafka vs synchronous worker?
 ↓
 ADR-0027
 Async processing architecture

That is much closer to how I would expect a Principal Engineer to approach the migration: the ADR isn’t “the architecture document.” It records one consequential decision, while Spikes provide evidence for decisions where uncertainty remains.

contact@corebaseit.com (Vincent Bevia) — Mon, 01 Jan 0001 00:00:00 +0000

Status

Accepted

Date

2026-08-31

Context

Deal Engine is progressively moving functionality from the existing monolithic platform into independently deployable services.

As part of this migration, some workflows — such as refund processing, ticket operations, notifications, and external GDS interactions — do not require all processing to complete within a single synchronous request.

The platform currently relies heavily on synchronous service-to-service communication. This creates several problems:

Cascade failures: if a downstream service or external GDS is unavailable, upstream services may also fail or remain blocked waiting for a response.
Tight runtime coupling: the calling service requires the downstream service to be available at the same time.
Latency amplification: workflows involving several synchronous calls accumulate latency across the complete request chain.
External-system instability: GDS and airline APIs may timeout, respond slowly, or become temporarily unavailable.
Retry complexity: retrying synchronous operations can create duplicate processing unless idempotency is carefully implemented.
Independent scaling: some workloads, particularly refund and ticket-processing jobs, may need to scale independently from the services producing the work.

Deal Engine already operates its infrastructure primarily within AWS.

We therefore need a managed asynchronous messaging mechanism that integrates naturally with the existing AWS environment while introducing minimal additional operational complexity.

Not all communication should become asynchronous. Operations requiring an immediate response will continue to use synchronous APIs.

Decision

We will use Amazon SQS as the default mechanism for asynchronous work distribution between services.

Amazon SNS will be used where a single business event needs to be delivered to multiple independent consumers.

The general distinction will be:

SQS
────
"Someone needs to process this work."

Producer
 │
 ▼
 SQS Queue
 │
 ▼
Consumer


SNS + SQS
─────────
"Something happened and several systems
may independently care about it."

Producer
 │
 ▼
 SNS Topic
 │
 ├──────────────┐
 ▼ ▼
SQS Queue A SQS Queue B
 │ │
 ▼ ▼
Consumer A Consumer B

Synchronous HTTP APIs will remain appropriate when the caller requires an immediate result.

For example:

Authentication request

Service
 │
 │ HTTP
 ▼
Auth Service
 │
 ▼
Immediate response

while long-running refund processing could use:

Refund API
 │
 │ create refund
 ▼
Refund Service
 │
 │ persist
 ▼
Refund DB
 │
 │ enqueue work
 ▼
 SQS Refund Queue
 │
 ▼
Refund Worker
 │
 ▼
GDS / Airline API

The API may therefore return:

202 Accepted
refundId = REF-123
status = PENDING

rather than keeping the original request open while the complete external refund operation is performed.

Message Processing Semantics

SQS Standard queues provide at-least-once delivery.

Consumers must therefore assume that the same message can be delivered more than once.

For example:

SQS
 │
 │ RefundRequested(REF-123)
 ▼
Worker
 │
 │ calls GDS
 ▼
GDS
 │
 │ REFUND SUCCESSFUL
 ▼
Worker
 │
 💥 crashes before
 deleting message

SQS may later redeliver the message:

SQS
 │
 │ RefundRequested(REF-123)
 ▼
Worker

The consumer must not issue a second refund simply because the message was delivered again.

All consumers processing business-critical operations must therefore be idempotent.

A stable business identifier such as refundId will be used to detect previously processed operations.

Retry Strategy

Transient failures will be retried through SQS.

The SQS visibility timeout will prevent another worker from immediately processing a message while it is already being handled.

If processing succeeds, the consumer deletes the message.

If processing fails, the message becomes visible again after the visibility timeout.

Conceptually:

 SQS
 │
 ▼
 Worker
 │
 ┌─────┴─────┐
 │ │
 SUCCESS FAILURE
 │ │
 ▼ ▼
Delete message Message becomes
 visible again
 │
 ▼
 Retry

Retry policies must use bounded retries and backoff where appropriate.

Messages that repeatedly fail will be moved to a Dead-Letter Queue (DLQ).

SQS Queue
 │
 │ retry
 ▼
Consumer
 │
 │ retry
 ▼
Consumer
 │
 │ maxReceiveCount exceeded
 ▼
Dead-Letter Queue

DLQs will be monitored and must have an operational recovery procedure.

External-System Timeouts

A timeout when communicating with a GDS or airline system must not automatically be interpreted as a failed business operation.

For example:

Refund Worker
 │
 │ refund REF-123
 ▼
 GDS
 │
 │ refund succeeds
 │
 X response lost

Worker sees:
TIMEOUT

The actual business state may now be:

Deal Engine → UNKNOWN
GDS → REFUNDED

Blindly retrying the refund could therefore create a duplicate operation.

For operations where the external system supports status queries or stable external references, the service will reconcile the operation before retrying.

TIMEOUT
 │
 ▼
UNKNOWN
 │
 ▼
Query GDS using
stable reference
 │
 ├──────────────┐
 ▼ ▼
REFUNDED NOT FOUND
 │ │
 ▼ ▼
SUCCEEDED Retry

Idempotency and reconciliation requirements will be documented separately for each business-critical workflow.

SNS will not automatically be placed in front of every SQS queue.

For point-to-point asynchronous work:

Refund Service
 │
 ▼
 SQS
 │
 ▼
Refund Worker

is sufficient.

SNS will be introduced when an event has multiple independent consumers.

For example:

 RefundCompleted
 │
 ▼
 SNS
 / | \
 / | \
 ▼ ▼ ▼
 SQS Audit SQS Email SQS Analytics
 │ │ │
 ▼ ▼ ▼
 Audit Notify Analytics
 Service Service Service

This prevents the producer from needing to know which systems consume the event.

Each consumer receives its own SQS queue so that:

consumers process events independently;
one slow consumer does not block another;
each consumer can define its own retry policy;
each consumer can have its own DLQ;
services can scale independently.

Alternatives Considered

Apache Kafka

Pros

Excellent support for high-throughput event streaming.
Strong event replay capabilities.
Consumer groups allow multiple processing models.
Long-lived event logs are useful for audit and event-driven architectures.
Strong ecosystem for stream processing.

Cons

Introduces additional concepts and operational complexity.
Requires Kafka-specific monitoring, partition management, consumer offset management, and schema governance.
A managed Kafka platform would introduce another significant infrastructure dependency.
Kafka’s strengths are not required for the initial asynchronous workloads identified.

Why rejected

Deal Engine already relies heavily on AWS, and the immediate requirement is reliable asynchronous job and event processing rather than large-scale stream processing.

SQS and SNS provide the required capabilities while integrating directly with the existing AWS environment and reducing operational overhead.

Kafka may be reconsidered if future requirements include:

high-volume event streaming;
long-term event replay;
event sourcing;
stream processing;
multiple consumers independently replaying historical events.

RabbitMQ

Pros

Mature messaging system.
Flexible routing capabilities.
Supports queues and publish/subscribe patterns.
Familiar messaging semantics.

Cons

Introduces additional infrastructure to operate or another managed service dependency.
Does not provide a significant advantage over AWS-native messaging for the identified use cases.
Requires additional operational knowledge and monitoring.

Why rejected

The platform is already AWS-centric. SQS and SNS provide the required messaging capabilities without introducing another messaging technology.

Continue Using Synchronous HTTP

Pros

Simple programming model.
Existing team knowledge.
Immediate responses.
Straightforward debugging for simple request/response interactions.

Cons

Services remain runtime-coupled.
Downstream failures can propagate upstream.
Long-running GDS operations occupy request resources.
Retry handling becomes difficult.
Scaling producers and consumers independently is harder.

Why rejected as the default for asynchronous workflows

Synchronous communication remains appropriate where an immediate result is required, but it should not be required for long-running or independently recoverable workflows.

The architecture will therefore intentionally support both:

Need immediate answer?
 │
 ┌───┴───┐
 YES NO
 │ │
 HTTP SQS
 │
 │
 Multiple consumers?
 ┌──┴──┐
 NO YES
 │ │
 SQS SNS
 │
 SQS queues

Consequences

Positive

Reduced runtime coupling between services.
Downstream outages do not necessarily cause upstream failures.
Work can remain queued while consumers are temporarily unavailable.
Consumers can scale independently.
AWS manages the messaging infrastructure.
Native integration with existing AWS infrastructure and monitoring.
Built-in retry and dead-letter queue capabilities.
SNS enables event fan-out without coupling producers to consumers.
Supports gradual extraction of capabilities from the monolith.

Negative

Asynchronous workflows introduce eventual consistency.
Message processing becomes more difficult to trace than synchronous calls.
Consumers must be idempotent.
Duplicate message delivery must be expected.
Business workflows require explicit state management.
Engineers must understand visibility timeout, retention, retry, DLQ, and delivery semantics.
Debugging requires correlation IDs and distributed observability.

Risks

Duplicate Processing

SQS Standard provides at-least-once delivery.

Mitigation: all business-critical consumers must implement idempotent processing using stable business identifiers.

Poison Messages

A malformed or permanently failing message could otherwise be retried repeatedly.

Mitigation: configure bounded retries and DLQs.

Lost Database-to-Queue Events

A service could persist a business operation and crash before publishing the corresponding SQS message:

BEGIN TRANSACTION

INSERT refund
COMMIT

💥 crash

Send SQS message ← never happens

The database and message broker do not participate in the same transaction.

Mitigation: use the Transactional Outbox Pattern for workflows where loss of the message would violate business guarantees.

 SAME DB TRANSACTION

┌──────────────────────────────┐
│ │
│ INSERT refund │
│ │
│ INSERT outbox_event │
│ │
└──────────────┬───────────────┘
 │ COMMIT
 ▼

 Outbox Publisher
 │
 ▼
 SQS

External Operation Succeeds but Response Is Lost

Blind retrying could create duplicate external operations.

Mitigation: use stable external references, explicit UNKNOWN states, provider-side status queries where available, and reconciliation before retrying uncertain operations.

Poor Observability

Asynchronous processing makes request chains less obvious.

Mitigation: propagate correlation IDs and trace context through messages and instrument producers and consumers using the platform’s standard observability tooling.

Operational Requirements

Each production queue must define:

queue owner;
expected processing latency;
visibility timeout;
message retention period;
retry policy;
maximum receive count;
DLQ;
DLQ alarms;
consumer concurrency;
idempotency strategy;
correlation/trace identifiers;
recovery procedure.

CloudWatch alarms should detect conditions such as:

ApproximateAgeOfOldestMessage ↑
 │
 ▼
Consumers may not be keeping up


DLQ Message Count > 0
 │
 ▼
Processing requires investigation

Migration Strategy

SQS/SNS adoption will be incremental.

Existing synchronous workflows will not be converted solely for architectural consistency.

A workflow should move to asynchronous processing when there is a concrete benefit such as:

long-running processing;
external-system latency;
retry requirements;
temporary downstream unavailability;
independent scaling;
fan-out to multiple consumers;
eventual consistency being acceptable.

For the monolith-to-services migration, a typical extraction can therefore evolve as:

PHASE 1

Monolith
 │
 │ HTTP
 ▼
Extracted Service


PHASE 2

Monolith
 │
 ▼
 SQS
 │
 ▼
Extracted Service


PHASE 3 — when fan-out is required

Extracted Service
 │
 ▼
 SNS
 / \
 ▼ ▼
 SQS SQS
 │ │
 ▼ ▼
Service Service
 A B

Success Criteria

The decision will be considered successful when:

asynchronous workflows continue processing after temporary consumer outages;
duplicate message delivery does not produce duplicate business operations;
failed messages are recoverable through DLQs;
queue backlogs are observable and alertable;
services can scale independently;
GDS/external-system failures do not unnecessarily propagate to upstream services;
new services can be extracted from the monolith without introducing a new messaging platform for each workflow.

ADR-0024: Idempotency Strategy for Asynchronous Consumers
ADR-0025: Transactional Outbox for Reliable Event Publication
ADR-0026: Retry and Dead-Letter Queue Strategy
ADR-0027: Distributed Tracing for Asynchronous Workflows
ADR-0028: GDS Timeout and Reconciliation Strategy

contact@corebaseit.com (Vincent Bevia) — Mon, 01 Jan 0001 00:00:00 +0000

Observability in Distributed Systems: From Technical Telemetry to Business Confidence

Executive summary

Modern software systems rarely execute a business transaction inside a single application. A customer request may pass through an API, database, message broker, background worker and one or more external services before reaching its final state.

This distribution improves scalability and resilience, but it also makes failures harder to understand. A technically successful API response may conceal a transaction that later becomes delayed, duplicated or stranded. A timeout may mean that an external operation failed—or that it succeeded but its response was lost.

Observability provides the evidence required to understand these situations. It allows engineering teams to infer the internal state of a system from the telemetry it produces and to investigate questions that were not anticipated when the system was designed.

Effective observability is not achieved merely by collecting logs or deploying dashboards. It requires:

Structured and correlated telemetry
End-to-end visibility across synchronous and asynchronous boundaries
Business-state instrumentation
Explicit reliability objectives
Actionable alerting
Recovery mechanisms for ambiguous outcomes

The ultimate goal is not simply to determine whether servers are running. It is to establish whether business operations are progressing correctly, explain failures efficiently and recover safely when the outcome is uncertain.

1. Monitoring tells us what is known; observability supports investigation

Traditional monitoring is usually built around predefined questions:

Is CPU utilization too high?
Is the API returning errors?
Is the database connection pool exhausted?
Is the queue growing?

These are important questions, but they are not sufficient for distributed systems.

An engineer may instead need to investigate:

Why are transactions taking longer for one category of customer?
Did an external operation succeed before the connection was interrupted?
Why was the same message processed twice?
Which software or calculation version produced a disputed result?
Where did a transaction stop progressing?

Observability supports this type of investigation by combining multiple telemetry signals and preserving the relationships between them.

The central operating model is:

Metrics → Something is wrong
Traces → Where it is going wrong
Logs → What happened at that location
State → What the system believes about the business operation

OpenTelemetry describes traces, metrics and logs as complementary signals through which a system’s internal activity can be examined from different perspectives. Its observability primer also emphasizes the ability to investigate novel problems rather than only known failure conditions. OpenTelemetry: Observability Primer

2. The core observability signals

2.1 Logs: what happened?

Logs record individual events. In production systems, they should be structured so that software can reliably search, validate and correlate them.

A weak log message provides little diagnostic value:

Transaction failed

A structured event is significantly more useful:

{
 "timestamp": "2026-09-09T10:42:17.381Z",
 "level": "error",
 "event": "external_operation.timeout",
 "service": "transaction-worker",
 "transactionId": "TX-18472",
 "traceId": "4bf92f3577b34da6a3ce929d0e0e4736",
 "spanId": "00f067aa0ba902b7",
 "messageId": "MSG-8125",
 "operation": "refund",
 "attempt": 2,
 "timeoutMs": 5000,
 "durationMs": 5008,
 "outcome": "unknown_external_state"
}

This event explains which transaction was affected, which execution produced the error, where it occurred and how the system classified the outcome.

A structured logging standard should define:

Canonical event names
Common field names and units
Business and execution identifiers
Error classifications
State-transition fields
Data-retention requirements
Privacy and redaction rules

Sensitive information should not be placed in telemetry simply because it might be useful during an investigation. Credentials, tokens, personal information and complete transaction payloads require explicit handling policies.

2.2 Metrics: how often and how much?

Metrics aggregate system behaviour over time. They reveal trends, rates, distributions and changing conditions.

Infrastructure metrics commonly include:

CPU and memory utilization
Request rate and error rate
Response-time percentiles
Database query latency
Connection-pool saturation
Queue depth
Age of the oldest queued message
Worker restarts
Dead-letter queue size

These signals describe the platform, but they do not necessarily describe the customer outcome.

A system can have normal CPU utilization while hundreds of transactions remain stuck. Business-oriented metrics are therefore equally important:

transactions_requested_total
transactions_completed_total
transactions_failed_total
transactions_unknown_total
transaction_processing_duration_seconds
transaction_retries_total
duplicate_messages_detected_total
idempotency_conflicts_total
reconciliation_mismatches_total

The most valuable metrics connect technical behaviour to business flow. For example:

Requested: 10,000
 ├── Completed: 9,650
 ├── Pending: 300
 └── Failed: 50

This funnel reveals a problem that a server-health dashboard could easily miss.

Metrics should use bounded dimensions such as operation, region, outcome or dependency category. Unique transaction, message and trace identifiers should not become metric labels because uncontrolled cardinality can make the monitoring platform expensive or unstable.

2.3 Traces: where did it happen?

A distributed trace represents the journey of one logical operation through multiple components. Each individual operation inside the trace is represented by a span.

Trace = the complete journey
Span = one operation within that journey

Consider a transaction processed asynchronously:

POST /transaction 45 ms
├── Database insert 8 ms
└── Publish message 11 ms
 ⋮
 ⋮ asynchronous queue delay
 ⋮
Background worker 5.3 s
├── Database read 7 ms
├── External service call 5.1 s
└── Database update 12 ms

The trace immediately identifies the external call as the dominant contributor to latency.

Without tracing, an engineer may have to search API logs, identify a message, locate the responsible worker, correlate timestamps and then search external-integration logs. At scale, requests from many customers are interleaved, retries create additional executions and differences between system clocks introduce ambiguity.

Tracing replaces this manual reconstruction with a causal graph.

The W3C Trace Context standard defines interoperable HTTP headers for propagating trace identity between participating systems. Its traceparent field identifies the trace and the calling operation, allowing downstream services to continue the same distributed trace. W3C Trace Context Recommendation

3. Correlation requires more than one identifier

Distributed systems commonly contain several identifiers with distinct purposes:

Identifier	Question answered
Business ID	Which customer transaction is this?
Trace ID	Which distributed execution is this?
Span ID	Which individual operation is this?
Message ID	Which transport delivery is this?
External request ID	Which operation does the external system recognize?

These identifiers should not be treated as interchangeable.

A business transaction may produce multiple traces across its lifetime: initial submission, automatic retry, manual intervention and later reconciliation. The business identifier connects the complete history, while each trace describes a particular execution.

A useful design persists durable business and external identifiers while propagating trace context between services. Relevant identifiers should appear together in structured logs so that engineers can move between business history, traces and detailed events.

4. Observability across asynchronous boundaries

Synchronous trace propagation is relatively direct:

Service A ── HTTP trace context ──► Service B

Asynchronous communication introduces a break in time and execution:

API ──► Message broker ── waits ──► Background worker

The original request may have completed before the worker starts. To preserve causality, trace context must be injected into message metadata by the producer and extracted by the consumer.

Producer
 ├── Create producer span
 ├── Attach trace context to message metadata
 └── Publish message
 │
 ▼
 Queue delay
 │
 ▼
Consumer
 ├── Extract trace context
 ├── Create consumer or processing span
 └── Record message and delivery attributes

The system should measure queue delay separately from processing duration. Otherwise, a transaction that waited ten minutes before one second of processing may appear to have completed quickly.

Retries, redeliveries, batches and fan-out patterns may not fit a strict parent-and-child relationship. Trace links can represent causal relationships without incorrectly implying that one operation executed directly inside another.

5. Business-state observability

Technical telemetry becomes substantially more valuable when it is connected to the business state machine.

An asynchronous transaction might follow this simplified lifecycle:

REQUESTED
 │
 ▼
QUEUED
 │
 ▼
PROCESSING
 │
 ▼
EXTERNAL_REQUESTED
 ├────────► COMPLETED
 ├────────► FAILED
 └────────► UNKNOWN_EXTERNAL_STATE

Every transition should produce durable and observable evidence:

{
 "event": "transaction.state_transition",
 "transactionId": "TX-18472",
 "from": "EXTERNAL_REQUESTED",
 "to": "UNKNOWN_EXTERNAL_STATE",
 "reason": "response_timeout",
 "attempt": 1,
 "recordVersion": 7,
 "calculationVersion": "rules-2026-08",
 "externalRequestId": "EXT-94721",
 "traceId": "4bf92f3577b34da6a3ce929d0e0e4736"
}

This enables the platform to identify conditions such as:

Transactions remaining in PROCESSING too long
Invalid or regressive transitions
Excessive retry counts
Concurrent modification conflicts
Growing populations of unknown outcomes
Results produced by an unexpected calculation version
Differences between local and external state

This is a crucial distinction: observability should describe not only whether software components are available, but whether business operations are progressing correctly.

6. Ambiguous outcomes: when a timeout is not a failure

One of the most dangerous assumptions in distributed transaction processing is that a timeout means the operation failed.

Consider the following sequence:

Local worker External system
 │ │
 ├──── Perform operation ──────►│
 │ │ Operation succeeds
 │◄──── Success response ───────X Connection interrupted
 │
 └──── Observes timeout

The local system did not receive confirmation, but the external effect may already have occurred. Recording the transaction as failed and retrying it blindly could duplicate the effect.

A more accurate state is:

UNKNOWN_EXTERNAL_STATE

This state does not mean that the system is broken. It means that the available evidence is insufficient to assert success or failure.

Managing this condition safely requires four complementary capabilities:

Observability
 +
Idempotency
 +
Explicit state
 +
Reconciliation

Observability exposes what was attempted and why the result is ambiguous. Idempotency prevents repeated requests from creating additional effects. The state machine represents uncertainty honestly. Reconciliation later compares local state with external truth and resolves the discrepancy.

Observability alone cannot make a distributed operation atomic. Its purpose is to provide the evidence needed for correctness and recovery mechanisms to work safely.

7. Reliability should be expressed as an outcome

A mature observability programme defines reliability in terms that users and business stakeholders can understand.

Three related concepts are commonly used:

Service Level Indicator (SLI): the measured performance of the service.
Service Level Objective (SLO): the internal target for that measurement.
Service Level Agreement (SLA): a contractual commitment made to customers.

An outcome-oriented SLI might be:

Transactions reaching a known terminal state within five minutes
─────────────────────────────────────────────────────────────────
Eligible transaction requests

The associated SLO could be:

At least 99.0% of eligible transactions will reach a known terminal state within five minutes over a rolling 28-day period.

This objective is more meaningful than stating that CPU utilization should remain below a threshold. Infrastructure metrics can help explain an SLO failure, but they are not the customer outcome.

Google’s SRE guidance emphasizes that SLOs should be user-focused and supported by objective measurements and error budgets. Google SRE: The Art of SLOs

A complete SLO definition must specify:

The eligible population
The event that starts measurement
What constitutes a good outcome
The permitted duration
The evaluation window
Any exclusions
How ambiguous outcomes are classified

8. Alert on symptoms that require action

An alert should indicate that a human needs to act. It should not merely report that something unusual occurred.

Potentially actionable symptoms include:

Rapid consumption of the SLO error budget
Transactions remaining nonterminal beyond the expected duration
A sustained increase in unknown external outcomes
Material growth in the age of queued work
Failure of reconciliation to resolve discrepancies
Evidence that idempotency controls are no longer preventing duplicate effects

A temporary increase in CPU utilization may be useful diagnostic information, but it should not necessarily wake an engineer if customers remain unaffected.

Google’s monitoring guidance distinguishes symptoms from causes and identifies latency, traffic, errors and saturation as four fundamental signals for user-facing services. It also recommends keeping paging rules actionable and low-noise. Google SRE: Monitoring Distributed Systems

A useful alert includes:

Impact: 6.8% of transactions missed the five-minute objective
Scope: one operation category in one region
Related evidence: queue age rising; external timeouts elevated
Links: dashboard, representative traces, runbook and recent changes
Action: verify processing progress and unknown-state accumulation

This gives the responder a starting point instead of presenting an isolated threshold violation.

9. The operational investigation path

A practical incident investigation should move from aggregate impact to authoritative business truth:

Metrics
 │ Define scope and customer impact
 ▼
Traces
 │ Identify the slow or failing stage
 ▼
Structured logs
 │ Explain decisions, errors and retries
 ▼
Business-state history
 │ Establish what the platform believes
 ▼
External evidence and reconciliation
 Establish the actual outcome

A typical investigation proceeds as follows:

Confirm the customer-facing symptom and affected scope.
Examine queue progress, processing latency and external dependencies.
Select representative healthy and unhealthy traces.
Identify the stage responsible for delay or failure.
Inspect correlated logs for the relevant spans.
Review the durable state-transition history.
Resolve ambiguous outcomes through external evidence or reconciliation.
Confirm that the SLI recovers and the backlog drains.
Record instrumentation gaps and preventative improvements.

Temporal correlation alone is not proof of causality. Traces, deployment markers, state history and controlled tests should be combined before declaring a root cause.

10. A practical implementation framework

Organizations can develop observability incrementally.

Phase 1: Establish semantic consistency

Define canonical event names, field names, outcome classifications, identifiers and units. Standardization is more important than the choice of telemetry backend.

Phase 2: Instrument the critical journey

Trace one important business operation from entry to terminal state. Include database access, message publication, queue delay, worker processing and external calls.

Phase 3: Add business-state telemetry

Measure state populations, state age, transition rates, retries, duplicate suppression and unknown outcomes.

Phase 4: Define reliability objectives

Create SLIs and SLOs from customer outcomes. Document precisely what enters the calculation and what qualifies as success.

Phase 5: Build the investigation workflow

Connect dashboard panels to representative traces, traces to structured logs and telemetry to durable business-state records.

Phase 6: Test failure and recovery

Deliberately exercise duplicate delivery, worker interruption, concurrent processing, external timeout and lost-response scenarios. Verify both the correctness mechanism and the evidence it produces.

Phase 7: Govern cost and privacy

Establish budgets for metric cardinality, log volume, trace sampling and retention. Test redaction policies and monitor the telemetry pipeline itself for dropped data or broken propagation.

Conclusion

Observability is a property of a well-designed system, not a product installed beside it.

Logs, metrics and traces provide different views of technical behaviour. Their full value emerges when they are correlated with business identities, lifecycle state and recovery processes.

For distributed transaction systems, the most important design principle is intellectual honesty: a system must distinguish what it knows from what it merely assumes. A timeout is not automatically a failure. A successful API response is not necessarily a completed business operation. A healthy server does not guarantee a healthy transaction flow.

The objective is therefore broader than operational visibility:

Every important transaction should be explainable, its correctness should be verifiable, and an ambiguous outcome should be recoverable.

That is the difference between collecting telemetry and engineering for observability.

References

OpenTelemetry, “Observability Primer”.
OpenTelemetry, “Signals”.
World Wide Web Consortium, “Trace Context”.
Google Site Reliability Engineering, “Monitoring Distributed Systems” and “The Art of SLOs”.

Suggested SEO description:

Learn how logs, metrics, distributed traces, business-state instrumentation, SLOs, idempotency and reconciliation combine to make modern distributed systems explainable and recoverable.

Deal-Engines on Corebaseit — Design System Engineering · POS · Payments · Security

ADR-0001: Use Apache Kafka for Inter-Service Communication

Status

Date

Context

Decision

Alternatives Considered

RabbitMQ

AWS SQS + SNS

Keep Synchronous HTTP (with circuit breakers)

Consequences

Positive

Negative

Risks

Related Decisions

ADR-0002: Use AWS SQS for Asynchronous Inter-Service Communication. (from ChatGPT)

Status

Date

Context

Decision

Alternatives Considered

Apache Kafka

RabbitMQ

Synchronous HTTP

Consequences

Positive

Negative

Related Decisions

ADR-0001: Use Amazon SNS + SQS for Inter-Service Communication (Claude)

Status

Date

Context

Decision

Alternatives Considered

Apache Kafka (Confluent Cloud or MSK)

RabbitMQ

Keep Synchronous HTTP (with circuit breakers)

EventBridge

Consequences

Positive

Negative

Risks

Related Decisions

ADR 003: Choice of Message Broker for Asynchronous Job Orchestration

1. Summary

2. Context: The Database-as-a-Queue Bottleneck

3. Options Evaluated

4. Decision & Justification

5. Consequences

6. Migration Strategy

ADR 003: Message Broker Selection for Asynchronous Job Orchestration

1. Status and Meta-Data

2. Context and Problem Statement

Operational Pain Points

3. Decision Drivers

Key Differentiators

4. Options Evaluated

Option 1: Google Cloud Tasks (The Targeted Queue)

Option 2: Google Cloud Pub/Sub (The Event-Driven Broker)

5. Decision Outcome & Justification

Strategic Alignment

6. Consequences and Impact Analysis

Positive Impacts

Negative / Neutral Impacts

7. Migration Roadmap

ADR-0023: Use Apache Kafka for Inter-Service Communication

Status

Date

Context

Decision

Alternatives Considered

RabbitMQ

AWS SQS + SNS

Keep Synchronous HTTP (with circuit breakers)

Consequences

Positive

Negative

Risks

Related Decisions

ADR-NNNN: [Short Title Describing the Decision]