Observability in Distributed Systems: From Technical Telemetry to Business Confidence

contact@corebaseit.com (Vincent Bevia) — Wed, 09 Sep 2026 11:00:00 +0200

A customer request in a distributed system rarely finishes inside one process. It may pass through an API, a database, a message broker, a background worker, and one or more external services before the business operation reaches a terminal state. Splitting work across those boundaries can raise capacity and isolate some faults. It also creates failures a single-process debugger does not see: an HTTP 200 that conceals a later-stranded transaction, or a timeout that is compatible both with “the remote operation failed” and with “it succeeded and the response was lost.”

This post treats observability as the evidence required to investigate those situations, including questions that were not anticipated when the system was designed. It covers structured logs, metrics, and traces; why those signals are incomplete without durable business identifiers and lifecycle state; how asynchronous boundaries break causality; why a timeout does not prove failure; and how SLOs and alerts should be tied to outcomes rather than to infrastructure health. Collecting logs and deploying dashboards is not the same problem. The system has to emit correlated telemetry, represent uncertainty honestly, and give operators a path from aggregate impact to an authoritative record of what happened.

Claims to keep separate

Confirmed by specifications and primary sources:

OpenTelemetry treats traces, metrics, and logs as complementary signals, and defines observability as the ability to investigate novel problems (“unknown unknowns”) from telemetry the system already emits (OpenTelemetry Observability Primer, Signals).
W3C Trace Context defines the traceparent HTTP header so participating systems can continue the same distributed trace. The header carries version, trace-id, parent-id (the caller’s span), and trace-flags (W3C Trace Context).
Google SRE defines SLIs, SLOs, and SLAs as distinct, and argues that reliability should be measured from the user-facing behaviour you care about, with error budgets as the control loop (Service Level Objectives).
The same SRE material distinguishes symptoms from causes, names latency, traffic, errors, and saturation as four golden signals for user-facing services, and requires pages to be urgent, actionable, and user-visible (Monitoring Distributed Systems).
Unbounded metric labels (user IDs, request IDs, trace IDs) create a new time series per value and can exhaust a metrics backend (Prometheus instrumentation).
OpenTelemetry uses span links, not parent-child, as the default correlation for many messaging patterns because a span has one parent and batching, fan-out, and ambient consumer context do not fit that tree (Traces, Messaging spans).

Interpretation (mine, not a standard):

“State” in the operating model below is durable business-lifecycle history. It is not an OpenTelemetry signal. OTel currently documents traces, metrics, logs, and baggage; profiles are still under development.
The Metrics → Traces → Logs → State sequence is an investigation heuristic, not a protocol. Signals are complementary; the order is how an on-call engineer usually narrows a problem.
UNKNOWN_EXTERNAL_STATE, the example SLI, and the seven-phase rollout are a design for asynchronous work with external side effects. They assume you can persist a state machine, the remote API accepts an idempotency key or equivalent, and some later process can read external truth. Those capabilities are not free, and not every vendor API provides them.
Figures in this post (the 10,000-transaction funnel, the 99.0% / five-minute SLO, the 6.8% alert) are illustrative. They are not measurements from a named system.

1. Monitoring answers known questions; observability supports investigation

Traditional monitoring is built around questions you already know to ask:

Is CPU utilization too high?
Is the API returning errors?
Is the database connection pool exhausted?
Is the queue growing?

Those questions matter. They are not sufficient once a request spans several services and an asynchronous worker.

An engineer then needs to ask questions like:

Why are transactions slower for one customer category?
Did an external operation succeed before the connection dropped?
Why was the same message processed twice?
Which calculation version produced a disputed result?
Where did a transaction stop progressing?

OpenTelemetry’s primer frames this gap directly: observability is the ability to ask questions about a system from the outside, including questions you did not anticipate, provided the application already emits enough telemetry. Monitoring still has a job. It watches known conditions. Observability is what you need when the failure is a combination you did not put on a dashboard.

A practical operating model is:

Metrics → Something is wrong
Traces → Where it is going wrong
Logs → What happened at that location
State → What the system believes about the business operation

The last line is the part most telemetry stacks do not give you for free. Traces reconstruct one execution. Business state reconstructs what the platform currently asserts, and what it asserted earlier, about a durable operation that may have spanned several traces.

2. The core observability signals

2.1 Logs: what happened?

Logs record individual events. In production they should be structured so software can search, validate, and correlate them.

A weak message:

Transaction failed

A structured event:

{
 "timestamp": "2026-09-09T10:42:17.381Z",
 "level": "error",
 "event": "external_operation.timeout",
 "service": "transaction-worker",
 "transactionId": "TX-18472",
 "traceId": "4bf92f3577b34da6a3ce929d0e0e4736",
 "spanId": "00f067aa0ba902b7",
 "messageId": "MSG-8125",
 "operation": "refund",
 "attempt": 2,
 "timeoutMs": 5000,
 "durationMs": 5008,
 "outcome": "unknown_external_state"
}

The identifiers in that example follow the sample values in the W3C Trace Context specification. The event names the transaction, the execution, the transport message, the attempt, and how the system classified the outcome. That last field is doing real work: unknown_external_state is not a synonym for failed.

A logging standard should define canonical event names, common field names and units, business and execution identifiers, error classifications, state-transition fields, retention, and redaction. Credentials, tokens, personal data, and complete payloads do not belong in telemetry by default. Usefulness during an incident is not a privacy policy.

Volume is the operational constraint. Structured logs are cheap to emit and expensive to keep. High-cardinality fields that belong on a log line (transaction ID, message ID) must not migrate onto metrics as labels. That split is a cost control, not a style preference.

2.2 Metrics: how often and how much?

Metrics aggregate behaviour over time: rates, distributions, saturation, and change.

Infrastructure metrics commonly include CPU and memory, request and error rates, latency percentiles, database query latency, connection-pool saturation, queue depth, age of the oldest queued message, worker restarts, and dead-letter size. Those describe the platform. They do not necessarily describe the customer outcome.

A system can show normal CPU while a population of transactions sits in a non-terminal state. Business-oriented counters close that gap:

transactions_requested_total
transactions_completed_total
transactions_failed_total
transactions_unknown_total
transaction_processing_duration_seconds
transaction_retries_total
duplicate_messages_detected_total
idempotency_conflicts_total
reconciliation_mismatches_total

The useful picture is a funnel, not a single success rate. An illustrative snapshot, not a measured workload:

Requested: 10,000
 ├── Completed: 9,650
 ├── Pending: 300
 └── Failed: 50

A server-health dashboard can look green while 300 operations have not reached a known terminal state. That is the class of problem these metrics exist to catch.

Keep metric dimensions bounded: operation, region, outcome, dependency category. Unique transaction, message, and trace identifiers belong on logs and spans. Prometheus’s instrumentation guidance is explicit: each label set is another time series, and unbounded values (user IDs, request IDs, and similar) can make the monitoring system expensive or unstable. The diagnostic power you want from those identifiers is real. It belongs in traces and structured logs, where a new value does not create a standing series.

2.3 Traces: where did it happen?

A distributed trace is the journey of one logical operation through multiple components. Each unit of work inside that journey is a span.

Trace = the complete journey
Span = one operation within that journey

An asynchronous transaction, timings illustrative:

POST /transaction 45 ms
├── Database insert 8 ms
└── Publish message 11 ms
 ⋮
 ⋮ asynchronous queue delay
 ⋮
Background worker 5.3 s
├── Database read 7 ms
├── External service call 5.1 s
└── Database update 12 ms

The trace identifies the external call as the dominant contributor to elapsed time. Without tracing, an engineer searches API logs, finds a message, locates a worker, aligns timestamps, then searches integration logs. At any serious request rate those streams are interleaved, retries create extra executions, and clock skew between hosts makes timestamp joins ambiguous. Tracing replaces that reconstruction with a graph of recorded parent-child relationships and links.

Interoperability across HTTP hops is a standards problem, not a vendor feature. W3C Trace Context’s traceparent field lets a downstream service continue the same trace. If a hop drops that header, the graph breaks even if every service is “instrumented.”

Sampling is the cost trade-off. Retaining every span for every request is often unaffordable. Sampling reduces storage and export load; the risk is dropping the traces you later need, especially rare failures whose outcome is not known when the sampling decision is made. How aggressively you sample depends on traffic, budget, and how bad a missing trace is for your incident process. It is not a universal setting.

3. Correlation requires more than one identifier

Distributed systems usually carry several identifiers with different jobs:

Identifier	Question answered
Business ID	Which customer transaction is this?
Trace ID	Which distributed execution is this?
Span ID	Which individual operation is this?
Message ID	Which transport delivery is this?
External request ID	Which operation does the external system recognize?

These are not interchangeable.

A business transaction can produce several traces over its lifetime: initial submission, automatic retry, manual intervention, later reconciliation. The business identifier joins that history. Each trace describes one execution. The external request ID is what you send when you ask the remote system what it actually did.

A useful design persists durable business and external identifiers, propagates trace context between services, and writes the relevant IDs together on structured logs so an engineer can move from a customer complaint to a trace to a specific event. In payment backends the same idea shows up as STAN/RRN correlation across hops; the identifiers differ, the joining problem does not.

4. Observability across asynchronous boundaries

Synchronous propagation is a header on the RPC:

Service A ── HTTP trace context ──► Service B

Asynchronous communication inserts a break in time and in process:

API ──► Message broker ── waits ──► Background worker

The original request may have completed before the worker starts. To preserve causality, the producer injects trace context into message metadata and the consumer extracts it.

Producer
 ├── Create producer span
 ├── Attach trace context to message metadata
 └── Publish message
 │
 ▼
 Queue delay
 │
 ▼
Consumer
 ├── Extract trace context
 ├── Create consumer or processing span
 └── Record message and delivery attributes

Measure queue delay separately from processing duration. Otherwise a transaction that waited ten minutes and then ran for one second looks fast.

Retries, redeliveries, batches, and fan-out often do not fit a single parent. OpenTelemetry’s messaging conventions default to span links for that reason: a span can have only one parent, a batch can originate in many traces, and a consumer may already have an ambient parent (for example an HTTP handler that then pulls a queue). Links record causality without pretending the consumer ran inside the producer. The waterfall becomes less pretty. The graph stays honest.

If you skip injection, you do not have one distributed trace. You have two unconnected executions and a timestamp coincidence.

5. Business-state observability

Technical telemetry becomes useful for correctness when it is joined to a durable state machine.

A simplified asynchronous lifecycle:

REQUESTED
 │
 ▼
QUEUED
 │
 ▼
PROCESSING
 │
 ▼
EXTERNAL_REQUESTED
 ├────────► COMPLETED
 ├────────► FAILED
 └────────► UNKNOWN_EXTERNAL_STATE

Every transition should leave evidence:

{
 "event": "transaction.state_transition",
 "transactionId": "TX-18472",
 "from": "EXTERNAL_REQUESTED",
 "to": "UNKNOWN_EXTERNAL_STATE",
 "reason": "response_timeout",
 "attempt": 1,
 "recordVersion": 7,
 "calculationVersion": "rules-2026-08",
 "externalRequestId": "EXT-94721",
 "traceId": "4bf92f3577b34da6a3ce929d0e0e4736"
}

That record lets you find transactions stuck in PROCESSING, illegal or regressive transitions, retry storms, optimistic-locking conflicts, a growing unknown population, results produced by an unexpected calculation version, and disagreements between local and external state.

recordVersion and calculationVersion are not telemetry decoration. Concurrent workers racing on the same business ID need a version to detect lost updates. Disputed outcomes need to know which rules produced the number. Those fields are cheap to store on the transition event and expensive to reconstruct later from logs.

This is the distinction the rest of the post depends on: component availability is not the same as “business operations are reaching a known, correct terminal state.”

6. Ambiguous outcomes: a timeout is not a failure

One of the most expensive assumptions in distributed processing is that a timeout means the operation failed.

Local worker External system
 │ │
 ├──── Perform operation ──────►│
 │ │ Operation succeeds
 │◄──── Success response ───────X Connection interrupted
 │
 └──── Observes timeout

The local system did not receive confirmation. The external effect may already have occurred. Recording the transaction as failed and retrying it blindly can duplicate that effect. In card payments this is the double-charge path: a lost authorization response followed by a retry with a new identity. The same shape appears in refunds, payouts, provisioning calls, and any other remote mutation.

A more accurate local state is:

UNKNOWN_EXTERNAL_STATE

That state does not mean the system is broken. It means the evidence is insufficient to assert success or failure.

Managing it safely takes four capabilities that do different jobs:

Observability
 +
Idempotency
 +
Explicit state
 +
Reconciliation

Observability records what was attempted and why the result is ambiguous. Idempotency makes a repeated request collapse onto the original effect, provided the remote system honours the same key. The state machine refuses to pretend it knows. Reconciliation later compares local state with external truth.

None of this makes the distributed operation atomic. Two-phase commit is a different design, with its own availability cost, and most external APIs will not join your transaction. The combination above is an eventual-consistency recovery path. It needs a queryable remote status, or an equivalent (webhooks plus idempotent application, statements, clearing files). If the vendor cannot tell you what happened, you cannot reconcile; you can only guess more slowly.

The payment-specific version of this argument is in double charging. The architectural point is general: an error is often a loss of visibility, not a result.

7. Reliability should be expressed as an outcome

Reliability is easier to argue about when it is stated in terms a user or a business owner can recognize.

Google’s SRE book keeps three terms distinct:

Service Level Indicator (SLI): a quantitative measure of the service level provided.
Service Level Objective (SLO): an internal target for that measure.
Service Level Agreement (SLA): a contractual commitment, usually with consequences.

An outcome-oriented SLI for asynchronous work might be:

Transactions reaching a known terminal state within five minutes
─────────────────────────────────────────────────────────────────
Eligible transaction requests

An associated SLO, as an example rather than a recommendation:

At least 99.0% of eligible transactions will reach a known terminal state within five minutes over a rolling 28-day period.

CPU below a threshold is not that statement. Infrastructure metrics help explain an SLO miss. They are not the customer outcome. The SRE SLO chapter and the OpenTelemetry primer both push in this direction: measure whether the service is doing what users expect, not whether a box is up.

A complete SLO definition has to specify the eligible population, the event that starts the clock, what counts as a good outcome, the allowed duration, the evaluation window, exclusions, and how ambiguous outcomes are classified.

That last item is a real trade-off, not paperwork. If UNKNOWN_EXTERNAL_STATE counts as success, the SLO hides the population that can duplicate effects. If it counts as failure, a slow but correct reconciliation burns error budget even when money or state eventually line up. A third option is to measure two SLIs: time to a known terminal state, and the residual unknown rate. The numbers you choose depend on how costly a duplicate is versus how costly a delayed confirmation is. There is no default that fits every product.

Error budgets follow from the SLO. They are a decision tool for how much change risk you can take, not a badge. If you do not have an SLO, you still have an implicit reliability target. It just is not written down, so arguments about “is this too risky to ship?” have no shared unit.

8. Alert on symptoms that require action

An alert should mean a human needs to act. It should not mean that something unusual occurred.

Google’s monitoring chapter is blunt: paging is expensive; pages should be urgent, actionable, and tied to user-visible (or imminently user-visible) harm; chasing causes instead of symptoms produces noise. Latency, traffic, errors, and saturation remain useful diagnostic signals. They are not automatically paging signals.

Symptoms that often deserve a page in this design:

Rapid consumption of the SLO error budget
Transactions remaining non-terminal beyond the expected duration
A sustained rise in unknown external outcomes
Material growth in the age of queued work
Reconciliation failing to close discrepancies
Evidence that idempotency is no longer collapsing duplicates

A brief CPU spike can be useful on a dashboard. It should not wake someone if customers are unaffected. Saturation is the usual exception: it is a leading indicator, and paging on “the queue’s oldest message is now older than the SLO window” is paging on a symptom of stalled work, not on a kernel metric.

A useful alert carries enough context to start work. Illustrative, not a recorded incident:

Impact: 6.8% of transactions missed the five-minute objective
Scope: one operation category in one region
Related evidence: queue age rising; external timeouts elevated
Links: dashboard, representative traces, runbook, recent changes
Action: verify processing progress and unknown-state accumulation

Threshold-only alerts (“error rate > 2%”) skip impact, scope, and the next action. They also tend to fire on traffic mix changes that are not failures.

9. The operational investigation path

Incident work should move from aggregate impact toward an authoritative record:

Metrics
 │ Define scope and customer impact
 ▼
Traces
 │ Identify the slow or failing stage
 ▼
Structured logs
 │ Explain decisions, errors and retries
 ▼
Business-state history
 │ Establish what the platform believes
 ▼
External evidence and reconciliation
 Establish the actual outcome

A typical sequence:

Confirm the customer-facing symptom and the affected scope.
Look at queue progress, processing latency, and external dependencies.
Select representative healthy and unhealthy traces.
Identify the stage responsible for delay or failure.
Read correlated logs for those spans.
Review the durable state-transition history.
Resolve ambiguous outcomes against external evidence.
Confirm that the SLI recovers and the backlog drains.
Record instrumentation gaps.

Timestamp alignment across hosts is not causality. Traces, deployment markers, state history, and (when needed) a controlled reproduction should be combined before you name a root cause. A deploy that coincides with a latency spike is a hypothesis. The trace that shows the new code path, or the state history that shows illegal transitions starting at that deploy, is closer to evidence.

10. A practical implementation sequence

This is a suggested order for a team that does not yet have the above, not a maturity model and not a requirement to finish one phase before emitting any telemetry. Waiting for a perfect schema delays evidence; shipping unstructured logs first creates a cleanup bill. Pick a bias and write it down.

Phase 1: Semantic consistency. Canonical event names, field names, outcome classifications, identifiers, and units. The backend (vendor A versus vendor B) matters less than whether two services call the same failure two different things.

Phase 2: One critical journey. Trace one important business operation from entry to terminal state, including database access, publish, queue delay, worker processing, and external calls. Prove propagation across the async boundary before you instrument everything.

Phase 3: Business-state telemetry. State populations, state age, transition rates, retries, duplicate suppression, unknown outcomes.

Phase 4: Reliability objectives. SLIs and SLOs from customer outcomes. Write down eligibility, success, clock start, window, and the unknown-state rule.

Phase 5: Investigation workflow. Dashboard panels to traces, traces to logs, telemetry to durable state records. If the join requires a hero query, it will not happen at 3 a.m.

Phase 6: Failure and recovery drills. Duplicate delivery, worker crash mid-flight, concurrent processors, external timeout, lost response. Verify both the correctness mechanism and the evidence it leaves. If idempotency works but leaves no trace, the next incident will look like a mystery.

Phase 7: Cost and privacy. Budgets for metric cardinality, log volume, trace sampling, and retention. Test redaction. Watch the telemetry pipeline itself for dropped spans, broken propagation, and silent discard. An unobservable observability stack is an easy way to lose the only copy of the truth.

How far you take each phase depends on scale, budget, and how expensive an unexplainable transaction is. A low-rate internal tool and a payment mutation path should not get the same spend.

Closing

Logs, metrics, and traces are different views of technical behaviour. Their value for correctness appears when they are correlated with business identities, lifecycle state, and a recovery path that can tolerate not knowing.

For systems that mutate state outside their own process, the design rule is intellectual honesty: distinguish what the system knows from what it is assuming. A timeout does not prove failure. A successful API response does not prove the business operation completed. A healthy server does not prove a healthy transaction flow.

The objective is therefore broader than operational visibility:

Every important transaction should be explainable, its correctness should be verifiable, and an ambiguous outcome should be recoverable.

That is the gap between collecting telemetry and engineering a system you can actually investigate.

References

OpenTelemetry, “Observability Primer”.
OpenTelemetry, “Signals”.
OpenTelemetry, “Traces” (span links; producer and consumer kinds).
OpenTelemetry Semantic Conventions, “Semantic Conventions for Messaging”.
World Wide Web Consortium, “Trace Context”.
Betsy Beyer, Chris Jones, Jennifer Petoff, and Niall Richard Murphy (eds.), Site Reliability Engineering (O’Reilly, 2016), “Service Level Objectives” and “Monitoring Distributed Systems”.
Prometheus, “Instrumentation” (label cardinality).

Opentelemetry on Corebaseit — Design System Engineering · POS · Payments · Security