← All perspectives

Perspective / Data & platform engineering

Observability around business transactions: distributed traces, data SLIs and error budgets

Infrastructure dashboards can remain healthy while a commercial workflow fails. Design observability around business transactions, using distributed traces, meaningful data SLIs and error budgets to prioritise reliability investment.

10 min readReviewed

One illuminated network connection traceable through an organised rack.
Conceptual editorial image.

Observability should explain whether important business transactions are completing correctly and where intervention is required. Server availability is part of that picture, but a responsive endpoint can still deliver stale information or acknowledge work that never reaches its intended outcome. For executives, the useful output is a prioritised account of customer and operational impact. Combining distributed traces with carefully defined service-level indicators makes that account more actionable, provided the instrumentation, denominators and evidence limitations are explicit.

The executive view

  • Define successful business outcomes before selecting infrastructure and application indicators.
  • Specify each SLI’s population, timing and evidence source, including incomplete and missing work.
  • Use observability, error budgets and response ownership to guide reliability investment proportionately.
01

Define completion from the user’s perspective

Start with one consequential workflow. In an illustrative ordering platform, a customer submits an order and expects it to become available for fulfilment. A request reaching the application is an intermediate event; an accepted order reaching the agreed downstream state is a different outcome.

Draw the workflow with its critical dependencies, expected transitions and points at which the user receives confirmation. State who can declare success and which failures require recovery. Include partial outcomes, such as an order accepted while its supporting document remains unavailable.

This exercise creates an observable transaction definition. It prevents the team from selecting convenient infrastructure measures and assuming that they represent the whole service. It also reveals responsibility gaps: if nobody owns the interval between acknowledgement and downstream readiness, a technically detailed dashboard may still leave an operational problem without an accountable decision-maker.

Use a state model with a terminal outcome and explicit exceptions. For an order, states might include accepted, awaiting downstream processing, ready for fulfilment, rejected with an authorised reason and cancelled. State which transitions are valid and how long a non-terminal state may persist. A transaction that vanishes before the first downstream span should still be visible in the source population. This is why observability needs operational records as well as tracing. Define which evidence is authoritative for customer-facing confirmation and how it will be reconciled with later outcomes.

02

Connect traces, logs and operational events

A distributed trace follows execution across service boundaries. OpenTelemetry’s context propagation guidance explains how correlated signals can retain their relationship as work crosses processes and networks. The architecture still needs a deliberate plan for which operations create spans and how asynchronous work remains traceable.

For the illustrative platform, assign a stable business transaction reference and connect it to relevant trace identifiers through controlled records. Distinguish this reference from personal customer information. A service log should make it possible to move from a failed transaction to the relevant technical execution without copying an entire payload into every event.

Document trust boundaries. Incoming trace metadata is external input, and propagated baggage may reach services outside the organisation. Establish an allowlist for attributes, remove sensitive content and define which destinations may receive context. Diagnostic convenience should not become an unrestricted data distribution mechanism.

Plan correlation through queues and batch jobs. A worker may process work long after the originating request has ended, and one batch can contain many business transactions. Preserve stable transaction references and use supported trace relationships where appropriate, rather than pretending every operation is a single uninterrupted request. A trace ID is diagnostic context, not an authentication credential or proof that two records belong to the same customer. Instrumentation should never let a caller select another tenant's records by supplying a convenient correlation value. Apply the same access boundaries to investigation tools as to the underlying business service.

03

Select indicators that expose meaningful failure

A service-level indicator is a measured description of behaviour. A service-level objective adds an agreed target and evaluation period. Google’s SRE guidance distinguishes request measures from pipeline freshness, correctness and coverage, providing a useful vocabulary for commercial workflows.

Select only indicators that support a decision. Availability might describe eligible requests completed successfully. Latency can describe the proportion completing within a threshold. Freshness measures whether information is recent enough for its use. Correctness requires evidence that the produced result is appropriate; a status code alone cannot establish it.

Treat each indicator as a small specification. Record its numerator, denominator, window, exclusions, source and owner. If cancelled requests are excluded, explain why and retain visibility of their volume. Otherwise, an apparent improvement may come from changing which failures count rather than improving the service itself.

Publish the calculation alongside the dashboard. For a freshness indicator, define the source event that determines age, the observation time and the permitted delay. For correctness, explain whether evidence comes from exhaustive reconciliation, a representative sample or a limited set of deterministic rules. These have different assurance limits. Keep invalid and unknown outcomes visible instead of silently excluding them. If an upstream dependency has no trustworthy timestamp, show the resulting measurement gap. Displaying a precise percentage without a defensible denominator can create more confidence than the instrumentation deserves.

04

Make time and completeness measurable

Distributed workflows often contain several clocks: when a business event occurred, when a service received it and when processing finished. Preserve those distinctions. Measuring from processing start can hide a long wait before the system begins work.

In the illustrative order flow, record the agreed submission time and downstream readiness time. Track transactions that remain unfinished, not only those that eventually complete. An average calculated from completed work can look healthy while a group of customers waits indefinitely.

Coverage also needs an independent reference where practical. Counting processed records against the records a worker happened to receive will not expose work that never reached the worker. Compare against the agreed upstream population or a reconciled business ledger. Document the limits of that reference, including delayed records and corrections. A completeness measure is only as strong as its explanation of what should have existed.

Account for late arrivals without making yesterday's report silently unstable. Define whether an outcome window is provisional until a cutoff and how corrections are restated. Track oldest unresolved work and cohort completion by age. A latency percentile based only on completed transactions omits those still waiting, so combine it with the unfinished population. Clock skew can also produce negative or misleading durations across systems. Prefer durations measured within a reliable clock domain where possible, and monitor cross-system timestamp assumptions. Time measurement belongs in the instrumentation contract, rather than being discovered during an incident.

05

Create an illustrative SLO contract

The following model demonstrates how an operating team could define evidence. Its thresholds would need to be chosen from actual user requirements, baseline behaviour and operating cost; they are not service promises.

IndicatorGood eventEligible population
Order readinessAccepted order reaches the defined fulfilment state within the agreed intervalAll accepted orders, with documented cancellations handled consistently
Information freshnessRequested availability data meets the agreed age limitRequests for which freshness is relevant
Processing coverageExpected record is processed or explicitly rejected with a valid reason within the windowReconciled upstream records
Correctness sampleReviewed result agrees with an independently established expected outcomeDefined, representative evaluation sample

Attach an error-budget policy explaining what the team does when reliability falls below its objective. Differentiate investigation, release restrictions and longer-term remediation. A target becomes useful governance when it changes priorities in an agreed way, rather than merely adding another number to a monthly report.

A budget policy needs both urgency and proportionality. A short sharp incident can consume a large share of the allowed failures, while a slower defect can accumulate comparable damage without crossing an instantaneous error threshold. Google's SLO alerting chapter explains the trade-offs in converting objectives into actionable alerts. Select windows and response policies appropriate to traffic and operating cover. Low-volume workflows may need event-level escalation for critical failures, because percentage alerts can be unstable. Keep the policy reviewable by the service owner and test it against realistic incident histories before relying on it.

06

Balance diagnostic detail against cost and signal quality

Telemetry has financial and operational costs. Unbounded attribute values can create large metric datasets, while exhaustive logging can overwhelm investigators and retain information unnecessarily. Decide what must be counted completely, what can be sampled and what needs a separate audit record.

For example, transaction outcome counts may need complete aggregation even when detailed traces are sampled. Preserve enough failure context to investigate rare events, but do not assume a sample will contain every important incident. Keep operational evidence distinct from legally or contractually required records, with separate retention decisions where necessary.

Alerts should identify an action and a responsible responder. Group related symptoms so one downstream failure does not generate a separate incident for every affected component. Evaluate both rapid deterioration and slower sustained degradation. Review false alarms and missed incidents using the same discipline applied to the application itself, because an unreliable alerting system consumes the capacity needed to improve reliability.

Decide which attributes belong in metrics and which belong in controlled diagnostic records. A bounded product category may be useful for aggregation; a unique order reference can create an unbounded series set if attached as a metric label. Keep detailed identifiers available through secure search or exemplars where supported. Sampling policies should preserve sufficient evidence for important failure classes without assuming every rare event will be retained. Set retention by investigation need and data sensitivity. Telemetry bills, investigation time and privacy exposure are all consequences of the instrumentation architecture, so review them together.

Test notification delivery as a separate dependency. A correct alert rule is ineffective if its contact route points to a departed employee or the responder lacks access to investigate. Exercise paging, acknowledgement, escalation and handover, including an out-of-hours case where the operating agreement requires cover. Keep a fallback route for failure of the primary notification system. Review an actual alert record from symptom to response so the service owner can see detection time, ownership and remedial action. This validates the operating chain rather than only the query that calculates an alert condition.

07

Worked example: calculate a transaction error budget

Suppose a hypothetical service accepts 100,000 eligible orders in a thirty-day window and targets 99.5 per cent readiness within its agreed interval. The allowable bad-event budget is 500 orders. If 300 orders miss the interval, the service has consumed sixty per cent of that event-based budget, leaving 200. This is not the same as five-nines uptime or a promise about minutes of outage; the objective is defined over business events.

Now suppose another 150 orders remain unresolved but are not yet past the readiness interval. They should be visible as exposure, rather than counted prematurely as either good or bad. When the deadline passes, their classification becomes determinable. If an incident creates fifty bad outcomes in an hour, compare that consumption with normal event volume and the remaining window before deciding the operational response. An average monthly dashboard can hide the speed of deterioration.

Keep commercial impact alongside the budget. One failed high-value order may warrant individual escalation even when aggregate performance remains inside the target. Conversely, a large group of low-value failures can create significant support and reputation costs. The budget helps coordinate release and reliability decisions, while the transaction ledger supports recovery and customer treatment. Neither should be used to dismiss a consequential exception simply because the aggregate percentage is still green.

08

Use evidence to choose the next reliability investment

A useful service review connects measured failure to a business decision. Describe the affected workflow, the number and type of users exposed, the uncertainty in the evidence and the proposed intervention. Compare likely improvement with engineering cost and the value of competing work.

Keep the instrumentation specification under change control. When a workflow or denominator changes, record the break in comparability. Retain the ability to distinguish a genuine service improvement from a revised definition. Review whether indicators still reflect the activities customers and operators depend on.

The practical deliverables are a transaction map, instrumentation contract, SLO document and response policy. Together they make reliability reviewable across engineering and commercial leadership. The purpose is a shorter, better-evidenced route from an operational symptom to a decision about what to fix, what to fund and what level of residual uncertainty the business understands.

Demand evidence that the observability system detects its own blind spots. Alert on a stopped ingestion pipeline, an unexplained drop in expected source events and missing synthetic checks, while avoiding circular checks that all depend on the same broken component. Conduct an incident exercise starting from a commercial symptom, such as orders failing to reach fulfilment. The team should locate affected transactions, identify the responsible boundary and explain the uncertainty within a useful period. The review should also confirm who can authorise remediation and how customers or operations will be informed of unresolved work.

Further questions

Questions this raises

What is business transaction observability?

It connects technical execution to a defined business workflow, such as an accepted order becoming ready for fulfilment. It combines diagnostic signals with evidence about completion, correctness and timing.

Are OpenTelemetry traces enough to establish an SLO?

No. Traces help diagnose execution, but an SLO also needs an indicator definition, eligible population, target, time window and operating policy. Sampling and missing events can affect the evidence.

How is a data freshness SLI different from uptime?

Uptime describes availability under a chosen definition. Freshness measures whether information is recent enough for its intended use. A system can be available while serving data that is too old to support a decision.

References

Sources & further reading

  1. OpenTelemetry: Context propagation
  2. Google Site Reliability Workbook: Implementing SLOs 2018
  3. Google Site Reliability Workbook: Alerting on SLOs

Connected capabilities

Software & CRMStrategy & digital transformation

From thinking to doing

What could this mean
for your business?

Your next chapter

What could your
business do better?

Tell us where you want to go, what’s getting in the way, or what you need to deliver. We’ll start there.

Start a conversation

Start a conversation

Tell us what
you have in mind.

Interactive preview. Your information stays in this page and is not sent.