Observability Beyond Logs

Abstract

Logs record events emitted by software, but they do not by themselves make a Data Platform observable. A pipeline can be technically healthy while producing stale, incomplete, semantically shifted, or policy-inadmissible data. This essay develops observability as the capacity to infer internal computational and data states from external signals. It distinguishes infrastructure, execution, data, semantic, lineage, and policy observability; formalises observability through state-space models; and examines metrics, traces, profiles, invariants, and active probes. It also addresses the limits of dashboards and alerting. The philosophical issue is one of knowability: a platform must not only operate but expose enough evidence to determine what kind of state it is in and what its outputs are entitled to claim.

1. The limits of logging

A log states that a task started, read a file, wrote a table, or raised an exception. These facts are useful, but they are emitted from the perspective of components. A platform operator needs to infer system state and output trustworthiness.

Let hidden platform state at time $t$ be $S_t$, transition under inputs $U_t$ be

and observed telemetry be

Observability asks whether the sequence $O_{0:T}$ contains enough information to distinguish relevant states. Two different failures may produce the same “job succeeded” log. Conversely, a warning may be harmless if business output remains within contract.

Logs are one measurement function $h$. Observability requires a designed family of measurements.

2. Layers of observability

Infrastructure observability covers CPU, memory, storage, network, and service availability. Execution observability covers task status, duration, retries, and queueing. Data observability covers volume, freshness, distribution, schema, and quality. Semantic observability monitors meaning-related invariants and reference versions. Lineage observability reveals dependency and propagation. Policy observability records authorisation and admissibility decisions.

Represent the observability vector as

A platform can be strong in one dimension and blind in another. Traditional operations may detect resource exhaustion while missing that a source began sending provisional records under a final status. Data observability may detect drift while lacking lineage to identify affected products.

Maturity means correlating these layers rather than accumulating disconnected dashboards.

3. Signals, symptoms, and causes

Telemetry signals are not causes. A row-count drop can result from source outage, legitimate seasonality, filter change, identity failure, or policy restriction. An alert should initiate diagnosis, not declare explanation.

Let observed anomaly be $a$ and candidate causes $C=\{c_1,\dots ,c_n\}$. Diagnosis estimates

where $E$ includes traces, lineage, recent changes, source telemetry, and historical patterns. Runbooks can encode prior knowledge, but uncertainty should remain explicit.

Correlation across signals improves discrimination. A volume drop plus unchanged source emission and increased quarantine suggests validation change. A freshness breach plus queue growth suggests capacity or dependency failure. A metric shift plus reference version change suggests semantic restatement.

Observability is therefore relational. The value lies less in individual metrics than in the graph connecting them.

4. Data profiles and invariants

Data profiles summarise distributions, null rates, cardinalities, quantiles, patterns, and correlations. They create a behavioural baseline. Yet baseline deviation is neither necessary nor sufficient for failure.

Contracts should identify invariants that have semantic force. Examples include conservation, temporal ordering, referential integrity, bounded rates, monotonic counters, and expected relationships:

An invariant breach is stronger evidence than generic anomaly because it expresses a declared law of the process. Still, invariants are models and may require revision when operations change.

5. Tracing data, not only requests

Distributed tracing follows a request across services through trace and span identifiers. Data platforms need analogous tracing for batches, partitions, records, and output versions.

A data trace may attach a run identifier, source snapshot, partition interval, contract version, and publication manifest. For a record or aggregate, provenance links allow traversal across tasks.

Let trace $T_y$ for output $y$ be a subgraph

containing material computations and controls. Trace sampling may be necessary at scale, but critical outputs should support exact reconstruction.

Data tracing differs from request tracing because processing may be asynchronous, many-to-many, and temporally extended. The trace can span hours or days and include late corrections.

6. Active observability

Passive telemetry waits for production behaviour. Active probes inject controlled signals or execute synthetic queries. A canary record can test end-to-end ingestion, mapping, quality, lineage, and publication. A synthetic query can verify metric definitions and access policy.

Let probe $p$ have expected observations $E_p$. Observability test succeeds when

Probes should be clearly identifiable and isolated from business effects. They are valuable because many failures remain invisible until a particular path is exercised.

Active observability turns monitoring into experimentation. The platform does not merely watch itself; it asks controlled questions about its own capabilities.

7. Alert quality and the economics of attention

Alerts consume scarce human attention. Excessive sensitivity creates fatigue; insufficient sensitivity delays detection. Alert design is a decision problem.

Let false-positive cost be $C_{FP}$, false-negative cost $C_{FN}$, and threshold $\theta$. Choose $\theta$ to minimise

Costs vary by domain and time. A freshness breach on a critical operational feed differs from a minor distribution shift in an exploratory dataset.

Alerts should include context: affected contracts, downstream products, recent changes, likely owner, and known runbook. An alert that merely states “null rate high” transfers diagnostic labour to the recipient.

Observability succeeds when it reduces time to understanding, not when it maximises notification volume.

8. Observability of meaning

Semantic observability is the least mature layer. It seeks signals that meaning or use has changed. Reference-data versions, code frequency, workflow milestones, source configuration, and domain canaries can provide evidence.

Suppose field $x$ is interpreted by mapping $m_v$. The platform should monitor both data distribution and mapping coverage:

A stable coverage rate does not prove stable meaning, but a sudden drop is actionable. Similarly, changes in the temporal relation between source events can reveal workflow redesign.

Semantic observability requires collaboration with domain owners because some state is not present in technical telemetry. A process change meeting can be as important a signal as a metric.

9. Dashboards, models, and epistemic restraint

Observability dashboards create an impression of control. Yet every dashboard is a projection of selected signals. Unknown failure modes remain outside the frame. Green indicators demonstrate only that monitored conditions passed.

A mature operations culture treats observability as fallible instrumentation. It records coverage, blind spots, and assumptions. Post-incident review should ask not only why the system failed but why available signals did not reveal the failure sooner.

This is epistemic restraint: knowing what the platform cannot currently know about itself. Observability is strongest when its limits are visible.

10. Designing an observability control plane

A platform control plane should unify contracts, lineage, execution metadata, quality results, profiles, policy decisions, and incident states. Correlation identifiers and versioned manifests connect signals. Retention should support trend analysis and forensic reconstruction without accumulating unnecessary sensitive detail.

Service health can be modelled as a vector rather than a single colour:

Consumers can define admissibility rules over $H$. A dataset may be available but stale, fresh but incomplete, complete but semantically provisional.

The control plane should expose these distinctions both to humans and machines.