Data Reliability Series
Today, teams often rely on observability tools to find out how their systems are running. A dashboard showing good uptime and low latency may give them little reason to investigate a service that responded on time. However, they still really should examine what came through before using it for a business decision. [1]
Some failure modes go undetected
A system can look healthy on its operational metrics and still be quietly wrong. We should really separate these into two categories:
Operational failure: a job fails, a service times out, latency spikes. Operational monitoring catches this well. That’s exactly its job.
Semantic failure: the pipeline runs perfectly, every green light stays green, but the data itself that serves a person or that an agent uses to generate output is miscalculated or misrepresented in some way. A current inventory stock count, for example, may include units reserved for other customers, but the reserved portion was missed due to a column name change somewhere else. Using it, with the error undetected, as the quantity available to sell, could lead an agent or human to accept an order the business cannot fulfill.
Stale, mismatched, or misattributed data can produce misleading results. These are data quality problems, some of which data observability can help identify. [2] However, there are silent data errors that require a new layer of detection and prevention, that alerts before an innocent change hits the business, and that traditional data observability monitoring tools do not cover today.
As agents move from generating text to also taking action, they may use a result before anyone reviews it. An agent querying three systems could receive a timely response from one and proceed, even though the records across systems disagree or only part of the data was gathered. The task completed successfully; yet the decision outcome would be wrong. What still needs to be checked is whether data is missing and whether the data patterns match those that led to correct decisions in the past. The team also needs to be alerted if something seems off.
Why this gap is widening
A total in the source system may exclude cancelled orders. Copy it into another application, and that qualification may be left behind. The team responsible for the source knows what was counted; someone using the copy may have only a field name to go on. An agent can make the same mistake, without human critical thinking and experience-based issue detection, and pass its result naively to another sequence of agents. As more systems and vendors enter the workflow, keeping those definitions and change-impact available becomes harder. The transfer can succeed and pass as green, and the next user has too little context to detect an issue and will go on using the incorrect number, unknowingly.
So, what?
Observability is of course necessary, but treating it as sufficient is where enterprises get exposed. The organizations building for the agentic era need a second layer alongside observability to establish whether the output is reliable, fully end-to-end traceable, and consistent with origin of data and information elsewhere in the business. That work needs to span the systems supplying the data – all the way from the source, and not just within data platforms. Data observability, tracing, and evaluation can support production uptime and part of the pipeline health. Yet, the business however will need more than that to move into reliable and scalable automation.
What trust actually requires
The layer we need forward has to answer a harder question: “Can I trust what the system just handed to an agent?” That takes more business context alongside the information observability provides.
End-to-end visibility, not one vendor’s slice of it. To investigate a decision, a team needs to follow the data the agent used back through the systems that supplied it. That might mean checking the ERP, the warehouse, a pipeline, and the third tool nobody remembers connecting five years ago. An agent doesn’t care whose product boundary a value crossed. It just uses what it finds. Teams need to be able to trace that path and see where the data was changed.
Intelligence about changes, not just a raw state log entry. Knowing what a value is right now isn’t enough. The system needs to show what it used to look like, when it shifted, and what upstream changes might explain it. “This number changed,” “this number is 12% lower,” and “this number is 12% lower, and here’s likely why” offer three different levels of context that could help an agent or business user make the right decision instead of the wrong one.
A natural way for humans to interact with data. Not a static lineage diagram nobody opens. Something a person can ask real questions of: What do we know about this change? What do we know about this data: where it came from, who owns it, how fresh it is? Are there other copies of this floating around under a different name in another system, quietly drifting out of sync with this one?
That’s the business context we need alongside observability. Do we actually understand what this data is, where it’s been, and whether it still agrees with the information elsewhere in the business? That understanding gives us a basis for deciding whether to trust it.
Dashboards and system alerts may tell us system healths are good and give us no further reason to investigate. Business can still be impacted. We have to build reliability around what the agents are using, add more insight into where data originates from, how it impacts what workflows downstream, and why it can or cannot be trusted for the action at hand
Sources
[1] Microsoft Learn: Observability for Generative AI and agentic AI systems