Data Reliability Series
Data reliability isn’t a new problem introduced by AI. It’s an old problem that automation makes magnitudes harder to contain.
The never-ending fight against silent errors
Anyone who has run a data pipeline knows the failure modes that cause the most pain: hard to debug because there is no evident issue, and yet executives are breathing down your neck asking why the data is wrong.
The job says it succeeded. Looking at the data, though, only half the expected partitions arrived. Or perhaps a column was renamed upstream, and the process was set up to skip missing fields. The total now leaves out values that should have been counted. With a changed join, the missing data may be entire records. In each case, the team has a result to work with and no failed job to investigate.
These aren’t dramatic outages, or broken down workflows. The dashboards still load, the data models are still looking intact, the applications are still showing information to their consumers. That’s exactly what makes them dangerous. They don’t fail loudly, they fail quietly, and quiet failures are the ones that travel the furthest before anyone notices.
Today, people are the safety net
Where automated checks miss a problem, the safety net is a person with enough scar tissue to look at a number and realize that something is off.
That instinct comes from experience, and is often hard to document. A data engineer who works with the same reports every week might expect end of quarter Monday revenue to be about 8% higher, for example, and investigate when it isn’t. An analyst may pause over a report without being able to explain immediately what looks wrong. Call it a fingertip feel, experience, or instinct. Years of working with the data give them something to go on that a documented check may miss.
Hard for humans
Even a fully staffed human checkpoint misses things sometimes. People working in data teams will recognize the situation: someone downstream questions a number three days after something broke. The team then spends another two days checking logs and asking colleagues in other teams what changed. Finding the real cause can involve a fair amount of forensic archaeology and raw log interpretation.
This is where some of the significant costs of silent data errors hide (other significant costs happen when incorrect data guides the business). Chasing down this kind of silent breakage takes time away from building strategic scale initiatives, preparing for AI, or implementing preventive measures. Time spent patching the past is time unavailable for building the future.
Even harder without
This is the part that should give every data and platform leader pause. The whole architecture we just described, with silent failures caught late by a human with tribal knowledge, is the architecture we risk unplugging as we move toward high-scale agentic systems. Where that review is removed, or fails to keep up, we put our business at risk.
Reducing human touchpoints is part of the appeal of agentic workflows. In the workflow described here, though, that person was catching errors that could impact the business. Before removing the review, there needs to be a reliable solution in its place. Otherwise, a problem they would have noticed may pass undetected and cause real harm to your business.
And when these errors finally do surface, the aftermath may have cascaded as agents can process tasks faster and in greater volume than people can manually review. A silent error that once affected a handful of reports could be used across multiple agents’ decisions and actions, non-stop, before someone discovers the damage.
The equation doesn’t balance
Finding the cause may still take someone who knows the business and remembers how the data has behaved before. If more automated work depends on the same data team, a missed error can now leave them with even more to investigate. Investments need to account for that extra work.
The question is whether investment in people will keep pace with that work. Leadership sees the potential savings from AI, but may not have worked out what it will cost to keep the new workflows reliable. When the budget reflects only the savings, the existing team is left to catch and investigate problems across more automated work, and will become a new stress and attrition cost center.
If the work of investigating errors grows and the team does not scale, something has to give. That equation doesn’t balance. This needs more investment, tooling, and attention, for data-driven organizations to thrive with AI.
So, what?
The instinct in many organizations right now is to move fast on automation without realizing it has to start with the data. Still! Treating data governance and reliability as something secondary or “a problem we have to live with”, or even something to harden later, once the agentic workflows are already live, is setting your organization and business up for failure. That ordering is exactly backwards!
Now is the moment to invest in data reliability, before automation expands the reach of silent errors already impacting or risking your pipelines. Not just because it’s best practice, but because fixing this after agents are running at scale can mean correcting both the data and the actions and their learnings with it. That work may already be growing in production, in front of customers, before anyone catches the errors.
Automation doesn’t fix a fragile data foundation, instead it puts it to the test. If the person who used to notice the cracks is no longer involved or already over-allocated, their reliability role needs to scale – with tooling, with people, with automation for them.
Related Reading