Forecast
Data teams keep solving the wrong problem
At my last job we had a data team reporting straight to the CTO. Smart people, good tooling, completely isolated. They'd spend three months building a pipeline nobody asked for, then get frustrated when engineers didn't use it. The engineers meanwhile were writing one-off SQL scripts because waiting two weeks for a data request felt insane. Both groups thought the other was being difficult.
The issue isn't reporting lines. It's that data work feels like infrastructure to engineers—necessary but not their problem—while data people see themselves as analysts or scientists. So you get two different mental models of what "success" looks like. Engineers want latency and availability guarantees like they'd expect from any other system. Data people want to own the truth, prevent bad queries, enforce governance. Those aren't compatible goals when nobody negotiates the actual tradeoffs.
I've seen it work better when you stop treating data as a separate kingdom and just make it a normal system dependency with normal SLAs. Build it like you'd build an API: versioned, documented, with breaking-change warnings. That means data people have to think like platform engineers instead of analysts, which some of them hate. But the ones who don't? They become genuinely useful because now they're solving for what engineers actually need instead of what looks good in a metrics dashboard.
Reporting to the CTO doesn't fix this if nobody's actually forcing the conversation about what a data system owes to its users.
2 comments
Log in to comment.
We had almost exactly this setup, except our data team reported to finance and engineers reported to the VP of product. So it was worse. Three months into a warehouse rebuild that nobody knew was happening, we found out the new schema didn't have a concept of "deleted" records—just soft deletes with a flag that wasn't indexed. A query that took five seconds now took two minutes. The data team's position was "that's how the truth should be modeled." The engineers' position was "we need to ship features, not philosophize."
The SLA framing actually helped when we finally got there, but it took a small outage first. Once we had to write down "latency for standard fact lookups should be under 100ms" in an actual document, suddenly the theoretical purity stuff had a cost attached. But yeah, the real problem was that data people thought they were building a system for analysts to query whenever, and engineers thought they were buying a utility like Stripe or Datadog. Those are fundamentally different products.
The version-and-breaking-changes thing is the part most data teams resist hardest, in my experience. They want to be able to truth-correct retroactively. But infrastructure that surprises you at runtime is just bad infrastructure, no matter how correct it is theoretically.
I'd predict the "normal system dependency with SLAs" approach works for about eighteen months before it breaks under the weight of someone's actual use case. You'll nail the happy path—engineers get fast, predictable access to a well-versed schema, data team stops building cathedrals nobody enters. Then someone needs historical corrections, or a backfill that violates the SLA, or discovers the pipeline was silently dropping nulls in a way that mattered, and suddenly you're negotiating exceptions. The framework holds until it doesn't.
The stickier problem is that data systems have this weird property where "correctness" and "speed" genuinely do conflict in ways APIs don't. An API can be wrong in ways that are somebody's bug. A data pipeline can be wrong in ways that are nobody's fault for six months. So data people do actually need some authority to say no, or at least slow you down, in a way that's different from how we think about infrastructure SLAs. I've seen teams try to solve this by making the SLA part of the contract—engineers get their query answered in X seconds, but accept that it might be two days stale—and that works until stakeholders want fresh data for something that matters. Then the engineer does the one-off script anyway, and you're back where you started.