Mechanism
Why data teams end up isolated from engineering
I watched this happen twice at the last place. We'd hire a strong data engineer, they'd report to the CTO like everyone else, and within six months they'd be living in a different organizational universe. Same Slack, same standup, completely different problems.
The thing that cracked it open for me was realizing data teams solve a different class of problem than infrastructure or product engineering. Product engineers own a service end-to-end—they're accountable for availability, latency, correctness in the small. Data engineers own pipelines that succeed or fail on completely different axes. A data pipeline can be "correct" and still run six hours slower than last month and nobody's paging. A data warehouse can have a schema nobody else understands and that's almost fine, as long as the business questions get answered eventually. The incentives are just orthogonal. And when you put two groups with orthogonal incentives on the same org chart, they naturally drift.
I think the reporting line masks the real problem. You can put a data team lead under a CTO and it still doesn't solve the fact that they're shipping to internal customers with very different SLAs and requirements than the external product. The data team's customer is "the analyst who needs this by Friday" or "the finance team closing the month." Product engineering's customer is someone paying money or churning. That's not a hierarchy problem. It's a velocity and visibility problem. When a product service is slow, hundreds of users see it immediately. When a data pipeline is slow, three people know, one of them is annoyed, the others accept it as normal. Over time, data teams optimize for being right over being fast, because their failure modes don't have the same social friction.
The fix isn't reorganization. It's treating data infrastructure like actual infrastructure—publishing the same kinds of status pages and SLOs, requiring the same rigor for incidents and postmortems, embedding data engineers into product teams rather than keeping them as a separate function. We tried that at my last job and it was weird for about two quarters. But then the data engineers started caring about tail latency instead of just accuracy, and the product engineers started asking better questions about data shape instead of just running queries. The isolation broke down because the incentive structure finally made sense.
1 comment
Log in to comment.
I'd actually push back on this a bit. We tried exactly this at my last place—data team under infra, same SLO rigor, incident postmortems, the whole thing. Published dashboards, everything. It worked fine for about nine months and then completely fell apart when the CFO needed some custom analysis on a Friday night and our data engineer couldn't touch it because we were in a "code freeze" for a critical production incident.
The real problem wasn't that data teams lack accountability. It's that you can't actually run data and product infrastructure on the same tempo. A data warehouse schema change that takes three weeks of testing and approval is correct governance. A product deploy that takes three weeks is a catastrophe. When you force them onto the same track, one of them has to lose—and in my experience it's always the data team that gets overruled when there's pressure, which just breeds resentment.
What actually helped was treating them as genuinely separate concerns. Not isolated, but with different operating rhythms and different definitions of "good." The data team reported sideways to whoever actually used their outputs. Product eng stayed product eng. When they needed to integrate—like, actually shipping a feature that depended on a pipeline—that became a specific project with its own accountability, not "data should work like us." Messy on the org chart, but it stopped the constant passive-aggressive collision.