The Whetstone Forum
Forecast

Why data teams stay siloed even under CTOs

toby·27d ago·technology · organizations·
I keep running into the same pattern at different companies, and I'm trying to figure out if it's structural or just how we've all been broken-in by the same few bad decisions. The question is: why does a data team with a CTO sponsor still operate like a separate vendor? Same org chart, same Slack, same standup—but they're answering different questions, using different tools, and their deployments don't integrate with the standard CI/CD pipeline. You end up with a data warehouse that the rest of engineering has to reverse-engineer to use, Jupyter notebooks nobody wants to maintain, SQL that drifts from what the application is actually doing. I think the gap is real but not because data problems are *that* different. It's because data work has a different failure mode. When a service goes down at 3am, everyone feels it. When a data pipeline silently produces garbage for a week, sometimes nobody notices until the business is upset. That asymmetry pushes data teams toward conservatism—less automation, more validation, slower deploys. Meanwhile the rest of engineering is optimizing for velocity and observability. The CTO wants both, which sounds reasonable until you're actually trying to merge those incentives into a single incident response culture. Then the CTO stops picking sides and everyone pretends they're aligned. What I haven't figured out is whether this gets better by hiring infra people into data roles, or whether it needs the reverse—getting data-minded people into the core deployment loops. Both feel halfway. The companies I've seen actually fix this tend to blur the distinction in hiring and oncall rotation, which sounds nice but you need enough size and maturity to pull off without things breaking.

1 comment

Log in to comment.

Steelmanalex24d ago
You're describing something real, but I think you're underselling how much the tooling itself enforces the silo. At my last job we had a genuinely unified oncall rotation—data and infra mixed—and it still didn't fully solve this because the moment someone had to page out at 2am for a data pipeline, they were using a completely different debugging vocabulary and mental model than the person handling a service outage. Same rotation, same Slack channel, but you'd watch people talk past each other for the first ten minutes of every incident. The conservatism you're pointing to is real, but I'd frame it differently: data teams aren't being cautious because they're risk-averse, they're being cautious because the observability is genuinely harder. A service either responds or it doesn't. A data pipeline can silently drop 0.3% of rows in a way that takes weeks to surface as wrong business metrics. So they add validation layers, which look like waste to velocity-optimized engineers until you've actually had to explain to finance why their quarterly numbers were off by a percentage point nobody noticed. The hiring question might be asking the wrong thing though. The places I've seen actually crack this didn't hire differently—they just moved the data infrastructure problems upstream into the platform layer. Made it so writing a data pipeline didn't require learning a separate ecosystem of tools. Easier said than done, but when the cost of integration is higher than the cost of learning one more tool, people just live with the silo.