Mechanism
Why data teams stay siloed even when they're technically integrated
...and I think it's because we've been organizing them wrong from the start. Not the reporting line—the actual work.
Most places I've seen, a data team gets spun up because someone (usually product or finance) needs a dashboard or a metric. So you hire a couple of analysts or data engineers, they live in their own repo, use their own tools, own their own infrastructure. Then when the engineering org tries to "integrate" them, they just add a dotted line to the CTO or fold them into a platform team on an org chart. But the daily work hasn't changed. The data team still isn't reviewing PRs with backend engineers. They're not sitting in on architecture meetings about the API design that'll make their lives either easier or impossible. They're not pairing on the code that generates the data they're supposed to query.
I watched this play out at my last job. We had seven data people. They were brilliant—could optimize a Spark job like nobody's business. But they'd find out about a schema change from a Slack message two days after it shipped. Not because anyone was being secretive; it just wasn't the same conversation. The backend team had their deployment process, their standards, their rigor. The data team had theirs. Both reasonable on their own. Totally separate galaxies.
The fix isn't reorganization. It's making it genuinely impossible to do data work without being entangled in how the platform gets built. Code reviews that matter. Shared on-call rotation for data pipelines so someone from backend sees what breaks. Data schema changes required the same sign-off as API changes. Takes a while to stick, but once it does, the disconnect mostly evaporates.
2 comments
Log in to comment.
I'd push back a bit here. We tried the "genuine entanglement" approach at my last place—shared on-call, code reviews across teams, the whole thing. What actually happened was the backend team started blocking data schema changes because they didn't want to own the operational risk, and the data team started writing less ambitious stuff because every change became a negotiation. We got more communication, sure, but we also got slower iteration and a lot of resentment about whose standards won.
The real problem wasn't separation; it was that we never actually defined what data infrastructure was supposed to be owned by whom. So when you force daily entanglement without clear ownership, you just get two groups arguing about who's responsible when a pipeline breaks at 2am. The data team knew their Spark jobs. Backend knew their APIs. Neither knew who was responsible for the contract between them. Shared on-call made that worse, not better—it just meant both teams were frustrated at the same time.
What worked was actually kind of boring: we hired a dedicated data infrastructure person who lived between both worlds and explicitly owned the boundary layer. Unglamorous work, but suddenly the schema changes had a single source of truth, backend engineers didn't have to care about warehouse internals, and data people could iterate without asking for permission. Might not scale past a certain size, but I'd be curious whether the places you've seen "fix" this problem actually got lucky with their staffing rather than their process.
I'd push back on the "fix" here a bit. Yes, the entanglement matters—I agree on that part. But shared on-call and forcing backend engineers into data pipeline reviews sounds like it just spreads the pain evenly instead of actually solving it.
We tried the shared on-call thing at my last place. What actually happened: backend engineers got paged at 2am because a Spark job timed out, had no idea how to debug it, and data engineers got paged for API gateway issues they couldn't touch. Everyone got worse sleep and no one got smarter about the other's domain. The real win came later when we just... made the data pipeline infrastructure actually observable in the same way production services are. Metrics, logs, alerting that didn't require domain knowledge to read. Suddenly the backend team could see *what* was wrong without needing to understand Spark. They'd file better tickets. The data team could respond faster because they weren't explaining fundamentals every time something broke.
The schema change thing though—yeah, that one stuck for us. Not because we made it bureaucratic, but because the data team started attending the actual architecture meetings and caught problems before they happened. Turns out when you're in the room discussing whether a field should be nullable, you avoid a lot of pipeline rewrites later. But that's just basic communication, not really a "system" fix.
The real issue in the post is probably simpler: data work felt optional to the core product team, so it lived in the margins. Once it wasn't optional, once a bad schema change actually blocked a release, people started caring where the conversation happened.