Method — the zero-copy discipline, expanded.
The method
This is the long version of the manifesto on the index. It's what we hand to a prospective client before a first call, so we can agree on the vocabulary before we argue about the architecture.
Zero-copy started as a systems-programming trick — write a byte once and let every consumer read it in place, no memcpy, no intermediate buffer, no cache line ping-pong. The insight scales. Most data platforms we're called into have four to seven copies of the same row between the system that made it and the person who reads it: an extract, a stage, a warehouse table, a mart, a semantic layer, a BI cache, a spreadsheet on someone's laptop. Every hop is a place for the number to drift and for two people to end a meeting unsure which version was right.
The discipline is not buying a tool that says 'zero-copy' on the box. It is looking at a diagram of your stack and asking, of each arrow, whether the copy at its head earns its keep. Most don't. They exist because somebody, once, needed a shape the source didn't offer, and nobody has revisited it since. We revisit it.
The best pipeline is the one you didn't build.
Before we build a pipeline, we ask whether a view will do. A view over the source is cheaper than an extract, cannot drift from it, and forces the consumer to face the real schema instead of a flattering translation. It also has no orchestration, no backfill, no 3am page. Most of what teams call a pipeline is a view that got promoted for reasons nobody wrote down.
In-place isn't always right — analytical loads on a transactional source will end careers, and some contracts genuinely need a physical boundary. But the default should flip. Start with the view. Move to a copy only when a specific downstream decision cannot tolerate the read pattern, and write down which decision that was.
A view is cheaper than a pipeline and tells the truth by construction.
If a copy is unavoidable, it should happen once, at the boundary where the data enters your world, under a schema you wrote on purpose. Everything downstream — marts, models, dashboards, the CFO's spreadsheet — reads through that contract, unchanged. No renames three joins deep, no clever coalesces to paper over an upstream rename, no field called `revenue` that means something different in two tables.
The contract is the copy. When you have one boundary and one contract, you can version it, deprecate it, and explain it. When you have seven, you have a folklore. We write contracts the way a magazine writes headlines: with the reader in mind, and with an editor.
The contract is the copy. There shouldn't be another.
If a copy fails silently, you don't have a pipeline. You have a rumor. Row counts, freshness, schema drift, cost per run — if those aren't on a dashboard someone actually looks at, no amount of Airflow will save you. The alarm isn't a nice-to-have you add after the migration; it is the migration.
We wire the alarm before the machine. Every engagement leaves with a written runbook: what fires, why it fires, who owns it, what the fix looks like at 3am. If we can't describe the failure mode, we haven't finished building.
If a copy fails silently, it was never a pipeline — it was a rumor.
The dashboard is not the deliverable. Neither is the model, the warehouse, or the notebook. The deliverable is the moment a specific person — a PM, a controller, a clinician, a marketer — reads a number, believes it, and does something different than they would have done otherwise.
Everything upstream of that moment is scaffolding. We spend as much time on the reading as on the writing: how the number is framed, what it's shown next to, what happens when someone disagrees with it. When the last mile is designed, the rest of the stack — including the number of copies — gets simpler almost by accident.
The dashboard is not the deliverable. The decision is.
Coda
We reply to every email personally. A first conversation is 30 minutes and free — we'll tell you honestly whether we're the right shop.
hello@xerocopy.sh →