Nothing survives twelve hours
Long-running migration units fail on first attempt. Design for cheap interruption, not reliable runs.
Here is a rule I would treat as close to a law: a migration unit that runs for more than about twelve hours will fail on its first attempt. Not might. Will.
I have watched it happen for every reason you would expect and several you would not. Space ran out. The network dropped. An upstream dependency failed part way through. And on one occasion the monitoring job someone had added to keep an eye on the migration took a lock that stalled the import — the instrumentation added to make the run safe was what broke it.
The specific cause does not matter, because it is different every time. What matters is the arithmetic: a long-running single unit accumulates exposure to every one of those risks, and the probability that none of them fires over a long enough window approaches zero. Design as though the run will be interrupted, because it will be.
The response is not to make the run more reliable. It is to make interruption cheap.
Modules, not runs
Break the migration into modules small enough to complete inside that window, and give every module three properties:
- Independently restartable. A failure costs you that module, not the migration. If a failure at 90% means starting again, the module is too big.
- Independently verifiable. Loaded is not the same as correct. Each module carries its own integrity check, run as part of the module rather than deferred to a validation phase at the end.
- A durable success point. Once a module is complete and verified, it is done, banked, and never revisited. This is what stops a long migration from being a long gamble.
The last property is the one people underweight. A migration built from success points has a monotonically improving position: every day, more of it is finished and stays finished. A migration built from long runs has a position that resets on every failure, which is why those projects feel like they are making no progress — because periodically they are not.
This is not a property of any particular toolset. It is a design decision you make regardless of whether you are using a managed service, a vendor tool or something you wrote yourself, and it applies to physical, logical and file-based methods alike.
Rolling catch-up: the pattern this enables
Modularity makes a genuinely useful project shape available, and it is the one I would reach for on any large historical estate.
Rather than moving everything in one window, migrate the historical data progressively over weeks. Start with the oldest data and work forward, module by module, while the source stays live and continues to accumulate new data. The gap between what is loaded and what is current narrows steadily. When that gap is small enough to be closed inside an acceptable window, you go live.
What this buys you is a cutover whose size you control. Instead of moving 35TB on the night, you move whatever has arrived since the last module — hours of data rather than years of it. The final window stops being a function of the platform’s total size and becomes a function of its daily volume, which is a much smaller and much more predictable number.
It also converts the migration from a single high-stakes event into a routine operational process with weeks of run time behind it. By cutover the team has executed the load hundreds of times, knows its real throughput, and has already met most of the failure modes under conditions where they cost nothing.
The discipline this demands is real, and it is where the pattern is usually let down. Over a period of weeks, every module must be verified as it completes — loaded, integrity-checked, and reloaded if it is corrupt. A backlog of unverified modules is not progress; it is deferred risk that will surface at the worst possible moment. Check as you go, keep the unverified count at or near zero, and treat a rising backlog as a stop condition rather than something to catch up on later.
Two practical notes. First, this pattern needs a mechanism to capture source changes during the catch-up period — a watermark, a change data capture feed or replication — and that mechanism needs designing at the start rather than bolted on when you near the end. Second, be careful with data that is not append-only: history that gets restated or corrected upstream will require modules you have already banked to be revisited, and you need to know in advance where that happens rather than discovering it in reconciliation.