Measuring the whole data path
The slowest stage sets the pace, and it is rarely the one that gets optimised.
Estimate, then measure: source extraction impact and available run windows, network throughput, Object Storage throughput, staging capacity, target load rate, validation elapsed time, and recovery time after a failure.
The slowest stage sets the pace, and it is rarely the one that gets optimised. A fast transfer helps nothing if the source cannot produce files quickly enough, or if validation overruns the window. Find the constraint before you tune anything.
[SG: insert your own figures here.] This section badly wants concrete numbers — indicative extraction rates, object storage ingest rates, load rates, and elapsed times for a representative rehearsal at a stated scale. I have deliberately not invented them. Even ranges with the assumptions stated (“on a 25Gb link, with N parallel streams, expect X”) would be more useful to the reader than anything else in this guide, and are the sort of thing only someone who has run these projects can supply.
Replace every estimate with a measured rate during rehearsal, and treat a material variance from forecast as a trigger to revisit the method, not just the schedule.