Portable pipeline specification
Manifest fields and schema-contract rules for restartable file-based migration.
Split the migration into fixed extract units with stable identifiers. Once produced, a unit does not change. Its manifest records what it contains and how it has been processed, so it can be validated or restarted from a known point.
| Manifest field | Control purpose |
|---|---|
| Source object and partition or watermark | Defines the exact extract scope |
| Extraction timestamp or SCN where available | Establishes consistency evidence |
| Row count, file count and checksum | Supports transfer and load reconciliation |
| Format and character set | Makes parsing and conversion explicit |
| Schema-contract version | Binds files to an approved structure |
| Landing location and encryption classification | Identifies custody and access boundary |
| Processing status and attempt identifier | Shows what completed and what can be restarted |
| Rejected-row reference | Preserves data-quality evidence and its disposition |
The schema contract defines types, timestamp handling, null semantics and the treatment of invalid data. When it changes, create a new version and decide explicitly whether old and new files can coexist. Never silently replace a definition that files still in flight depend on.