Handling both historical and ongoing data in integration
These are two separate exercises and should be built as two separate workflows:
- The recurring integration handles everything from cutover forward
- The historical load runs once, then is retired
Serving both from a single workflow is what produces duplicate postings and reconciliation problems later.
Decide what "historical" means for this client
| Approach | When it applies | What it requires |
|---|---|---|
| Full transactional history | The target needs complete detail, typically because the client will report or audit out of the target | Mechanically identical to the ongoing integration, run once with a date parameter covering the window. Chunk by month or quarter: volume and target API throughput are the constraints |
| Open items only | Most common for AR/AP cutover, and usually the right default | A status filter on the collector, plus a decision on how partially-paid items are represented |
| Opening balances only | The legacy system stays available for historical lookup | Summarised balances posted as of cutover; no transactional detail moves. Lowest risk and effort |
Which applies is determined by how the target system is configured and how the client expects to use the historical data, not by what's easiest to build.
Establish these before design begins
- The exact cutover date and time, including timezone, mismatches here are the usual cause of gapped or duplicated records
- Whether historical records carry their original dates or post as-of cutover
- Whether the target has open periods for the historical dates
- How records straddling the cutover are treated: created before, modified after. This needs an explicit decision, because a Modified Date filter on the ongoing workflow will pull them in whether or not you intended it
Go-live sequence
- Freeze the source
- Run the historical load
- Reconcile against source control totals
- Enable the ongoing workflow, date parameter starting at cutover
Reconcile before enabling, not after.