Data migration

Moving your data from one system to another so that what arrives can be shown to match what left.

Parts of a migration

A migration reaches past the copy itself. The parts below are the ones it usually touches.

Reading the source: Finding out what the old system actually holds, including the records that break the rules the schema claims to enforce.
Mapping the fields: Deciding where each field goes in the new system, and what happens to data that has no place to land.
Reconciliation before: Counting and checksumming the source so there is a known figure to compare the result against.
The cut-over plan: Working out the order of the move, how long each part takes, and when the old system is switched off.
Reconciliation after: Checking the loaded data against the figures taken from the source, down to the record where it matters.
Rollback: A tested way back to the old system if the reconciliation fails or something is found after the switch.
Handover: Leaving your own people able to run the new system and to read what the migration did.

Why migrations go wrong, and what stops them

Most migrations are planned as if the source data is clean, and it rarely is. Dates stored as text and duplicate customers are typical, as are fields that two teams filled in two different ways. Problems like these tend to surface during the load itself, when the schedule has no room left for them, which is why the source is read closely before anything is mapped.

The part that gets cut when time runs short is reconciliation, and it is the part that carries the most risk. The source is counted and measured first, which gives a figure to check the result against. After the load, the same measures are taken on the new system and the two sets of figures are compared. Where the records carry weight, the way a ledger of balances and orders does, the comparison goes down to the individual record, because a matching total can still hide errors that cancel each other out.

A cut-over can still fail after the checks pass, because some problems only show up once people start using the new system. For that reason the move is planned with a way back to the old system, tried in advance. Amos writes the reconciliation checks himself, so the person who plans the move is also the one who can prove it worked.

How to start

The brief

Say which system the data is moving from and to, and which records you cannot afford to get wrong. Mention when the old system has to be switched off.

A trial load first

Work usually begins with a trial load of a slice of your real data, so the mapping and the checks can be judged before the full move.

Who does the work

Amos Weiskopf has built software and data pipelines for more than twenty years, and he builds what he recommends.

Questions buyers ask

What is data migration?

It is the move of your data from one system to another, done so that the records in the new system can be shown to match the ones in the old. Most of the work is in reading the source and checking both sides against each other across the cut-over.

How much does data migration cost?

The price depends mainly on how much data is moving and how many records have to be reconciled one by one. It is set once your written brief has arrived.

What happens if the cut-over fails?

The move is planned with a tried way back to the old system, so a problem found during or after the switch does not leave you stranded. The rollback is prepared before the cut-over and tested, so it is ready if it is needed.

How do you know the data arrived correctly?

The source is counted and measured before anything moves, and the same measures are taken on the new system afterwards and compared. For records that carry weight, the check goes down to the individual record, because a matching total can hide offsetting errors.

How long does a data migration take?

That depends on the volume of data and the state it is in. The timetable is agreed in the brief, with the cut-over date set around when the old system can be switched off.

Will our data stay under our own control?

Yes. The migration runs on systems you control, or on copies that stay with you. A non-disclosure agreement can be signed first if you want one before sending anything sensitive.

Engagements by written brief.

Work begins from a documented problem statement, not a sales call.

Write a brief Or book a meeting