Data Migration
The Noah’s Ark of Business Transformations
An Introduction to data migration
Data is nothing but information — a collection of facts and statistics that is critical for businesses to operate.
Whilst new technologies emerge, it is of paramount importance for businesses to adapt. This brings a need for Business Transformations. As organisations across the world are embarking on such Business Transformation journeys, data migration is one of the most crucial but often the last thought about and most overlooked step.
So, what is data migration?
Data migration is the process of transferring data from one system to another, often involving, preparation and transformation of the data before loading it into the final target system. The data that is migrated covers both structured data like SQL databases and unstructured data such as documents, emails, etc. and their associated metadata. Data Migration often also involves an exercise of de-duplication and Master Data Management (MDM) as well.
Think of the process of data migration as the Noah’s ark of Business Transformations. Just like how Noah used an ark to move life during the great deluge in the Bible, Data migration is used to safely move data between systems and applications in a Business Transformation.
Strategies & Approaches
How do we go ahead with a Data Migration?
Data Migration can be achieved by incorporating one of two main strategies.
Big Bang Data Migration:
Big Bang data migration is a full transfer of all data intended for migration from Source to Target systems that is completed within a relatively short time window.
During the process, the source and target systems experience downtime and are unavailable for users and Big Bang data migration are typically executed at a time that will have the least interruption for users.
Hence, Big Bang data migration are more suited for small organisations or businesses that work with small volumes of data and does not involve any mission-critical applications that must be available 24x7.
Advantages: relatively less expensive, reduced complexity, takes much lesser time, all changes and migrations happen once
Disadvantages: a failed Big Bang data migration is expensive, and the impact is usually high, requires downtime
Trickle Data Migration
Trickle Data migration, also known as Phased or Iterative migrations as the name suggests, handles the data migration in a phased or iterative manner i.e. the entire data migration process is broken down into sub-migrations that are managed individually.
Trickle Data migration involve parallel running of the Source and Target System and migrating data in small increments, resulting in minimal to zero downtime.
This leads to increased complexity in terms of design, build and executions.
Hence Trickle Data migration are more suited for medium to big organisations which have a large volume of data and cannot afford to undergo a system downtime.
Advantages: comparatively less failures, zero to minimal downtime
Disadvantages: more expensive due to the iterative nature, takes more time to design and build, needs additional efforts and resources to keep both source and target systems running
Key Phases
The Ark wasn’t built in a single day and it had to undergo a series of activities before it could set sail. Similarly, Data migration accounts for a series of steps that needs to be performed under six phases that are critical for success.
Phase 1: Pre-migration planning
Pre-migration planning is the most important phase apart from the actual migration itself and involves several steps and activities that need to be completed before the Data Migration project is initiated.
· Step 1 — Define scope and asses the viability of the data migration project.
· Step 2 — Perform high level assessment of source and target systems.
· Step 3 — Set high level timelines and budget estimates.
· Step 4 — Finalise technology elements and environments required to support the process.
· Step 5 — Inform Business and IT teams of their involvement in the process.
· Step 6 — Determine data migration strategy and approach.
· Step 7 — Develop data standards, guidelines and relevant documentation.
· Step 8 — Confirm data governance structure to deal with issues that may be encountered
· Step 9 — Agree on status reporting cadences and sign-off policy.
Phase 2: Discovery
· Step 1 — Refine scope and volume of data to be migrated.
· Step 2 — Analyse source and target systems and their respective data models.
· Step 3 — Understand downstream and upstream impacts.
· Step 4 — Perform data profiling and prepare data quality report.
· Step 5 — Finalise cohorts of data that needs to be migrated.
· Step 6 — Define success criteria for each of the user base for all the source systems.
· Step 7 — Update migration plan and estimates based on findings.
Phase 3: Solution design
· Step 1 — Design Data Migration / Common Model and relationships to support migration.
· Step 2 — Create source to target mapping rules, business rules and data load specifications.
· Step 3 — Prepare interface design specifications to extract or load data from source to target.
· Step 4 — Develop data quality management specifications aligned to the success criteria.
· Step 5 — Formulate de-duplication and data mastering strategies.
· Step 6 — Devise Reconciliation frameworks and test strategies.
· Step 7 — Establish cutover and rollback strategies.
· Step 8 — Finalise source system decommissioning strategy and approach.
Phase 4: Build and Test
· Step 1 — Take backup of source data.
· Step 2 — Initiate manual source data cleansing and data enrichment activities as required.
· Step 3 — Create database objects based on Data Migration / Common Model.
· Step 4 — Build ingestion and data load routines based on mapping and design specifications.
· Step 3 — Implement de-duplication and data mastering.
· Step 4 — Develop reconciliation and testing routines.
· Step 5 — Perform unit and system testing of data loads.
· Step 6 — Engage in test triaging and remediation.
· Step 7 — Ensure business is consulted on all findings.
Phase 5: Execution and Validation
· Step 1 — Perform end to end test data loads as part of trial conversions and dry runs.
· Step 2 — Reconcile, test and validate data to ensure data migration quality and quantity.
· Step 3 — Engage in final cut-over and go-live activities.
· Step 4 — Complete final testing and reconciliation of data post migration.
· Step 5 — Obtain business confirmation and sign-off.
Phase 6: Monitor and Decommission
· Step 1 — Monitor target system for possible data issues for agreed timeframe.
· Step 2 — Remediation and support of potential data issues encountered.
· Step 3 — Perform data quality audit checks and engage in backup restoration if required.
· Step 4 — Decommission and shutdown legacy source systems as required.
Known Challenges and hurdles
Noah faced myriad challenges and hurdles in building the Ark and these challenges came from many a front. In the same vein Data migration comes with its own set of challenges, some of which are unique to a particular migration engagement. The following are some of critical ones which act as a Kryponite for Data migration and go for the jugular.
1. Not performing a detailed impact assessment — Performing an impact assessment is pivotal to any data migration exercise and its extremely important to understand the impacts from both technical and functional standpoints. A detailed impact assessment will also help unearth and inter-dependencies between business engagements and systems which could potentially disrupt the data migration timelines.
2. Not engaging with key stakeholders — One of the key steps is to identify all relevant stakeholders who are likely to be benefitted as well as impacted from the data migration exercise. Not identifying and liaising with these stakeholders is most likely to cause a hindrance, impact progress and cause unwanted delays.
3. Not keeping Business informed — It is of paramount importance in keeping Business Stakeholders and Product Owners informed and updated of the progress and challenges encountered. Weekly status reports and regular cadences help both Business be prepared and helps avoid any unwanted surprises if things go awry.
4. Not planning effectively — Engaging in planning is arguably the most important activity in a Data Migration exercise. Having a robust plan helps avoid most major roadblocks and challenges and ensures a relatively smooth sail.
5. No data governance and standards — Involving data governance is another critical aspect which ensures that all data standards, roles and access rights are established well before the data migration process is initiated.
6. Not selecting suitable data migration strategy — Choosing the right data migration strategy and approach that is suitable for the business, type, volume and cohort of data provides the assurance for a successful migration.
7. Not performing enough data quality checks — Data profiling activities before the actual implementation of the data migration routines helps in the identification of key data quality challenges and supports data preparation activities. Insufficient data preparation leads to poor quality data being migrated into the target solution, leading to operational hassles.
Best Practices and Rule of the thumb
There are a few golden rules that need to be followed to ensure a relatively successful data migration journey. These rules reiterate the activities that need to be performed that avoids some of the data migration challenges.
- Prioritise data migration to only data that has operational requirements and is critical for Business teams to function.
- Perform detailed data profiling activities of source data to avoid data quality challenges.
- Planning as a primordial activity and allocating realistic timelines for the different phases streamlines the process
- Assigning a dedicated team focussed on Data Migration activities is key for success.
- Provide sufficient time is spent as part post migration validation and support before the source system is decommissioned.
- Fully understand how the data in the source system is utilised by all the users of the source system.
- Understand how the process will change with the new system to ensure the data that is being migrated will be appropriated validated/enriched to support future state processes.
