Data Migration Strategy and Design

Project
[BPR]

Prepared by
Binod

Revision and Signoff Sheet

Change Record
Date Author Version Change reference

Review
Name Version approved Position Date
Alastair
 Table of Contents

Change Record .................................................................................................................. 1

Table of Contents .............................................................................................................. 2
1 Introduction ................................................................................................................ 3
Purpose of the document ....................................................................................................... 3
Systems overview ................................................................................................................... 4
2 Migration Planning ..................................................................................................... 5
Responsibilities ....................................................................................................................... 5
Roles .........................................................................................Error! Bookmark not defined.
Data Types .............................................................................................................................. 6
Process ......................................................................................Error! Bookmark not defined.
Sequence of Upload................................................................................................................ 8
3 High level Strategic approach ...................................................................................... 9
Identify which data is to be migrated ..................................................................................... 9
Identify data source ................................................................................................................ 9
Identify target data system .................................................................................................... 9
Identify extraction mechanism ............................................................................................... 9
Identify transformation process ............................................................................................. 9
Identify upload/entry process ................................................................................................ 9
Define the data validation process ......................................................................................... 9
4 Migration cycle ......................................................................................................... 10
Precursors ............................................................................................................................. 10
Deployment Phase................................................................................................................ 13
5 Project Risks and assumptions ................................................................................... 14
Risks ...................................................................................................................................... 14
Assumptions .............................................................................Error! Bookmark not defined.
6 Miscellaneous ................................................................. Error! Bookmark not defined.4
 1 Introduction

Data migration is the process of obtaining data from the existing (legacy - DB2) systems, mapping this to
the new data requirements and other systems, and importing data to the new system. The data
mapping will highlight the transformations required as well as the cleansing activities required to
ensure the new system(s) is/are populated with current (and not redundant) data.

In order to achieve the above objectives we aim to:

 Have all master data (see definitions below) entered/uploaded as early as possible.
 Have templates and processes in place for migration of any planned transaction data at
go-live. These migrations should be fully tested and reconciled with the legacy system,
this will be an iterative process.
 The deployment team must have confidence that the take-on process is accurate and
able to sign the data take-on off before the system can be put into production.

Purpose of the document

The purpose of this document is to outline the broad strategy that will be applied, when migrating data
from existing legacy data stores into the new D365 system.

Various types of data will need to be migrated, dependent on what the business will require.
Systems overview

Technical/Functional
System Purpose Notes
Contact
Vendor, bank
Legacy 1 Ex: Purchase orders Ex: Alastair/Binod
and pricing
Legacy 1 Ex: Sales Orders
Legacy 1 Ex: Products
Legacy 1 Ex: Vendors
Legacy 2 Ex: Customers

Table 1: System Overview

 2 Migration Planning
Migration of data, from DB2 to D365 is a key activity of each site deployment. At a high-level, data
migration consists of the following activities:

 Analysing the data as to identify what needs to be migrated to D365

 Profiling the data present in the legacy systems to identify what data is available and/or
needs transformation
 Extracting the required data from the legacy system
 Transforming the extracted data to the form/format as required
 Cleansing the transformed data of any inconsistencies and errors – This can be also
done in the legacy system!
 Loading the cleansed data into D365.

Responsibilities
The following is a summarized table of responsibilities, these responsibilities must be allocated an owner
for each site deployment.

Activity Responsibility
Configuration: (before data can be migrated the TBA
system needs to be configured according to
requirements, and all necessary set-up data
created) – for each master data – early on.
Install and configure migration tools (DIEF, Atlas TBA
etc.)
Create migration templates Binod
Data Mapping Binod
Developing data conversion routines Binod
Master data Cleansing Binod
Master data entry Binod
Master and Transaction data import Binod
Enter if any point missed Binod

Table 2: Responsibilities
Data Types
There are a number of different ways that data makes its way into D365. and it is therefore important to
understand the differences and how each type is managed:

Configuration Data
During the setup and configuration of D365. parameters and values that will support the agreed solution
will/have be(en) defined and set. The data will also need to be transferred to multiple environments as
these environments are commissioned to ensure that all testing is carried out with consistent system
behavior and functionality.
e.g. system and module parameters

Transactional Data
Transactional data is the data that is currently in the legacy system that will be extracted, transformed to
adhere to the new validation rules and then loaded into D365..
As part of the data migration strategy must decide whether it will be only current or open transactions that
will be migrated or if historical data will be migrated as well. This is defined below in Error! Reference
source not found..

Master Data
The following section provides information on Master Data with regard to data which changes
infrequently. This table will need to be re-evaluated and adjusted for each table deployment.

Base Data
The BASE data set is the basis of the future solution, in other words before the data migration activities
start this Base set must be populated. Depending on project specific decisions this will be combined
Configuration data, Setup data and some or all of the Master data.
Agreed Migration objects
This list is for illustration only add or remove data objects with Exact object name

Data objects Legacy/Sourc To Transactional/ Migration Guideline Responsibility

e Tool data
Master
Volume

Ex: Items Master

Ex: Bill of Materials Master

Ex: BOM Versions Master

Ex: Resources Master

Ex: Calendars Master

Ex: Routes Master

Ex: Route Versions Master

Ex: Customers Master

Ex: Vendors Master

Ex: Departments Master

Ex: Sales Trade Transactional

Agreements

Ex: Purchase Trade Transactional

Agreements

Ex: Warehouse Bin Master

Locations

Ex: Sales Orders Transactional

Ex: Purchase Orders Transactional

Ex: Production Orders Transactional

Ex: Inventory Transactional

Ex: Accounts Transactional

Receivable

Ex: Accounts Payable Transactional

Process
Configuration and Setup MetaData will be copied from the Base data environment into the Data
Migration (and all other) environments periodically. The Master and Transactional data will then be
migrated into the Data Migration environment.
Once the data migration has been completed successfully, the Key Users will test, validate and verify
that they are able to sign off the data take-on on at go-live. Migration will be iterative and key users will
need to validate the data after each iteration and initiate any needed changes.

Sequence of Upload
The Master Data is copied to the Migration Instance to form the basis of the data load, each subsequent
activity must be performed in an agreed sequence, i.e. ensure that referenced data is loaded first. At a
high level the process is

Configuration data
Set UP Data
Master Data
Transactional data
Within each high level step there will be sub-steps e.g. customer records will need to be uploaded
before customer addresses can be uploaded, vendor accounts will need to be created before vendor
bank account details can be uploaded.
 3 High level Strategic approach

Identify which data is to be migrated

 General category e.g. Customers, Sales orders etc.
Identify data source
 Legacy system
 Discrete source – excel sheets etc.

Identify target data system

 D365

Identify extraction mechanism

 Existing tools/reports

Identify transformation process

 Rules based – automated

Identify upload/entry process

 Packages
 Script

Define the data validation process

 Inspection (should be scripted) – manual

 D365 reports/extracts – expected results defined
 Custom reports
 Trial processes – orders/journals/record creation. Migration test scripts.
 4 Migration cycle

Precursors
Configuration & setup data to entered or seeded during the design phase/process
Setup Data – supporting tables entered/uploaded as part of design
Cycle 1
Upload sample Master Data
Example:-
Customers
Products
Vendors
Employees
etc.

Test verify sample data

Run migration test scripts including:
Data inspection
Run reports
Transact

Review cycle 1

Validate the approach, strategy, data quality and tools used. If necessary:
Further data cleansing
Change migration approach (tool)
Modify templates
Adjust Base data

At this point an assessment will be made to determine if the results are sufficiently acceptable
that the migration can move onto cycle 2 or if cycle 1 should be repeated.
Cycle 2
Upload volume Master Data
Customers
Products
Vendors
Employees
etc.
Test/verify volume data
Run migration test scripts including:
Data inspection
Run reports
Transact
Review cycle 2
Validate the approach, strategy, data quality and tools used. If necessary:
Further data cleansing
Change migration approach (tool)
Modify templates
Adjust Master data
Cycle 3
Upload volume Master Data and representative transactional data
Customers
Products
Vendors
Employees
Sales orders
Production orders
Purchase orders
etc.

Test/verify volume data

Run migration test scripts including:
Data inspection
Run reports
Transact
Review cycle 3
Validate the approach, strategy, data quality and tools used. If necessary:
Further data cleansing
Change migration approach (tool)
Modify templates
Adjust Base data
It is intended that at this stage that the process is sufficiently refined that it can be signed of as fit for
purpose to use for the go live. If results are not acceptable further migration test cycles will be performed.
Deployment Phase
Following data migration activities are performed during deployment (prior to go-live):
1. Actual migration of data to the production (live) environment from the legacy systems.
2. Validation of the migrated data using reports. This acts as the basis of the sign-off for
completion of the data migration activities.
3. Put process in place to manage changes between migration and go-live
Migration of data (from the source or legacy systems) to the D365. production environment is the final
critical activity of data migration, and is the key to project success as well as user adoption. Data
migrated into Production has to be timed accurately so as to ensure that users see the most recent data
when the system is available to them. This activity is usually split into:
Initial migration of data into Production
The initial migration of data into production is performed to avoid overloading the systems during the
go-live weekend, and is usually done prior to the actual go-live. This data typically includes Master Data.
Final data migration into Production
The final data migration is done over the weekend prior to go-live. The final data load will consist of all
transactional data.
Sign-off
The last step in the data migration process is sign-off. The system cannot be transferred into operation if
the data take-on reports are not balancing.
 5 Project Risks
Risks
Effective data migration strikes the optimal balance between data accuracy, migration speed,
low or no downtime, and minimum costs. The following are data migration success factors
requirements and / or possible risks:
 Acceptance criteria (for the migration) need to be specified and agreed.
 Multiple streams of data migration (for multiple sites) may constrain resources.
 Some data will inevitably not be migrated – anything that is out-of-scope (and so will be
lost) will need to be clearly communicated and agreed (e.g. notes to master records)
 Multiple divisions may be using fields in different ways.
 6 Miscellaneous

Metadata and Content Data

For the purposes of data migration, metadata can be defined as the information that describes the
location of the source (database name, filename/table name, field/column name) and the characteristics
of each column, such as its length and type (character, numeric, date). Content is what is contained
within each actual occurrence of each field.
A metadata-driven migration assumes that the content reflects its description, which may not actually be
the case. For example, a source described as telephone numbers may contain only a few telephone
numbers or something else. The real telephone numbers may be stored elsewhere. Only content
analysis, profiling, and auditing can confirm the actual content. These processes must ascertain the
migration rules that should, in fact, be applied.
  Data is extracted from the Source data system(s)
 Data needs to be cleansed as much as possible at the time of extraction
 Values not consistently populated need to be cleansed/ assumed
 Keep Record Count- Total Records extracted should be recorded on ongoing basis.
 Validate the extracted data
 Maintain and Version control for changes to Data Mapping Templates


Checks and Balances:-

 Maintain Log of Data Extraction for each object.

 Control Totals on the number of Records extracted
 Sampling of extracted data for accuracy and confirmation to mapping
 Maintain screen shots of sample data for validating after completion of data load into
D365.
 Load the extract files in shared space.

Iterative Loading Process:-

 Defect Resolution until no error

 Log the error with number of records
 Adjustment of Source Data extract values
 Fix Load Errors
 Re-test errors followed by data re-load
 Re-load Data until satisfactory data is obtained

Keep Record Count

 Total Records loaded should be recorded on ongoing basis.

Preload Checks to ensure Data integrity

 Maintain List of Data Migration Program (LSMW) objects

 Source files to be read only from Shared space
 Verify number of records read and tally with input count
 Verify number of records converted into intermediate format
 Create and Run the Batch Program
 Maintain screen shots of Batch or Direct Input Program results
 Load Error report and Analysis into shared space for Analysis
Preload Validation Checks on each input data file before loading the same into D365

 Verify the File received and type of the file as per Rules
 Convert the file into Tab Delimited text type if received as Excel data
 Visual verification of the same to confirm the content is as per the appropriate template
format
 Verify and change the name of the file to match the specification in the data upload
program
 Ensure the file is placed in the appropriate directory as expected by the data upload
program

Testing

Unit testing:

All inbound and outbound directory structures are created properly with appropriate permissions and
sufficient disk space—Here I am assuming all data first comes from DB2 to Excel than move to
D365.

 Boundary conditions are satisfied− e.g. check for date fields with leap year dates
 Surrogate keys have been generated properly.
 NULL values have been populated where expected
 Rejects have occurred where expected and log for rejects is created with sufficient
details
 Error recovery methods.
 All fields are loaded with full contents− i.e. no data field is truncated while transforming
 No duplicates are loaded
 Aggregations take place in the target properly