Digital Transformation is Worthless Without This…
Your team has done the planning, met with your stakeholders, written requirements, produced test cases, and collaboratively designed the interface; redesigned business processes and executed an effective organizational change management plan. All is a go, everyone is happy and the business is set up for its step into the Digital Age. Then, seemingly out of nowhere, you find that the data from the old system refuses to fit neatly into your new system. Suddenly your project that has been going so well just plunged into chaos. WHY DID THIS HAPPEN?
Data migration is an often-overlooked component of a Digital Transformation effort. There can be many people involved in discovering the new business requirements or in designing a new interface, and so the data migration task itself tends to be forgotten about or delegated to one person (typically a more junior member of the team, or worse – the only technical person on the team). Many people think data migration should be one of the easier tasks to complete. It is just transferring the data from one system to another. This perceived simplicity leads many to think that data migration can be separated from the main body of tasks needed to deliver the system – or ends up as a single line item on the project plan.
This perception typically leads to scheduling the data migration near the end effort. Unfortunately, leaving the migration analysis until later or not understanding the full implications can have long reaching impacts from data integrity issues, data security issues, greater resistance for the change from users resulting in slower adoption, blown budgets and much larger investments in the transformation strategy. The organization may take a long time or never fully realize the benefits of the transformation effort, and ultimately, the result is lower value in the company due to mishandling of a primary asset.
There is no substitute for a structured plan to reduce the risk of problems occurring during a digital transformation. So, adopt and follow a best practice data migration planning and strategy approach on all systems projects to maximize quality, time to deliver, and effective communication. Taking the time to review these details as part of the design and planning will ensure that you have identified risk, understood technical details, and resourced and established realistic timelines. Most importantly, it will ensure data is considered an equal component and help determine if a complex ETL tool is required or if a simple spreadsheet will suffice. Ultimately, you will optimize a project launch that goes smoothly and meets stakeholder expectations.
The I.D.E.A.L. methodology outlined below is a good starting point to consider. I have used this on many projects I have been personally responsible for in the past several years, and refined it for my own work. Below are some key considerations. Try to use these at the onset of your next project and incorporate the best practices into your design:
Identify the business processes involved
- Ensure there is key business ownership and documented processes on how to manually create and maintain master, transactional, and historical data sets.
Determine, select, and locate data to migrate at the outset
- Know what data you are migrating, where it resides, what form it’s in and document the form it will need to take when it arrives at its destination.
- Determine what technical tools (like an ETL) are needed. Does the source or target system also have facilities to export and import data.
- Do you understand the layout of data in both systems and can you create a logical mapping between the two? Have you created a record manually in the new system and recorded all the fields and data types for example?
Extract, clean, transform, and de-duplicate the data
- All data has problems. Use data migration as an opportunity to clean up the data and test business rule adherence.
Audit and document the process
- Regulatory compliance requires you to document each stage of the data migration process and to preserve a clear audit trail of who did what to which data and when.
Leverage the movement of the data in a systematic execution through enforcement of data migration policies
- For example, restrict data migrations to overnight hours when network usage is low and won’t interfere with your project.
- Test the migrated data to ensure it’s accurate and in the expected format. Without testing and validating the migrated data, you can’t be confident in its integrity.
As you can see, several factors go into ensuring a successful data migration strategy. While not every detail was covered in this article, the first step outlined above is the enablement of that strategy. It is an important first step because it starts discussion and promotes critical thinking about an often-ignored part of a technical systems project, and it gets people engaged data-centrically. I hope this short summary helps you on your next project and I am always interested in hearing about how you have adapted these concepts and where they have succeeded.
