Data Migration In Detail

No two data migration projects are ever the same. The Latest research figures* suggest:

  • resource run over by an average of 31%
  • Time by an average of 40%
  • A shocking 80% of all Data Migration projects fail.

Get guidance from experts that have decades of migration experience and some of the painful lessons, so your business doesn't end up on the wrong side of these statistics. We see classic mistakes almost every time. Here are a few examples :

  • Selecting your product before your integrator
  • Not performing any data profiling
  • Not configuring a DMZ (central location for all parties to share information) 
  • lack of client requirements and vendor responses
  • no clearly defined deliverables
  • target software system not stable during data migration effort
  • not working on live data from the outset  

The major issue with data and system migrations is that if these initial steps are not completed properly, the problems snowball and will affect almost ever major step of the process until completion.

Follow a framework - In our experience following Agile principles within an Enterprise framework model is essential to the success of data migrations of even modest size. A data migration is just too complex not to adhere to strict guidelines.

Go Agile - We also recommend adopting one of the agile development methodologies (scrum being the most common) that your organisation is hopefully familiar with in other areas of the business. The bigger the migration, the more unsuitable a Waterfall approach becomes. (Did you know that Waterfall was actually based on a misinterpretation-  Waterfall, a costly mistake? )

Perform proper Data Profiling - When you have the requirements or the desired target architecture. You need to perform a period of data profiling to establish some metrics on the quality of the data. This process will highlight any data that violates business rules. This will then give you clear direction on what you have to do to use that data for future projects. 

We will walk you through the techniques of data profiling, the tools and the steps that follow a natural progression such as data cleansing and managing this process in the wider data migration project.

Avoid the pitfalls  - How do we avoid the major pitfalls :

  • Follow a framework. The market is missing a choice of frameworks, we start with an enterprise framework and refine this down with something like PDMv2 (practical Data Migration)
  • Onboard all stakeholders. If you have resistance from a key business stakeholder this will greatly increase the chance of failure
  • Expert at every level. To achieve success you need a project manager, Business Analyst, Scrum-master, ETL specialist, infrastructure and data architects. (we can handle the entire project or offer some of these disciplines piecemeal to fill the gaps)
  • Use a collaborative space for all parties to host a bug tracking tool, project plan, documents and artifacts from the enterprise architecture. 
  • SVN and automated deployment. We source control the whole effort and have advanced control of the database. In an advanced state, all metadata, reference data and unit test will be under a database source control system such as LiquiBase or Redgate Source control. 
  • Manage Environments from a central location. This is where continuous integration comes into its own as you can easily control different versions of reference data in different environments.
  • Do not mix disciplines. A data migration might tempt you into data cleansing at the same time, but they must remain distinct projects. We generally advise cleansing your source data sets before you attempt the data migration.
  • Automation. We automate large portions of the migration flow to perfect the system. We also tie this in with JIRA, Continuous integration and ETL tools, so that when it comes to the real deal, it is often just hitting a button as you have done it 30 times before.


This is not an exhaustive list. Each business scenario will always need special procedures in place for a smooth transition. Please get in contact for an informal chat about any data migration you are embarking on.