Data migration is one of those projects, that every company has to deal with sooner or later.
No matter how good your system is, if data in this system is messy, it is going to lead to a disaster.
Your users will try to avoid using the system and who can blame them. If you try to find a piece of information and end up spending a lot of time searching for it and then trying to make sense of it, you will soon get back to Excel spread-sheets. Your relationship with customers may be ruined if you send their orders or invoices to a wrong address, for example, or if you expose wrong information via a customer community. Both of the above points usually lead to revenue decrease and a failure to analyse data properly.
If you search for data migration best practices and issues on the internet, you will find many good articles. However, data migration still remains a problem for many businesses. Most of the people I spoke with about data migration projects said that everything that could have gone wrong, went wrong. Why is data migration so complex and is there a way to make it easy? I started reviewing this subject in my presentation, ‘Data Migration Made Easy’.
Data Migration Steps
In the presentation I highlighted a number of points that are important for a successful data migration. I am going to give a brief overview of each of them in this blog post.
Finding the right stakeholders is essential for any project, including data migration. Very often, data migration is a part of a bigger project, for example, retiring a legacy system and switching to a new one or integrating a number of different systems. It is important to remember that data migration stakeholders are not necessarily the same people as stakeholders for other parts of the project. Very often they are not high level managers that know a business strategy but only see summaries of data analysis. They are usually people who work with data directly and know it’s good and bad aspects. Depending on the data that needs to be migrated, these people may work in different departments and/or in different teams of the same department. It is essential that they are available throughout all stages of the project.
Stakeholders will help you understand the data and answer your questions, such as what data needs to be migrated. Is it all data from a legacy system or just a subset for a specific time frame and tables? Where is the data stored? A different approach is required for data migrated from a legacy system and for data migrated due to an integration of two or more systems.
How is data structured? Is all the required information there? Are any serious data changes required in order to prepare it for migration? How and where should the data be stored in Salesforce? Based on the answers to these questions, you will need to create new objects and fields, add new validations or amend existing ones, map field values in the old, legacy system to those in Salesforce.
Is the data clean? I haven’t seen a system where it is. You will need to allocate time for fixing data.
You don’t want to create duplicates in your destination Salesforce org, so it is important to know whether some of the data you are planning to migrate is already there. Duplicates may exist if your company uses multiple systems to capture customer data.
How will data migration affect existing Salesforce data and its current users? You don’t want to break anything as this will have a negative impact.
Prepare Data and a Destination Org
In this step you will need to clean and format data in an appropriate way that makes it suitable for migration.
You will also need to configure Salesforce and add/amend all required objects, fields, validations, workflows, etc. and ensure that no unwanted email alerts are sent to the users or which even more important customers. Think who and how should be able to access data after the migration. If possible, automate data migration whenever you can. You are likely to require developers’ help for this. They can setup the automation, but you are the one that needs to design the logic for this automation (with the help of stakeholders). If the logic is not very good, automation will not make much sense and will create data issues.
Complete and Verify a Test Migration
Once configuration is completed, you can test data migration in a sandbox. This is a process that you are likely to repeat a number of times, depending on how many issues are identified by a test migration. It is recommended to start with a subset of data. Once you can migrate it smoothly, a full data migration into a sandbox can be completed. A full data migration may also lead to the identification of a number of issues. This often happens when you do migration from one integrated system to another. A full data migration test can be time consuming. In fact, due to the number of fixes and test repeats, it requires much more time than migration into a production org. But it is not recommended that you try and save time by skipping proper testing. If this testing is not completed, the migration can cause serious problems and you will still miss those deadlines.
Migrate and Validate Data
Based on the test migration you should have a good understanding about all required steps and resources. To make a migration to production successful, ensure you allocate enough time and resources. Don’t forget to communicate upcoming changes and their consequences to your Salesforce users. Most of them will feel frustrated if they see a lot of new data, especially if this data is not completed due to the migration process being split into stages. Salesforce users need to understand how new data (if they have access to it) will affect their work. Will any of the processes change? Are they supposed to maintain records in a different way?
Once everything is ready, complete data migration and validation. Even though you validated data in a sandbox, you need to repeat this step in production to ensure that everything was done as planned.
Common Data Migration Issues
All data migration projects are different but most of the issues for such projects are common. I have listed those that tend to happen more often:
- Poor data quality
- Insufficient tools and human resources
- Failure to translate data into a new structure and format
- Absence of data governance policies – especially important when data migration happens in stages
- Unexpected effect of existing data, validations and processes to newly migrated data
- Unexpected effect of newly migrated data to existing Salesforce data that is already used
- Lack of testing
- Unforeseen data issues and exceptions
- Communication issues
- Access and permissions problems
- Training issues – especially important when data migration happens in stages as Salesforce users may amend it in an unwanted way (or delete) when proper training is not provided.
Data Migration Tools
There are multiple data migration tools available. Some of them, such as, Data Import Wizard and Data Loader, are native Salesforce tools. Others are provided by Salesforce partners. Some are bespoke tools, designed by individual developers, for specific companies. There is no perfect tool and unless you worked with a number of them before, you cannot know in advance which tools are going to work best for your project. You need to try them and choose the ones that meet your requirements.
Experience Based Best Practices
Is there a perfect tool that will make every data migration project easy? No, depending on your project some tools may work better than the others. Often you will need to use a combination of tools and not just one.
Who should be involved into the process? It is not just Admins or Admins and Developers who should be involved with a data migration project. It is important to collaborate with people who actually work with data, understand it and know it’s good and bad traits. They will notice potential issues quicker than anyone else and they will be able to conduct proper testing.
Start early. Everyone recommends an early start, but what does it actually mean? As data migration is usually a sub-project within a bigger project, early means before the bigger project starts. You won’t be able to complete all the data operations at that stage but you can clean it, re-structure if required, identify good and bad aspects. All of this will make the whole process smoother when you come to data migration part.
Allocate enough time. Where human beings are involved, data cannot be perfect. There will always be an amount of data that follows specific rules and exceptions. Exceptions are not always easy to notice and they tend to take a lot of time to resolve.
Identify the right people and provide correct tools. Deadlines should be realistic, otherwise the team that is working on a project will start skipping some steps to try and meet those deadlines. It is not going to do any good as it will make your system messy and eliminate the whole point of data migration. Also, remember that even the right people will not be able to produce good results without the right tools.
Don’t skip testing and verify results after each migration.
Don’t leave migrated data on its own. If you don’t look after it, it will quickly become messy and all the efforts will be for nothing.
Click to download presentation slides
There is a lot to cover when it comes to data migration. In the next blog posts, I will stop in more details on how parts of data migration can be automated and what follows data migration.