Revised on May 15, 2023
Data Migration is standard practice when your organization develops a new member portal or membership site, and this can be a daunting task. Your association is going to want to keep your data in order to continue tracking reports on member engagement, member retention, and the popularity of your membership benefits, especially if your organization relies heavily on data to manage its operations and achieve its mission.
However, with the right tools and strategies in place, data migration can be a smooth and successful process. This is where CiviCRM comes in – an open-source CRM software that can help organizations migrate their data effectively and efficiently. Whether your association changes its CMS, AMS, or LMS. This blog will guide you through the steps of setting your data's broad objectives and cleaning it up to prepare it for successful data migration
Here are some tips for achieving data migration success using CiviCRM
Step 1: Plan Ahead
Before going deep into the data, take a step back to think of what your association's goals are for your data. How do you want to use it? How does it benefit you? What saved reports do you want to create with it? What dynamic (liquid) fields are you going to want to create from the data? What automation will you want from the data?
Make a list of the goals you have for your data. From this point, make sure you have the data you want to reach those goals, and that you create fields to house that data in our new data management system.
Start by asking your association’s team the following questions:
1. What data do we want to migrate?
Your team will likely have inherited data from previous years and programs that are no longer relevant. Perhaps your data has gaps that need to be addressed. Perhaps there are chunks of it that need to be updated or deleted altogether. Your data may contain everything and anything! Let’s take a closer look.
The Must-Haves
The main content types your associations likely need to transfer for a member portal are your contacts, staff, members, membership types, events, activities, donations, user profiles, relationships, and partners. There may be other content types that are specific to your association’s needs.
2. Is all the data required for future work and reports?
There will be data that is not useful to your mission and reporting. Either it’s historic, not relevant, or incomplete. We encourage you to feel comfortable with letting go of the data your association doesn’t need. Avoid the what-if mentality of thinking, “We may use it one day!!” Focus on your association’s mission, how you serve your members, and the practical applications of the data.
How do we know if we will need the data in the future or not?
Use the following framework to decide what data to keep; ask yourself:
Does the data support your association’s mission?
Does this data help you better meet the needs of your membership?
Do you see practical applications for the data that you will use in the near future?
Saving a Backup of Historic Data
The data that your association doesn't need doesn’t have to be thrown out. It is reasonable to assuage your what-if fears by saving a backup in a location other than your association’s new CRM or database. This way you can access it if ever the need arises and it won’t clutter up your new database and reports.
Housekeeping
Before your association rushes to shuttle all your existing data into the new member portal, this is a welcome opportunity to clean up your data. You’ll want to make sure your association only carries over relevant and up-to-date data.
It’s like moving to a new home; you’ll want to go through the basement and ask yourself, “Do we really want to clutter up our new home with all the stuff we haven’t used in ten years?” Your task here would include deduplication, normalization, and other data cleaning task
Step 2: Getting a Closer Look at the Data Fields
Now that we know the data content types we want to move forward with, let’s look at the data fields of each content type. The fields are the titles of your columns in your data; the categories so you can make sense of the date.
1. Identify Fields to Migrate
Look at all the fields in the data. First name, last name, email, location, donation amount, relevant tags — there are plenty of important fields. There may also be granular data that your system stores and isn’t relevant to your association’s work and mission — for example, time stamps on an event from years back or the food preferences of people who passed away.
Delete the data fields you do not need. Not sure if you should delete them? Go back to the questions above: does the field support your association’s mission? Is there practical application for the data?
2. Identify the Field Type
Confirm the field type options for each field, i.e. is the field a radio button, checkboxes, text field, numeric value, etc. The person responsible for the data migration will need this data to make sure no errors occur.
Step 3: Map Out The Fields
Go through each field to identify which fields of the old data will correspond to which field on the new platform. Not all databases are created the same and it’s normal for there to be data fields from one application that don’t map one-to-one to the new application. For example, data in one system that is stored as an activity may be stored as an event in another system. Your data will still be there, just stored differently.
Will our association be able to have the same reports if the data is stored differently?
All the functionality is available in your new CRM and database (assuming you are using a powerful, detailed one such as Drupal, WordPress, and CiviCRM). You just need to find the functionality. It may not be the same steps as the previous platform, but you’ll learn the ropes on the new platform quickly. For support and tips, use the CiviCRM and Drupal forums to ask for help on how to do something.
Identify any missing fields in the new platform and create those fields.
Step 4: How are you going to do the migration?
How the migration is performed depends on the platform: are you using Drupal, WordPress, or CiviCRM?
CiviCRM already has an import mechanism feature for the main content types, such as contacts, membership, activities, contributions, and participation. Drupal and WordPress do have an import mechanism but it’s not automatic. Some coding is required.
Drupal has a migration mechanism called Migrate API, which is the module you will use to migrate the data batch by batch. Custom coding will be required as well.
Wordpress has several migration plugins to migrate data from old Wordpress sites. Custom coding will be required as well if you want to migrate from different CMS.
Provide ALL your Data to the Person Who Will Perform the Migration
Your files may be big. Like, really big with tens of thousands of rows. Provide all of your data to the person performing the migration to review the data.
Providing a sample of the data isn’t enough because there will always be surprises and inconsistencies hidden in the data. It’s best to identify these issues sooner rather than later.
Double and triple check that the data files you are working with are the most up-to-date and cleaned up you can get them. Changing the contents of the data files midway through can cause delays to the point of bringing you back to the beginning of the data migration planning process.
Step 5: Test Drive the Data aka The Deep Clean
1. Preliminarily Test the Migration of ALL The Data Through The Migration Process. In the planning phase, we review the fields manually to get a sense of the data. Your data, however, may be tens or hundreds of thousands of rows and it’s impossible to go through them manually. With a migration test, the Drupal, WordPress or CiviCRM system will identify errors in the data.
Run all the data through the migration process to identify any errors in the data. Error messages and hiccups are to be expected. For example, the system will identify duplicate fields or when a piece of data doesn’t match the data type. For instance, if two organizations have been labeled as having a parent/child relationship. You’ll have the opportunity to clean up these issues and correct, add or remove data.
2. Complete a Full Data Migration as a Test to Simulate the Launch. Now that all the errors in the data have been addressed, test the migration again. We want to see all the data transfer with zero error messages.
3. Identify the Batch Size Limitation for the Data Files
Depending on the size of your data files, it is likely that the system will not allow you to migrate the data all at once. The limitation depends on file size and number of rows. First try with 100 rows of data. If this has no issue, increase to 500 rows. Keep increasing gradually until the system prompts you that the size limit has been reached.
Step 6: Perform Final Migration
When all data has been cleaned through a trial migration run, use the maximum migration batch size you identified to perform the final migration. The process will go smoothly because there will be no errors.
Final Test Drive aka Perform Quality Assurance Testing in the New Platform
The best way to test if your data migrated successfully and that the functionality has been captured is to look at your saved reports. If the results of the reports match the results in the old system, you are golden!
For good measure, spot check a few records in the data and see if they match the old data.
Step 7: Train Your Team
Train your team: Once the data migration process is complete, it is important to train your team on how to use CiviCRM effectively. This may involve providing documentation, training sessions, and ongoing support to ensure that everyone is comfortable using the new system.
By following these tips, your organizations can migrate their data to CiviCRM with confidence, knowing that they have the right tools and strategies in place to achieve data migration success. With its robust features, flexibility, and ease of use, CiviCRM can help organizations manage their data effectively and achieve their goals more efficiently.
Learn more about data migration and get helpful tools and downloads on our website.
Comments
Thanks for this. Data migration is essentially hard, and I think your title is misleading. But sure, it's great to make is as less hard as possible.