Data Migration Success: Data with a Purpose

Published
2021-10-03 12:09
Written by

Data migration is par for the course when your association builds a new member portal or membership site using CiviCRM. Your association will want to retain your data to continue tracking reports on your member engagement, member retention and the popularity of your membership benefits. This blog will walk you through the process of establishing your big-picture goals for your data and cleaning up your data for a successful data migration.

Start with the End in Mind

Being going deep into the data, take a step back to think of what your association's goals are for your data. How do you want to use it? How does it benefit you? What saved reports do you want to create with it? What dynamic (liquid) fields are you going to want to create from the data? What automations will you want from the data?

Make a list of the goals you have for your data. From this point, make sure you have the data you want to reach those goals, and that you create fields to house that data in our new data management system (more on that below).

Housekeeping
Before your association rushes to shuttle all your existing data into the new member portal, this is a welcome opportunity to clean up your data. You’ll want to make sure your association only carries over relevant and up-to-date data.

It’s like moving to a new home; you’ll want to go through the basement and ask yourself, “Do we really want to clutter up our new home with all the stuff we haven’t used in ten years?”

Step 1: Planning
Your association wants data that is useful and will produce insightful reports. We want to be systematic about reviewing the data we want to transfer to the new platform.

Start by asking your association’s team the following questions:

1. What data do we want to migrate?
Your team will likely have inherited data from previous years and programs that are no longer relevant. Perhaps your data has gaps that need to be addressed. Perhaps there are chunks of it that need to be updated or deleted altogether. Your data may contain everything and anything! Let’s take a closer look.

The Must-Haves
The main content types your associations likely need to transfer for a member portal are your contacts, staff, members, membership types, events, activities, donations, user profiles, relationships, and partners. There may be other content types that are specific to your association’s needs.

2. Is all the data required for future work and reports?
There will be data that is not useful to your mission and reporting. Either it’s historic, not relevant, or incomplete. We encourage you to feel comfortable with letting go of the data your association doesn’t need. Avoid the what-if mentality of thinking, “We may use it one day!!” Focus on your association’s mission, how you serve your members and the practical applications of the data.

How do we know if we will need the data in the future or not?
Use the following framework to decide what data to keep; ask yourself:
Does the data support your association’s mission?
Does this data help you better meet the needs for your membership?
Do you see practical applications for the data that you will use in the near future?

Saving a Backup of Historic Data
The data that your association doesn't need doesn’t have to be thrown out. It is reasonable to assuage your what-if fears by saving a backup in a location other than your association’s new CRM or database. This way you can access it if ever the need arises and it won’t clutter up your new database and reports.

Step 2: Getting a Closer Look at the Data Fields
Now that we know the data content types we want to move forward with, let’s look at the data fields of each content type. The fields are the titles of your columns in your data; the categories so you can make sense of the date.

1. Identify Fields to Migrate
Look at all the fields in the data. First name, last name, email, location, donation amount, relevant tags — there are plenty of important fields. There may also be granular data that your system stores and isn’t relevant to your association’s work and mission — for example, time stamps on an event from years back or the food preferences of people who passed away.

Delete the data fields you do not need. Not sure if you should delete them? Go back to the questions above: does the field support your association’s mission? Is there practical application for the data?

2. Identify the Field Type 
Confirm the field type options for each field, i.e. is the field a radio button, checkboxes, text field, numeric value, etc. The person responsible for the data migration will need this data to make sure no errors occur.

Step 3: Map Out The Fields
Go through each field to identify which fields of the old data will correspond to which field on the new platform. Not all databases are created the same and it’s normal for there to be data fields from one application that don’t map one-to-one to the new application. For example, data in one system that is stored as an activity may be stored as an event in another system. Your data will still be there, just stored differently.

Will our association be able to have the same reports if the data is stored differently?
All the functionality is available in your new CRM and database (assuming you are using a powerful, detailed one such as Drupal, WordPress and CiviCRM). You just need to find the functionality. It may not be the same steps as the previous platform, but you’ll learn the ropes on the new platform quickly. For support and tips, use the CiviCRM and Drupal forums to ask for help on how to do something.
Identify any missing fields in the new platform and create those fields.

Step 4: How are you going to do the migration?
How the migration is performed depends on the platform: are you using Drupal, WordPress or CiviCRM?
CiviCRM already has an import mechanism feature for the main content types, such as contacts, membership, activities, contributions, and participation.
Drupal and Wordpress do have an import mechanism but it’s not automatic. Some coding is required.
Drupal has a migration mechanism called Migrate API, which is the module you will use to migrate the data batch by batch. Custom coding will be required as well.
Wordpress has several migration plugins to migrate data from old Wordpress sites. Custom coding will be required as well if you want to migrate from different CMS.

Provide ALL your Data to the Person Who Will Perform the Migration
Your files may be big. Like, really big with tens of thousands of rows. Provide all of your data to the person performing the migration to review the data.

Providing a sample of the data isn’t enough because there will always be surprises and inconsistencies hidden in the data. It’s best to identify these issues sooner rather than later.

Double and triple check that the data files you are working with are the most up-to-date and cleaned up you can get them. Changing the contents of the data files midway through can cause delays to the point of bringing you back to the beginning of the data migration planning process.

Step 5: Test Drive the Data aka The Deep Clean
1. Preliminarily Test the Migration of ALL The Data Through The Migration Process. In the planning phase, we review the fields manually to get a sense of the data. Your data, however, may be tens or hundreds of thousands of rows and it’s impossible to go through them manually. With a migration test, the Drupal, WordPress or CiviCRM system will identify errors in the data. 

Run all the data through the migration process to identify any errors in the data. Error messages and hiccups are to be expected. For example, the system will identify duplicate fields or when a piece of data doesn’t match the data type. For instance, if two organizations have been labeled as having a parent/child relationship. You’ll have the opportunity to clean up these issues and correct, add or remove data.

2. Complete a Full Data Migration as a Test to Simulate the Launch. Now that all the errors in the data have been addressed, test the migration again. We want to see all the data transfer with zero error messages.

3. Identify the Batch Size Limitation for the Data Files
Depending on the size of your data files, it is likely that the system will not allow you to migrate the data all at once. The limitation depends on file size and number of rows. First try with 100 rows of data. If this has no issue, increase to 500 rows. Keep increasing gradually until the system prompts you that the size limit has been reached.

Step 6: Perform Final Migration
When all data has been cleaned through a trial migration run, use the maximum migration batch size you identified to perform the final migration. The process will go smoothly because there will be no errors.

Final Test Drive aka Perform Quality Assurance Testing in the New Platform
The best way to test if your data migrated successfully and that the functionality has been captured is to look at your saved reports. If the results of the reports match the results in the old system, you are golden!

For good measure, spot check a few records in the data and see if they match the old data.

Learn more about data migration and get helpful tools and downloads on our website.

Click thumbs up if you thought this blog post was useful (login to vote or to comment)