My clients regularly need to import data from sources such as Just Giving, CAF, and various other sources like Mailchimp downloads. The problem they had was that the data was a mix of contact and contributions, and some of the rows belonged to new contacts, others belonged to existing contacts. Sometimes they had a scrappy list of names and emails where the name was all in one field instead of split out. You know the sort of thing.
At the time (a few years ago) I knocked together a Drupal module to help pre-process this data so that it could be used efficiently with CiviCRM's built in import functions. I made it in Drupal because that was quickest for me to develop a solution and they used Drupal, but I've now re-written it as a native CiviCRM extension that should work for Drupal users, and the ever proud and cheerful (but wrong) Wordpress fans alike. Maybe even Joomla. ;-)
How does it help me?
It takes your spreadsheet (in CSV format), uses a mix of automated look-ups and human interaction to efficiently find the right contact for the rows, and returns a new CSV spreadsheet that now includes a CiviCRM Internal ID column at the start. The result of this is you have a new CSV spreadsheet that will feed really efficiently and accurately into CiviCRM's well-developed build-in import routines for Contacts, Contributions, Activities, and any others.
The tool groups the data received by names and emails, so for example if you have a spreadsheet with 100 records but there's only 20 different names used (such as several contributions from the same person), you'll only have to deal with 20 records.
Well I've written full documentation including a worked example, so take a look. It's also listed in the CiviCRM extensions listings, although it won't appear in the one-click-installable extensions in your own CiviCRM installations yet because I don't want to call it 'stable' until it's had a bit more real-world use. The Drupal version has been in use for a few years, but this is a re-write and a lot has changed.
Backup your database before trying any new extension. This extension does not alter any existing data in CiviCRM, so it should be safe to play with, with the exception that if you click the "Create New Contacts" button, it will do just that.
If you get on with it, drop me a happy comment below - I'll release it as stable and make it available through the automated distribution thing (the one-click install extensions page) if/when it looks like it's working well. If you find a problem, please submit an issue. If you have a feature request or suggestion pls. see wishlist.
This is just wonderful, thats all
Great idea - could not get it to upload a file in Joomla --
I can't wait to try it out.