One of the new features in version 2.2 of CiviCRM (in alpha release as of this posting) is a new contact import system. I'll delve into the technical details in a bit, but at a conceptual level, this new design should allow more flexibility in the import system down the road. The first hint of this in the 2.2 release is the new SQL Query data source option. This allows you to query another database that the CiviCRM database user has appropriate permissions on (on the same MySQL server) to get the source data for the import. For many applications where the data start out in another database anyway, this is much easier and faster than dumping to CSV before importing to CiviCRM.

The CSV data source is, of course, still there. But it's now implemented as a pluggable data source alongside the SQL Query option. If you're a PHP programmer, you can write your own data source plugins too (and submit them to be included in the next version of CiviCRM). Just have a look at CRM/Import/DataSource. You'll see the SQL and CSV plugin classes in there. Your class just has to present a form snippet to get any required info from the user, grab the data based on the form input, and put it into a temporary database table (no massaging of data required, you can just dump everything into varchar or text fields if you don't know what data type the fields contain). The import system takes it from there. Feel free to get in touch with me if you need help writing a data source plugin.

Down the road, I hope to implement (or see others implement) even more nice features on top of this framework. Some ideas I have are:

1. Scheduled / recurring imports. Create an import job, save it, schedule it or set it up to happen every week or month or whatever. This would get us rudimentary data sync w/ external systems.

2. Asynchronous imports. Imports could just run in the background, with a progress meter on your dashboard. But you can work on other things while they're happening.

3. Speed improvements. This new import framework has the potential to speed up importing, though we haven't really started work on that yet, so it could end up being not as significant as we think.

Technical Details

The import system in 2.1 and previous versions of CiviCRM relied on the incoming data being in a CSV file. The file was uploaded to the server, and it was read from for each step of the import process (mapping fields, de-duping, etc.). The new import system instead relies on the incoming data being in a database table for the majority of the processing. This means the SQL Query data source plugin is very simple since it just creates the temporary table based on the user-supplied query. The CSV data source plugin reads the CSV into a database table and the same processing code takes over from there. So data source plugins need only to get their data into a database table and then can let the import code deal with the rest. They don't need to worry about formatting dates, de-duping, mapping fields, or anything else for that matter. This means we should be able to offload some of the heavy lifting of getting the data ready to import to the MySQL server (rather than having PHP do it).