Implementing a batch import API ...

At CiviCon SF 2010, we met the folks from Emotive LLC (chang and matthew b) who expressed an interest in helping scale CiviCRM and in specific help with building an import system that can handle large data sets. I started discussing this with chang on the forums and IRC. You can read our forum thread here: http://forum.civicrm.org/index.php/topic,13630.0.html We decided on implementing an Import API and started work on this on an svn branch. We also decided to incorporate unit tests with this feature at a very early stage. The API is composed of five functions:

• civicrm_import_table_create
• civicrm_import_table_drop
• civicrm_import_mapping_create
• civicrm_import_mapping_delete
• civicrm_import_rows

We've specified out the API function and documented them in subversion We'll be writing a set of tests from simple to complex to test and exercise the import functionality. If you are interested and have some time and desire to help write tests or implement the API please drop by the #civicrm IRC channel tomorrow. I think having a batch import is quite important to CiviCRM. This style of implementing and extending CiviCRM functionality is also a new experiment and we do want it to succeed. So please consider stepping up, stopping by and helping out :) You can track the issue in JIRA

Comments

Great stuff! Will be very helpful in data migrations. I am on hols till Tuesday but prepared to do some testing afterwards if required.