Import enhancements

A while back I did a lot of cleanup and bug fixing on the Core Import code. This clean-up facilitated some extra features to be added in a core extension - ‘civi-import’ which is in 5.55. In addition Greenpeace Germany sponsored improvements to CiviCRM to support background queuing and in 5.57 ‘civi-import’ can leverage that. I’m going to describe some of the enhancements in civi-import in this blog based on the 5.57 code base (not all are in 5.55). 

What is civi-import

civi-import is an extension that ships with core CiviCRM. When we add new functionality to core we try to do it in an extension for a few reasons. Firstly, it allows people to adopt it on their own timeframe. In some cases we intend the functionality to fully replace existing functionality (e.g we are on the cusp of making flexmailer a mandatory extension replacement of the legacy mailer code) and in others it is an optional feature (eg. civigrant). The civi-import changes are likely to eventually replace the old code - but not for some time as we work through the challenges. Secondly, moving code to extensions is just better form in terms of how code is organised. If you are on 5.55 or later you can enable Civi-import from the extensions screen.

civi-import provides 3 main areas of functionality.

• Contribution import features
• Search kit access to manage, edit and re-run imports
• Support for background processing

Contribution import features

civi-import replaces the ‘Mapping’ screen for the contribution import. The key difference is that without civi-import you need to import to an existing contact (and have some way to match to them). civi-import allows you to choose how you interact with the existing contact, allowing you to potentially create the contact and contribution via the same import. Similar options are provided for SoftCredits. In addition you can

• Choose the dedupe rule
• Specify a default value for a field to be used if the csv does not have a value for that row.
• Specify the same things for the soft credit contact.

Note that Contribution imports were the focus for a few reasons - they add complexity not in Memberships, Participants etc but not as much as Contacts and there were specific issues around the soft credit that were buggy. Extending the features to other entities is do-able albeit not funded / prioritized at the moment.

Search kit access to manage, edit and re-run imports

Civi-import adds a new menu item ‘My Imports’ to the ‘Reports’ menu.

Each import has its own search kit - 

And if the user chooses to fix the data within the UI they can re-do the validation or import from that screen

Background Processing

Background processing allows the user to set the import up & close their browser. Then can come back to see the process status by bookmarking the link, or using the links from ‘My Imports’

At the server level the key parts are 

• Install coworker (sponsored by Greenpeace Germany)
• Set the `enableBackGroundProcessing` setting to TRUE (it is not exposed in the UI but can be set via apiv4Explorer or via command line.

Further reading

Tracking issue for enhancements I’d like to or plan to get to  - you can add comments / suggestions here https://lab.civicrm.org/dev/core/-/issues/4016