Overhauling Search and Reporting in CiviCRM

CiviCRM was built to be a CRM for organizational data with which users may leverage powerful components that provide a range of functionality and build complex workflows to manage constituent information, all under one system. There exist many ways in which data can come into CiviCRM, and there’s a far greater number of ways in which the data is used across the thousands of organizations that currently leverage CiviCRM.

But, once organizational data is in CiviCRM, how do organizations easily organize and act on it? Again, there are a few ways that’s done, but perhaps two of the most important are 1) search functionality and 2) reporting. This is probably fairly obvious because you have to search for the data that you wish to act on, and more than likely you’ll want to report on the impact of your actions.

CiviCRM addresses both of these through a range of search options and through CiviReport. But, let’s be honest for a second. Both search and reporting can be tough in CiviCRM. Take search for example. Where do you start? Basic search? Advanced search? Search builder? Custom searches? Or, do you just jump into the API explorer? Or, if you’re like many users, do you try to search for what you’re looking for through reports only to realize that reports, well, they just report on the data… they don’t allow you to take many other actions.

All of this is a long way of saying that CiviCRM’s search and reporting capabilities are in need of some lovin’.

Enter Search Builder

Thanks to an initiative led by WikiMedia Foundation, the CiviCRM Core Team undertook a discovery project that included research and review of various search and reporting interfaces, culminating in a recommendation to consolidate those in CiviCRM into single, unified utility. For now, we’re calling this “Search Builder”.

The recommendation is to leverage emerging developments, specifically APIv4 and Form Builder, to create a single, powerful search and reporting utility that will replace the current disjointed approach to both in CiviCRM. APIv4 will serve as the query builder for all future search, exports and reports, and a new user interface will be designed to replace the current Advanced Search and Search Builder UI’s.

Search and reporting has been an area in CiviCRM long ripe for improvement. In addition to discussion and reflecton how to do so, such as here and here, efforts have been made to start MIH campaigns to fund improvements, and extensions, such as Data Processor have been built to address many of search's shortcomings. The recommendation below includes a review of these approaches, review of WMF's specific needs, as well as a comprehensive review of competitor search interfaces.

The new search builder UI will be geared towards advanced users and will provide the ability to take actions on the data, such as creating smart groups, as well as to save configurations into custom searches and reports. These latter two items will benefit from the flexibility of Form Builder, allowing users to easily customize saved searches and reports through a drag and drop interface.

The new search builder functionality is in development starting with a first phase slated for completion by June 30, 2020. The first round of development is focused on improvements to APIv4, to implementation of a search builder interface powered by APIv4, and to smart group functionality, again powered by the APIv4 query engine.

For reference, interested users can view a summary of the new search builder and can track the progress on the first phase of development on Gitlab. Similar to Form Builder, this is a significant undertaking that stands to dramatically improve underlying functionality within CiviCRM. Community participation is vital to the success of the effort. Likewise, if you’re interested in supporting the initiative financially, please contact josh@civicrm.org.

Special thanks to WikiMedia Foundation for seeding the Search Builder overhaul initiative.