Published
Friday, May 24, 2013 - 17:18
Written by

CiviCRM configuration is largely driven through the web interface and the database: if an administrator wants to add a new "report" or new "relationship type", he can accomplish this with a few clicks of the web interface. The new item is inserted into the database and immediately becomes live. This is great for web-based administration, but it's inconvenient for developers: if a developer writes a module or extension that registers something in the database, then he needs to write an installation routine to insert the item (and an uninstallation routine to delete the item). CiviCRM 4.2+ includes a better way: use the API and hook_civicrm_managed. This technique is already used in "civix" based extensions, but it also works with Drupal modules, Joomla plugins, etc.

Example Use-Case

As an example, suppose we are creating a module/extension called "fancyreports" which defines three new report classes. Each of these classes must be registered in CiviCRM's database. (Specifically, new rows must be inserted into table "civicrm_option_value".)

Cumbersome Approach: hook_civicrm_install, etc

One's first impulse is to write an installation routine which inserts the new reports in the database. For example:

function fancyreports_civicrm_install() {
  $result = civicrm_api('ReportTemplate', 'create', array(
    'version' => 3,
    'label' => 'Fancy membership report',
    'description' => 'Membership report with some cool doodads',
    'class_name' => 'CRM_Fancyreport_Form_Report_Membership',
    'report_url' => 'fancy/member',
    'component' => 'CiviMember',
  ));
  if ($result['is_error']) {
    CRM_Core_Session:setStatus(ts('Failed to register report'));
  }
  $result = civicrm_api('ReportTemplate', 'create', array(
    'version' => 3,
    'label' => 'Fancy contribution report',
    'description' => 'Contribution report with some cool doodads',
    'class_name' => 'CRM_Fancyreport_Form_Report_Contribution',
    'report_url' => 'fancy/contribute',
    'component' => 'CiviContribute',
  ));
  if ($result['is_error']) {
    CRM_Core_Session:setStatus(ts('Failed to register report'));
  }
  $result = civicrm_api('ReportTemplate', 'create', array(
    'version' => 3,
    'label' => 'Fancy event report',
    'description' => 'Event report with some cool doodads',
    'class_name' => 'CRM_Fancyreport_Form_Report_Event',
    'report_url' => 'fancy/event',
    'component' => 'CiviEvent',
  ));
  if ($result['is_error']) {
    CRM_Core_Session:setStatus(ts('Failed to register report'));
  }
}

This is serviceable, but (by itself) leaves some bugs. When an administrator uninstalls the extension, the reports are still listed in the database. So we need to define an uninstallation process, e.g.:

function fancyreports_civicrm_uninstall() {
  $getResult = civicrm_api('ReportTemplate', 'getsingle', array(
    'version' => 3,
    'name' => 'CRM_Fancyreport_Form_Report_Membership',
  ));
  if ($getResult['id']) {
    $delResult = civicrm_api('ReportTemplate', 'delete', array(
      'version' => 3,
      'id' => $getResult['id'],
    ));;
    if ($delResult['is_error']) {
      CRM_Core_Session:setStatus(ts('Failed to register report'));
    }
  }
}

That gets us closer, but still there are issues:

  • The uninstall needs to run for all three reports.
  • We've handled "hook_civicrm_install" and "hook_civicrm_uninstall" but not "hook_civicrm_enable" or "hook_civicrm_disable". We should implement those hooks, too -- when the extension is disabled, we should flag the reports as inactive; when re-enabled, we should flag the reports as active again.
  • These four hooks -- "hook_civicrm_install", "hook_civicrm_uninstall", "hook_civicrm_enable", and "hook_civicrm_disable" -- only work with CiviCRM native extensions. With Drupal modules or Joomla plugins, one must identify a different (but comparable) hook for that platform.
  • If we release an upgraded version of the module/extension, the upgrade might define additional reports, might remove defunct reports, or might rename existing reports. All of these require extra upgrade logic.
  • As we write the install/upgrade logic and fine-tune the report, we'll need to test the install/upgrade logic -- usually, that means repeatedly and manually installing/uninstalling the extension.
  • If an error arises during installation/uninstallation, then the administrator is left with a database in an inconsistent state and no simple way to recover.

Joomla and Drupal Approaches

The difficulty of using hook_civicrm_install arises because it's a procedural approach to managing the full lifecycle. Many platforms adopt a declarative approach to registering items -- an author declares what should be in the system, and the system takes care of any needed installation (or uninstallation or deactivation re-activation)  steps. For example, in Joomla, an extension author declares permissions by creating an XML file called "access.xml". In Drupal, a module author declares permissions by implementing hook_permission and setting it to return a certain list of permissions.

Unfortunately, adopting a Joomla or Drupal approach within the CiviCRM 4.x series would present a challenge -- inertia. We already have a long list of items which are managed as database resources -- report-templates, report-instances, payment-processor-types, relationship-types, activity-types, ad nauseum. Adapting each would require a lot of incremental work (coding, documentation, tests, etc) because our code assumes that these records are in the database -- aside from "stored in the database", there's no single standard, format, or entry-point that applies to all these interesting resources.

Managed Entity Approach: hook_civicrm_managed

Correction: there is a single standard that we've adopted, documented, and tested for most interesting resources -- APIv3! But APIv3 is clearly procedural. Can we use APIv3 declaratively? As of CiviCRM 4.2+, yes.

The solution is hook_civicrm_managed. Whenever the CiviCRM cache is cleared, CiviCRM will invoke the hook and perform reconciliation:

  • Invoke this hook (to build a list of entities which should be in the database)
  • Insert new entities in the database (using the API)
  • Update existing entities in the database (using the API)
  • Delete stale/unnecessary entities from the database (using the API)
  • Flag entities as active or inactive depending on their module's status (using the API and is_active property)

For our "fancyreports" example, one would declare:

function fancyreports_civicrm_managed(&$entities) {
  $entities[] = array(
    'module' => 'com.example.fancyreports',
    'name' => 'fancymember',
    'entity' => 'ReportTemplate',
    'params' => array(
      'version' => 3,
      'label' => 'Fancy membership report',
      'description' => 'Membership report with some cool doodads',
      'class_name' => 'CRM_Fancyreport_Form_Report_Membership',
      'report_url' => 'fancy/member',
      'component' => 'CiviMember',
    ),
  );
  $entities[] = array(
    'module' => 'com.example.fancyreports',
    'name' => 'fancycontribute',
    'entity' => 'ReportTemplate',
    'params' => array(
      'version' => 3,
      'label' => 'Fancy contribution report',
      'description' => 'Contribution report with some cool doodads',
      'class_name' => 'CRM_Fancyreport_Form_Report_Contribution',
      'report_url' => 'fancy/contribute',
      'component' => 'CiviContribute',
    ),
  );
  $entities[] = array(
    'module' => 'com.example.fancyreports',
    'name' => 'fancyevent',
    'entity' => 'ReportTemplate',
    'params' => array(
      'version' => 3,
      'label' => 'Fancy event report',
      'description' => 'Event report with some cool doodads',
      'class_name' => 'CRM_Fancyreport_Form_Report_Event',
      'report_url' => 'fancy/event',
      'component' => 'CiviEvent',
    ),
  );
}

The entity and params are exactly the same as normal APIv3. There are only two additions (which are needed for reconciliation):

  • module: The fully-qualified name of the module which declares the entity. (If this is a CiviCRM module, then the name looks like "org.example.fancyreports". If this is a Drupal module, then the name looks like "drupal.fancyreports".)
  • name: A locally-unique name for the entity.

Managed Entity Approach: *.mgd.php

When the civix code-generator creates a new module, it provides this glue code as part of hook_civicrm_managed:

function _fancyreports_civix_civicrm_managed(&$entities) {
  $mgdFiles = _fancyreports_civix_find_files(__DIR__, '*.mgd.php');
  foreach ($mgdFiles as $file) {
    $es = include $file;
    foreach ($es as $e) {
      if (empty($e['module'])) {
        $e['module'] = 'org.example.fancyreports';
      }
      $entities[] = $e;
    }
  }
}

Any files with the extension "*.mgd.php" will be automatically added to hook_civicrm_managed. Consequently, you don't need to put all declarations in one file -- you can find a more convenient place to put them. For example, when creating a report class, it's handy to put the *.php and the *.mgd.php next to each other:

Limitations

The "Managed Entity" feature is convenient for several use-cases but can't do everything -- sometimes it may be better to use the more flexible (but more cumbersome) install/uninstall hooks. A few key limitations with managed entities:

  • The reconciliation process is not designed for high-performance -- if there were a couple hundred entities, then reconciliation could be slow. However, reconciliation only runs rarely -- when CiviCRM is upgraded, when new modules are installed, when an administrator explicitly flushes the cache, etc. It shouldn't happen on a day-to-day basis.
  • It's only been used with independent entities. If you need to create a sequence of related entities (e.g. create a UFGroup and then create a UFField inside that group), then it may be hard to link the related entities. (This is generally untried, though. API chaining might help, but again --I haven't tried.)
  • For upgrades, it handles simple replacements but not complex migrations. For example, suppose v1.0 of a module defines two reports (Reports A and B), but v2.0 uses a generalized report to replace both of them (Report C). If the module author simply replaces reports A+B with C, there may be some stale references to A+B (e.g. old hyperlinks or report-instances) which should be transitioned.

[EDITED 27-May-2013: Add more hyperlinks and more limitations]

Comments

Hey Tim,

 

Thanks for documenting this. I certainly hadn't understood all of this & I've played with extensions a reasonabley amount. We should link to this from the wiki? the booki? too.

 

I can see wanting to have the ability to create report instances and dashlets too. Api feature request I guess

OK, I've added links from a few wiki pages to this blog (e.g. links from the "hook_civicrm_managed" page, the "hook_civicrm_install" page, and the "Create a Module Extension" tutorial).

Thanks for documenting this Tim. Will certainly give a good go!

Hi,

While working on sepa, we have tried this approach. It worked fine with a payment processor type, but we failed on other entities. It seems it need a is_active field to work, could you confirm?

As a side note, we worked on it before this post so the documentation wasn't as complete, and that was a bit of a "change semi randomly something, enable/disable the extension, rince and repeat until it works.

It would be handy to have a --dry-mode or equivalent to make it easier to debug. might be something to add for the sprint?

 

X+

1. It is pretty naive about is_active, but I think that's fixable. We would need a way to determine which entities support "is_active" -- maybe a simple white-list, a simple black-list, or some kind of wrapper around getfields. Alternatively, we could introduce API actions called "enable" and "disable" (e.g. "drush cvapi ReportTemplate.disable id=4") -- for some entities, they would trigger an update to is_active; for others, they'd be nullops. From "KISS" perspective, I'd lean toward "white-list"; from "make it flexible" perspective, I'd lean toward "enable/disable" actions. What do you guys think?

2. The --dry-mode sounds like a good idea. How would we want to expose it? e.g.

$ http://example.com/civicrm/cacheclear?dryrun=1

or

$ drush cvapi ManagedEntity.reconcile dryrun=1

FWIW, I think the second approach would be easier. The cacheclear does a lot things besides reconciling managed entities, and making each of them support "dryrun" would take more work.

For is_active, if the field doesn't exist, we can simply not disable it and display a message "some content hasn't been disabled automatically {list of manged entity type}, you might need to manually disable them"

 

For dry-run, it's has low level has you want it to be, meant for dev, so api explorer/drush is fine. I was thinking about the api.extension.disable?dry-run=1 that would list the entities it want to disable/can't disable without disabling them (or the extension)

I don't have a problem with dryrun being command line only (or potentially api explorerer if we need a UIn for other CMS that don't have drush)

Hi,

great approach to improve extensions installation!!!

What about custom fields creation on this managed hook?
I use to create them in hook_civicrm_install, because there is a chain of actions and IDs needed (custom_group_id, option_group_id, etc) that I don't know if the can be achived with this hook_civicrm_managed.

Let's say I want to create a custom field, "select" type with 3 options.
How would be the process to do that?

 

cheers!

a) Yup, it's important to get all those items (CustomGroup, CustomField, OptionGroup, OptionValue) created in the right order with the right foreign-key references between them.

b) I've had the same issue while prototyping extensions for CiviHR. For that, I've re-used the migration XML tool instead of using managed-entities. That tool already has some logic to handle ordering and FKs.

b) If we really wanted to make the managed-entity system work here, we could specify that API operations are automatically ordered based on their entity-type (e.g. first insert/update all OptionGroup's, then insert/update all OptionValue's, then CustomGroup's, then CustomField's). We'd also need to revise APIs to set foreign-key references using portable symbols instead of IDs (e.g. set an OptionValue.option_group_id using the name of the OG instead of the numeric ID).

c) However, I don't think we'd want to do this for managing custom data -- when it comes time to change the schema (e.g. use a different data type, or split one column in two, or consolidate columns), the automatic "reconciliation" process will be too crude and likely to cause data loss. Some systems -- including Drupal's Schema API and Civi's core SQL schema -- require one to separate the schema declarations (used on new installs) from the upgrade steps. The separation seems good/necessary. You can get this separation by using the XML for new installs and using the Upgrader ("function upgrade_1001", etc) to handle schema changes.

 

Aside: To create the "xml/customdata.xml" file used by org.civicrm.hrqual, I originally tried running the CLI tool ('bin/migrate/export.php'), but it produced way too much data to edit by hand. So instead I used a one-off/throw-away script:

<?php
// FILE: /tmp/export-my-groups.php
// USAGE: drush scr /tmp/export-my-groups.php
civicrm_initialize();

$export = new CRM_Utils_Migrate_Export();
$export->buildCustomGroups(array(
  // list specific custom group ids; all required CustomFields/OptionGroups/OptionValues will be fetched
  1,3
));
print $export->toXML();

I'd like to simplify this and post a more detailed article on the topic, but if you're in crunch then you can write a throw-away script, too. (To do the export, you need a very recent version of 4.3.x -- i.e. checkout Civi from git. This is only needed on the development box.)