CiviCRM Extensions framework - taking off in 3.3

Published
2010-09-09 06:42
Written by

This blog post is outdated. For latest information about how to create extension, please refer to Extensions Wiki Page.

 

It had to happen sooner or later - CiviCRM is growing with with variety of functionality, where people can plug in their own, custom pieces and make CiviCRM more tailored to their needs. Most prominent examples at the moment include payment processors, custom searches and custom reports. Don't confuse it with "larger scale" customisation, like writing Drupal modules which - using API and hooks - modify CiviCRM behaviour. We're talking about well defined, self-contained pieces of code which throw in some useful functionality into your existing installation. As of now, it's a bit of a hassle to install them - you need to put files in proper places, register them using administrator section and so on: nothing extremely complicated, but also definitely not the easiest part of CiviCRM setup and configuration. More to that, some very useful extensions (like some payment processors) are shipped with vanilla CiviCRM, but not supported by core team, some  of them are not shipped, but available only from forums or issue tracker. In general - you can find a lot of useful things, it's just requires some effort.

So now, what if you could just install CiviCRM, than go to an "Extensions" admin page, review available pieces, pick what you need from the list and just click "Install" button? We felt tempted to make it happen and some time ago we started talking about such feature. Now it's time to start introducing it - CiviCRM 3.3 will provide the first pieces of more long term effort in making CiviCRM extensibility easier. Work is currently in progress, most of pieces are there already (expect first demo in a few days), but before pushing out the first version, we would like to get your feedback and/or constructive criticism on the path we're taking. During next few weeks, before we hit alpha with 3.3, I will be publishing blog posts describing different aspects of how our extensions will work. If you're interested in this topic, follow the Extensions category on the blog and be active in comments (and in other ways as well, but that's to be announced later on). :-)

 

Goals for the first phase (and a bit beyond)

 

What will you see in the first phase of building our extensions framework? In CiviCRM 3.3 you will see:

  • extension packaging format for custom searches and payment processors (describing it below)
  • simple user interface for manipulating exensions (install, deinstall, turn on and off)
  • first take on custom report packaging

I suspect that whole 3.3 life cycle period will be mainly about gathering experience with with packaged extensions - in that sense it will be more of an experimental functionality. Old ways of installing payment processors, custom searches and custom reports will be still available. After this "trial period", as the number of packaged extension grows, we'll start eliminating pieces of old functionality, but backwards compatibility will be held over at least 1-2 versions ahead.

 

CiviCRM Extension info file

 

Here we go, let's go with introducing first, most fundamental concept of our extension framework. We need to know what given extension does, who wrote it, who supports it and quite a few other things. That's what info file is for. It's in XML format and it's relatively simple. Here's an example:

 

<?xml version="1.0" encoding="UTF-8" ?>
 <extension key="org.civicrm.activity" type="search">
  <callback>ActivitySearch</callback>
  <name>Activity Search</name>
  <description>
   This custom search allows to search through activities 
   and returns activities as results (not contacts as 
   regular search).
  </description>
  <url>http://civicrm.org</url>
  <license>AGPL</license>
  <maintainer>CiviCRM Core Team &lt;noreply@civicrm.org&gt;</maintainer>
  <releaseDate>2010-09-01</releaseDate>
  <version>1.0</version>
  <compatibility>
   <ver>3.3</ver>
   <ver>3.4</ver>
  </compatibility>
  <develStage>beta</develStage>
  <comments>For support, please contact project team on the forums. (http://forum.civicrm.org)</comments>
</extension>

Quick description of XML elements:

  • extension - enclosing element: everything needs to sit inside of it. Attributes are:
    1. key: unique name of the extension (should match the name of directory this extension resides in). We're going with Java like reverse domain naming to make it easier with identifying unique extensions and giving developers more flexibility on providing different extensions.
    2. type: one of "search", "payment", "report" for now, meaning that this extension is - respectively - a custom search, payment processor or custom report.
  • callback - the name of the class that should be called in order to run this extension. Following the namespacing convention in CiviCRM, it will also be the name of the php file. Above example means that the file $extension_dir/search/ActivitySearch.php should exist and contain Extension_Search_ActivitySearch class. Extension_Search part of the namespace is obligatory, the rest is under extension author's control.
  • name - well, that one's easy. It's a name of the extension.
  • description - easy as well.
  • url - address to extensions website with documentation, more information, etc.
  • license - the name of the license under given extension is offered.
  • mainainer - self explanatory, hopefully. Email address inside required.
  • releaseDate - date when given extension has been released on.
  • version - version of the extension.
  • develStage - if you want to push out you extension in unstable state, here's the opportunity to be sincere about it. Allowed values are: "alpha", "beta", "stable". Beta is the default if you don't define develStage - 'cause you must have been unsure, if you forgot to provide such crucial piece of information. :-)
  • compatibility, ver - lists compatible versions. It cares only about first two elements of version name and ignores the rest to avoid the need for frequent updating. So if your extension is compatible with CiviCRM 3.3, also means it supports 3.3.beta4. However, if you state it supports 3.3.beta4 (<compatibility><ver>3.3.beta4</ver></compatibility>), it won't support any other version - no go on 3.3.1.
  • comments - last one, the opportunity to say anything you want. :-)

 

Loving all the things localised, we've added that too. There is possibility to provide extension description in other languages - optional localisedInfo section of the info file does that:

 
<extension ...>
    ...regular stuff
    <localisedInfo>
        <pl_PL>
            <description>Opis po polsku.</description>
            <maintainer>Zespół CiviCRM &lt;noreply@civicrm.org&gt;</maintainer>
            <comments>Wsparcie dla tego rozszerzenia dostępne na forum CiviCRM (http://forum.civicrm.org).</comments>
        </pl_PL>
        <pt_BR>
            ...
        </pt_BR>
    </localisedInfo>
</extension>

 

It only supports name, description, maintainer and comments sections - all other extension information defined here will be ignored.

 

That's more or less everything about the info file. Got suggestions? Comments under this post await your voices. :-) Coming up next in a few days - step by step tutorial on packaging your extension.

Filed under

Comments

If I understand it, you're basically beginning to role out a module framework (a la drupal) for CiviCRM - that's awesome!

 

One comment, and this is gonna be annoying: why xml? it would be nice to have a file format more like the .info file for drupal modules. Main reason: readability. I imagine at this point it's probably too late, but if this was possible i think it would be great.

 

Looking forward to more information on the general structure extensions, and how these will fit with the existing API (+ drupal modules). Seems that for CiviCRM's sake a lot of custom drupal modules we are writing would probably be a better fit for CiviCRM extensions. Also, it would be great to get info on how these can/will work with ugprades. Is there going to be an extensions section of civicrm.org? 

I know that many people are used to the Drupal style .ini files, but XML isn't too bad to deal with, and it's used throughout CiviCRM anyway.

There are a number of very good deployment models for extensions, and while the Drupal model is very useful, there are better models for end-users.  For example, the Java model of just putting something into a zip-type archive (which could contain the .xml info) into a directory is deal easy, and lets you do neat things like signing archives for validation -- a real good thing if you are using sensitive code like payment processors -- and makes it easier to do things like on-line updating, which is very hard for Drupal to do securely without developer tools like drush.

 

The main thing I'd want for a deployment model would be auto-configuration with reasonable defaults, and a good simple hook to display configuration UI and store configuration preferences (i.e., one that doesn't require developers to write to QuickForms objects).  Ability to register templates and other auxiliary files would be nice too, as well as a way to register preferences or change existing settings.  This way, things are as easy as possible for end users -- meaning people who manage CiviCRM sites -- without being too hard for developers.

Hey, good point on archive signing - while it was quite obvious to me that we'll need some kind of extension bundle integrity checking implemented, I didn't really get to detailed thinking here. You got my head moving here. :-)

 

As for ideas in last paragraph - definitely worth thinking more about, maybe even dumping more detailed idea descriptions on the forum: http://forum.civicrm.org/index.php/board,57.0.html - however, most of it won't be on the roadmap for the first version.

Kyle,

 

I chose XML mainly for consistency - it's used in many different places in CiviCRM and I wouldn't like to introduce another standard.

 

As for other types of extensions - the idea was to provide some mechanism for "things pluggable" that are already there, see how it works out and than plan for extension types other than custom search, payment processor and custom report. Totally agree, that this could work out quite well for small scale CiviCRM modifications, where Drupal module is a bit of an overkill, but I don't think we have clear idea on how to structure it. Since current work refers to already existing pluggable code and focuses on providing easier installation for them, I don't think it intervenes with thinking about how to handle different types of extensions, it doesn't close any paths here IMO - only provides the way to quickly add/remove different mods to CiviCRM. I think whole 3.3 lifecycle might be a good time to think about and gather ideas/experience on how to structure other types of extensions.

 

As for extension section - yeah, in one of next steps (hopefully for 3.4), we'll try to build a centralised repository of extensions. The idea (and I'm "sneak previewing" one of next blog posts here) is to be able to go into extensions management page after core upgrade and basically "click-upgrade" those extensions which got outdated because of an upgrade.

That's good news! One question so far: Will the extensions be located outside of the modules/civicrm directory in a Drupal site? That would make upgrading CiviCRM easier.

What happens if a report and search are both entitled "activity"?  What if I call my custom extension "events", but then an extension gets bundled with CiviCRM core called "events"?

I recommend removing abbreviations from the XML file.  Actually from anywhere and everywhere.  With modern coding tools abbreviations just create confusion (ex. is vid version id, vocabulary id, or video, or something else?). 

I think it would be a good idea to also track development version in addition to development stage.  What happens when I want to create a new version of my payment processor to integrate with the provider's new API?

This whole feature should be easily disabled.  None of my clients would be comfortable with a system that can install code via the UI. 

I also agree with another commenter that suggested the code live outside of the CiviCRM codebase for easier upgrading. 

Dave,

- abbreviations: since in one of next steps we want to introduce centralised source of information about extensions, we need some kind of uniqueness, which is provided by the combination of extension type and extension "key". So if there is a report and search called "activity", it's a completely valid situation, since the are in different, type defined namespaces. Not sure what you mean by extension being bundled with CiviCRM core. However, I'm not fundamentally attached to the idea - if you have an idea on how to assure unique identifiers to extensions - go ahead, describe it here or on forums: http://forum.civicrm.org/index.php/board,57.0.html - I'll gladly introduce it if it meets all the needs. Please note that I would like to freeze this part of code around the end of this week, so the time matters.

- development version: good point, I'll introduce that.

- disabling: wouldn't administrator privileges be enough to protect the installation from adding unnecessary code?

- code living outside - yes, all the extensions will live in separately configured directory. That's might also be an answer to previous concern - it might be enough to leave extensions directory setting blank to disable any extension operations

 

Thx,

m

I am not sure about the idea that an extension is always and only an extension of just one thing like a search or a report. I think it will be quite useful to package together custom fields, custom profiles, custom searches, custom reports, and perhaps overrides of a few things into a single 'feature' extension like CiviEngage. To allow that to happen, I'd prefer to avoid putting extension type into the definition of extension namespace.

Just a +1 on avoiding abbreviations. I find it irritating when Drupal modules don't have the same module name in svn as they do at http://drupal.org/project. There's good reason for keeping the human readable name of an extension equivalent to an element of its name in code.

Hi,

 

This sounds good but I'm wondering about where the various extensions will be hosting / whether there will be a central repository.

 

There is a risk that addins will not be shared if there is not a easy entry into sharing them. Also having them available from one location will facilitate drush installs & installation profile installs of them

Eileen,

 

This will be the next step. For 3.3, we just want to provide and test drive the extension mechanism in CiviCRM, in 3.4 or next after that, we are planning to introduce central repository for extensions. I'll write about those plans in one of the following blog posts.

 

Thx,

m

I think the following elements would be useful to have in the new CiviCRM Extension info file file:

URL: URL of website describing the extension, additional documentation
Version: version of the extension itself
Date: Date the extension was released
License: GPL2, or AGPL, etc.

civikey: com.pogstone.civistuff.mycoolreport

 

Notice the "civikey" element is meant to be a univerally unique handle for my extension.  I am following the Java convention of  reversing my domain name as the "prefix" for all my extensions. So I would be responsible for making sure I do not create 2 extensions called "civistuff.mycoolreport" within the "com.pogstone" namespace.

 

Question: What if a CiviCRM extension needs to call the APIs? Will the CiviCRM API be pre-initialized? Or will each extension need to do this seperately?

I agree that extension versions are appropriate, separate from CiviCRM versions they are compatible with. Drupal's approach on this front seems mature and transferrable. However, in a particular instance of an extension only one would ever appear.

I think it would be better to not have a developer person / org level in the namespace. Although that is a reasonable standard for something like a Windows installation space, in open source it doesn't make as much sense where the maintainers may be an ad hoc team where people come and go.

In 3.4 or whenever, it would be good to have system for claiming an extension name, and rules about giving it up if unused, etc.

Hi,

Namebadges uses the same convention than the custom searches. Planning to integrate them?

 

As for the xml, have you ever had to struggle with one of these monstrous config files for java framework ?

My personal experience is that  "because that's the java way" tend to justify NOT to use it ;)

 

</troll>

I have had to wade through enormous XML files for Java, I am not looking to repeat that with CiviCRM. However, the idea of reversing the domain name as a unique prefix is used in many areas, not just Java.  I also think many of these fields should be optional.

 

I was also thinking of how the "plugins" area in WordPress works, which gives extra information to the novice administrator.

It would be good to start thinking about dependencies early on. Does an extension depend on one or more other extensions, and / or CiviCRM component being enabled, and / or a Drupal module or Joomla! extension? And which version(s) of extensions, modules etc. are required?