Civix v22.05: How to remove a million lines of extra code

Published
2022-05-02 19:20
Written by
totten - member of the CiviCRM community and Core Team member - about the Core Team

civix is a development tool. It generates code for extensions, providing a baseline for developers and hackers who want to improve and add onto CiviCRM.

civix generates a lot of code. Much of this code is educational; hopefully, some of it is even useful. But some parts of it are redundant or excessive. The excess bits don't matter much with a single extension. However, in practice, they are copied to almost every extension. Individually, these are small bits. Collectively, they add up.

civix v22.05 reduces code in each extension by ~210 lines (if you run the update). Of course, with 4,800 extensions reported to the statistics tracker, it's not realistic to run the update on all of them. But, if we did, it would remove ~1 million lines of code from the universe.

The rest of this posting is targeted at developers -- outlining what has changed and how you can use it.

TLDR

  • Download civix v22.05 (or newer). (If you use civicrm-buildkit, you can update buildkit.)
  • In your extension, run "civix upgrade". The command will apply updates:
    • In the main PHP files ("myextension.php" and "myextension.civix.php"), it removes some boilerplate.
    • In the "info.xml" file, it enables <mixin> tags. This provides equivalent functionality for CiviCRM >=v5.45.
    • In the "mixin" folder, it creates optional compatibility files. This provides equivalent functionality for CiviCRM <=v5.44
    • It checks for other changes+suggestions from past releases. It may add or remove other bits of boilerplate.

Demonstration

Most CiviCRM extensions should be developed in a version-control system, such as git. A typical workflow for updating an extension would look like this:

cd path/to/myextension
git checkout -b update-civix
civix --version
civix upgrade
git status
git add .
git commit -m 'Update civix templates'
git push -u myrepo update-civix

For example:

TIP: The demo may scroll too fast or slow. To play at your own speed, press pause and use the arrow keys.

In Depth: What changed?

Some small responsibilities were moved from boilerplate functions (hooks) to mixins. Mixins are features that you can add or remove from an extension. They are primarily configured in info.xml.

diff --git a/info.xml b/info.xml
index c2cf456..8a987f4 100644
--- a/info.xml
+++ b/info.xml
@@ -29,9 +29,16 @@
   <comments>To help contribute please contact us on support@vedaconsulting.co.uk or initiate a conversation / issue on github.</comments>
   <civix>
     <namespace>CRM/Mosaico</namespace>
+    <format>22.05.0</format>
   </civix>
   <classloader>
     <psr0 prefix="CRM_" path=""/>
     <psr4 prefix="Civi\" path="Civi"/>
   </classloader>
+  <mixins>
+    <mixin>ang-php@1.0.0</mixin>
+    <mixin>menu-xml@1.0.0</mixin>
+    <mixin>mgd-php@1.0.0</mixin>
+    <mixin>setting-php@1.0.0</mixin>
+  </mixins>
 </extension>

In the mosaico example, it enabled four mixins:

Mixin (@Expected Version)Description
ang-php@1.0.0Autoload Angular modules from PHP files (./ang/*.ang.php)
menu-xml@1.0.0Autoload router ("Menu") configuration from XML files (./xml/Menu/*.xml)
mgd-php@1.0.0Autoload managed entities from PHP files (**/*.mgd.php)
setting-php@1.0.0Autoload settings from PHP files (./settings/*.setting.php)

These mixins serve as glue, telling CiviCRM how to load files from the extension. This same behavior was
previously implemented with boilerplate functions -- which can now be removed. For example, after adding "menu-xml@1.0.0", it removes two functions ("mosaico_civicrm_xmlMenu()" and "_mosaico_civix_civicrm_xmlMenu()").

diff --git a/mosaico.php b/mosaico.php
index 91f2bef..7d3a900 100644
--- a/mosaico.php
+++ b/mosaico.php
@@ -12,15 +12,6 @@ function mosaico_civicrm_config(&$config) {
   _mosaico_civix_civicrm_config($config);
 }

-/**
- * Implements hook_civicrm_xmlMenu().
- *
- * @link http://wiki.civicrm.org/confluence/display/CRMDOC/hook_civicrm_xmlMenu
- */
-function mosaico_civicrm_xmlMenu(&$files) {
-  _mosaico_civix_civicrm_xmlMenu($files);
-}
-
 /**
diff --git a/mosaico.civix.php b/mosaico.civix.php
index f4b8064..bf1cbaa 100644
--- a/mosaico.civix.php
+++ b/mosaico.civix.php
@@ -105,19 +113,8 @@ function _mosaico_civix_civicrm_config(&$config = NULL) {
-/**
- * (Delegated) Implements hook_civicrm_xmlMenu().
- *
- * @param $files array(string)
- *
- * @link https://docs.civicrm.org/dev/en/latest/hooks/hook_civicrm_xmlMenu
- */
-function _mosaico_civix_civicrm_xmlMenu(&$files) {
-  foreach (_mosaico_civix_glob(__DIR__ . '/xml/Menu/*.xml') as $file) {
-    $files[] = $file;
-  }
-}

Similar boilerplate functions existed for several hooks (hook_alterSettingsFolders, hook_angularModules, hook_caseTypes, and hook_managed). All of these become unnecessary. Moreover, these hooks relied on internal helper functions (_mosaico_civix_glob(), _mosaico_civix_find_files()) which also became unnecessary. So they are all removed.

Finally, the upgrade created a new folder with mixin files:

$ ls -l mixin
-rw-r--r-- 1 me users 3301 May  1 02:20 polyfill.php
-rw-r--r-- 1 me users 1213 May  1 02:20 ang-php@1.0.0.mixin.php
-rw-r--r-- 1 me users  866 May  1 02:20 menu-xml@1.0.0.mixin.php
-rw-r--r-- 1 me users 1256 May  1 02:20 mgd-php@1.0.0.mixin.php
-rw-r--r-- 1 me users 1052 May  1 02:20 setting-php@1.0.0.mixin.php

These files provide backward compatibility. They are copied from civicrm-core@5.45 into the extension; if you install the extension on an older version of CiviCRM, they will fill-in missing functionality. (If you install on a newer CiviCRM, then these files are ignored.)

In depth: Improvements

How is this better? Broadly, it means loading, executing, and maintaining less code. More specifically:

  • De-duplicated: If 20 extensions implement hook_managed with similar boilerplate, then they have de facto duplicates (consuming extra space in RAM+CPU). A mixin file is loaded once and shared across extensions.
  • Easier updates: If a boilerplate function changes, then you have to propagate that change to every extension individually. With mixins, you can update one time.
    • Example: In the upcoming CiviCRM v5.50, hook_managed adds extra options to improve the performance. You could update the boilerplate in every extension so that they all get the performance benefit, but that's a lot of effort. With a mixin, you can update one file (civicrm-core:mixin/mgd-php@1/mixin.php) and provide the benefit to all extensions.
  • Machine editing: civix can enable or disable the <mixin> tags depending on whether they are actually needed. This is easier and more accurate than machine-edits on myextension.php file. This makes it more responsive to actual usage, which leads to the next improvement:
  • Less file scanning: The old boilerplate functions performed broad file-scans (just in case). Since mixins only load when needed, they will do less file-scans.
    • Example: I would wager that 95%+ of extensions do not define CiviCase types. Never-the-less, almost every extension has had a boilerplate function that scans for CiviCase files (eg _myextension_civix_civicrm_caseTypes() scans ./xml/case/*.xml). But mixins are only activated if needed, so they perform fewer scans.
  • Less maintenance noise: The old boilerplate functions were added to all extensions, regardless of whether they were needed. If you're doing maintenance tasks (such as examining hook-usage or migrating hooks), these extra functions will add noise.
    • Example: In the near future (perhaps CiviCRM v5.50 or 5.51), hook_caseTypes may be deprecated in favor of the more powerful hook_managed. But if hook_caseTypes is deprecated, then all those boilerplate functions would be deprecated as well - which could provoke a number of pro-forma warnings or upgrade-tasks.

This system preserves some benefits of boilerplate:

  • Portable: Even though mixins have a canonical home in civicrm-core.git, they can be used on different versions of CiviCRM. Extension developers often need to meet their own schedule (which may not align with civicrm-core updates). To get through these edge-cases, you can copy a mixin file from core to an extension. (civix does this automatically for the mixins that it relies on.)
  • Hackable: You are not required to use the mixins from civicrm-core. For example, if you want to change the way it scans for ./xml/Menu/* files, you could copy the "menu-xml" mixin, rename the file, and edit the logic. Similarly, if you have some special behavior or file-layout that you like to re-use in all your extensions, you can copy a mixin to each.

FAQ

Q: What happens if I customized one of these hooks? Will changes be removed?

The upgrader should be conservative about changes in the main PHP file ("myextension.php"). If it doesn't strictly recognize the code, it will prompt for directions.

Q: If I upgrade an extension, will it change the system requirements?

A: The requirements should stay the same. civix (with mixins) aims to provide a similar level of backward/forward compatibility as it had before (with boilerplate).

Specifically, civix v22.05 generates a mixin folder for compatibility purposes. We have run the standard civix tests on a few recent versions, with and without builtin mixins (v5.39, v5.44, v5.45, master).

However, it has not been deeply tested across all versions; so if you need to support a specific version, then it is prudent to try it and see.

Q: Are there any differences between how mixins work on older and newer versions of CiviCRM?

Compare:

  • On CiviCRM <=5.44, mixins are loaded by {$myextension}/mixin/polyfill.php. It simply reads all files in {$myextension}/mixin/* and runs them. This is simple and portable, but it has suboptimal caching, version-checks, and deduplication. This trade-off is acceptable for a polyfill that is widely copied but rarely used.
  • On CiviCRM >=5.45, mixins are loaded by CRM_Extension_MixinLoader. This file is maintained centrally and updated with civicrm-core. The MixinLoader reads info.xml and activates the best available version of each mixin. It provides better caching and deduplication -- but it has more complexity and more tests. This trade-off is more suitable for long-term usage and maintenance.

Q: Has anybody used mixins before?

A: The extensions in civicrm-core have been using mixins for several months.

Q: You said it deleted lots of lines. But it also adds a bunch of lines in the mixin folder!

A: Yeah… well… I may have oversold it. But it reduces the code that matters!

  • Fewer lines are loaded into a typical runtime. Most mixin files will not be loaded in a typical runtime.
  • Fewer lines require maintenance. The mixin files are dumb copies, and you'll get updates without needing to patch each extension.
  • If you don't need backward compatibility, then you can simply delete the files.

Q: What if I need to use the old templates (with boilerplate, not mixins)?

A: You can still download prior versions of civix. civix v22.02.0 is the highest release supporting the older templates.

There are more questions and answers in civicrm-core#22198: FAQ.

More information


Comments

Fantastic, thanks for wrestling with this Tim from my early and naive suggestion about class inheritance. It is a (trademark) genius solution you've developed. I think it might be helpful with things like reducing opcache requirements too.

If our extension release does not support CiviCRM versions prior to 5.45 can we just delete the mixin folder?

@eileen Yup, that should be fine.

FWIW:

* The core extensions on 5.45+ are similar to that. (They rely on `civicrm-core:mixin/*` and don't need their own `mixin/` folders.)
* There could be other use-cases for having a `mixin/` folder, but code generated by civix 22.05 doesn't need anything else.
* If you delete the `mixin/` folder and accidentally run on 5.44, I would expect a hard fail in the `require $polyfill` line:

```php
function _myextension_civix_mixin_polyfill() {
if (!class_exists('CRM_Extension_MixInfo')) {
$polyfill = __DIR__ . '/mixin/polyfill.php';
(require $polyfill)(E::LONG_NAME, E::SHORT_NAME, E::path());
}
}
```

Sounds like a great step forward. Thanks Tim!

How do you add a mixin with civix?
I had an extension and needed the menu-xml mixin added but couldn't find a way to do it apart from adding a page (which I then deleted). I know normally you are creating pages/forms/etc. via civix and it takes care of adding the appropriate mixin, but I had a situation where I was defining a menu route manually and needed the mixin to be added.

@lcdweb Use `civix mixin` eg `civix mixin --enable=menu-xml@1`

You may need to update your version of civix