Getting out of limbo with core extensions

(helpful note - the intended audience for this content is developers and people having trouble falling asleep)

For some time our approach to taking CiviCRM forwards has been “Leap by Extension, Iterate by Month (LExIM)” LeXIM - with the intention that any significant improvements or new functionality is developed in an extension. This has been a learning process and we’ve worked through a lot of issues and it has led to lots of improvements in our integration points and extension management. However, one area where we continue to have problems is product maintenance. As we develop replacement functionality in an extension we have not developed a path to phase-out the old while phasing-in the new and we wind up under-maintaining both-- all while overworking (violins please) to do so.

A good example is FlexMailer. This replaces some no-good-very-bad code for the inner-workings of CiviMail. It was developed as an extension which is currently used by Mosaico. However, the code it was intended to replace is still used by many, perhaps most, CiviCRM sites. So we wind up with situations like this bug where we don’t really want to spend more time on the to-be-replaced code, but we don’t have a clear path to replace it.

As a first step towards addressing this, we are starting to move code that is intended to be maintained as part of the core product into the “civicrm-core” repo, while still keeping this code modular as an extension. There are a number of technical issues still to be solved so we are doing this in small chunks of work - move the code in, then work to enable on new installs and finally review the best way to handle it on existing installs. FlexMailer is in the core repo from 5.28 and you can choose to disable and remove any other copies of it in favour of the core version, but we will not be enabling on upgrade until we feel the process is solid and well tested and we are confident there is no downside. Before we look at that we will move to enable-on-install on new installs. Core extensions are located in the “civicrm/ext/” folder (within the CiviCRM download).

In general the idea is that the code on our roadmap  will reside in the core codebase and will be ‘upgraded’ from its current semi-maintained state to being under our product maintenance regime. In the case of the new search builder (which will in the long term replace some of our hard-to-maintain search code) we have put it straight into the core repo (but not yet UI-installable), deferring some technical challenges until later.

Another example is the flexible form builder, Afform. The goal has always been to replace core forms with forms built using the afform extension. Doing that in core is not really possible until we can be confident afform is itself part of core. In the near future we will be moving afform into the main repo as a core extension. Once we have worked through all the issues that entails, we can look to switching over some of the simpler forms and progressing towards more complex forms.

Our goal of revamping forms with Afform also highlights that we need a process for doing that. For example the ‘event self transfer form’ is one I’ve recently sunk time into reviewing, and it was pretty obvious it’s a clunky form. This is also somewhat niche functionality, and many sites would be happy to disable in favour of a slimmer product.

If we wished to leap this chunk of functionality, we would ideally extract the existing implementation into a core-extension and enable that on upgrade and on new installs (resulting in no change in behaviour). We would then create a new parallel extension with the new afform version of the functionality and switch to this being the one enabled on new installs. After some months we would review existing installs, perhaps suggesting on upgrade that sites consider disabling the old one and enabling (or not) the new one. At some point we would hope to move the old extension out of core - but that is in the very distant future at the moment and would be case by case.

The difficulty with the above is that the first step - extracting to an extension - can be too big to do in one round of work. Matt Wire has bravely waded in to start extensionising event-cart, but this code has many tentacles and if we required everything to be disentangled before any work could be merged we would be where we are now - in limbo. 

In order to be able to work through the complex process of moving code already deeply entwined in core into a more modular structure with better architecture, we needed to create a way to bridge the chasm. To this end core-extensions can be hidden and when this is done it means that they are not visible in the UI and disabling them is unsupported and might whitescreen the site. The goal is a more modular CiviCRM with better use of integration points but in order to get there we need some stepping stones.

In general core extensions are structured as extensions but sit in the core repo and are under the product maintenance remit. Some of them will be there because they are part of the future of the product and these ones should interact with core with clearly defined appropriate integration points (hooks, api, listeners). Others will be there because they are part of the existing product and we are working on restructuring them into interacting with the core product through appropriate integration points. This latter category will be ‘hidden’ for the medium term as we work through the technical issues. Note that moving core code to a core extension should be understood as a technical cleanup and any decision as to whether that code remains as part of the core product is entirely separate and not to be assumed.