Getting out of limbo with core extensions

Published
2020-07-10 19:39
Written by

(helpful note - the intended audience for this content is developers and people having trouble falling asleep)

For some time our approach to taking CiviCRM forwards has been “Leap by Extension, Iterate by Month (LExIM)” LeXIM - with the intention that any significant improvements or new functionality is developed in an extension. This has been a learning process and we’ve worked through a lot of issues and it has led to lots of improvements in our integration points and extension management. However, one area where we continue to have problems is product maintenance. As we develop replacement functionality in an extension we have not developed a path to phase-out the old while phasing-in the new and we wind up under-maintaining both-- all while overworking (violins please) to do so.

A good example is FlexMailer. This replaces some no-good-very-bad code for the inner-workings of CiviMail. It was developed as an extension which is currently used by Mosaico. However, the code it was intended to replace is still used by many, perhaps most, CiviCRM sites. So we wind up with situations like this bug where we don’t really want to spend more time on the to-be-replaced code, but we don’t have a clear path to replace it.

As a first step towards addressing this, we are starting to move code that is intended to be maintained as part of the core product into the “civicrm-core” repo, while still keeping this code modular as an extension. There are a number of technical issues still to be solved so we are doing this in small chunks of work - move the code in, then work to enable on new installs and finally review the best way to handle it on existing installs. FlexMailer is in the core repo from 5.28 and you can choose to disable and remove any other copies of it in favour of the core version, but we will not be enabling on upgrade until we feel the process is solid and well tested and we are confident there is no downside. Before we look at that we will move to enable-on-install on new installs. Core extensions are located in the “civicrm/ext/” folder (within the CiviCRM download).

In general the idea is that the code on our roadmap  will reside in the core codebase and will be ‘upgraded’ from its current semi-maintained state to being under our product maintenance regime. In the case of the new search builder (which will in the long term replace some of our hard-to-maintain search code) we have put it straight into the core repo (but not yet UI-installable), deferring some technical challenges until later.

Another example is the flexible form builder, Afform. The goal has always been to replace core forms with forms built using the afform extension. Doing that in core is not really possible until we can be confident afform is itself part of core. In the near future we will be moving afform into the main repo as a core extension. Once we have worked through all the issues that entails, we can look to switching over some of the simpler forms and progressing towards more complex forms.

Our goal of revamping forms with Afform also highlights that we need a process for doing that. For example the ‘event self transfer form’ is one I’ve recently sunk time into reviewing, and it was pretty obvious it’s a clunky form. This is also somewhat niche functionality, and many sites would be happy to disable in favour of a slimmer product.

If we wished to leap this chunk of functionality, we would ideally extract the existing implementation into a core-extension and enable that on upgrade and on new installs (resulting in no change in behaviour). We would then create a new parallel extension with the new afform version of the functionality and switch to this being the one enabled on new installs. After some months we would review existing installs, perhaps suggesting on upgrade that sites consider disabling the old one and enabling (or not) the new one. At some point we would hope to move the old extension out of core - but that is in the very distant future at the moment and would be case by case.

The difficulty with the above is that the first step - extracting to an extension - can be too big to do in one round of work. Matt Wire has bravely waded in to start extensionising event-cart, but this code has many tentacles and if we required everything to be disentangled before any work could be merged we would be where we are now - in limbo. 

In order to be able to work through the complex process of moving code already deeply entwined in core into a more modular structure with better architecture, we needed to create a way to bridge the chasm. To this end core-extensions can be hidden and when this is done it means that they are not visible in the UI and disabling them is unsupported and might whitescreen the site. The goal is a more modular CiviCRM with better use of integration points but in order to get there we need some stepping stones.

In general core extensions are structured as extensions but sit in the core repo and are under the product maintenance remit. Some of them will be there because they are part of the future of the product and these ones should interact with core with clearly defined appropriate integration points (hooks, api, listeners). Others will be there because they are part of the existing product and we are working on restructuring them into interacting with the core product through appropriate integration points. This latter category will be ‘hidden’ for the medium term as we work through the technical issues. Note that moving core code to a core extension should be understood as a technical cleanup and any decision as to whether that code remains as part of the core product is entirely separate and not to be assumed.

Comments

Eileen thank you for detailing and writing this up.

I think the plan makes sense. As we saw with API4 the migration from extension, to core bundled extension to included and enabled in core is a tough path. We had many sites with multiple versions of API4 installed as it could exist in two locations.

If a new feature is planned it makes sense to bundle as an extension and phase it in. Having it in core but not enabled should give time for comments and improvements before it is turned on for all. Add to that that buildkit installs will have these turned on it gives us an opportunity to really test these out.

Kevin - yes - I think the api4 experience is not one we want to repeat - it also had a really detrimental effect on the speed of it's progression, cost a huge amount of time in terms of the actual technical challenges of the migration, and setting up the unit tests first for the extension and then redoing that work, and left us inventing things in apiv3 for about 2 years after we really should have been doing new stuff in apiv4, and left us unable to leverage its goodness in core until relatively recently

I'd like to remind readers that 'extensions' is a thing that is bigger than just the ones you manage via your web interface, and it's a bit confusing.

A civicrm install will access extensions from a variety of paths and typically only one of those is the one that you as administrator will have access to adding to. See the "other extension locations" part of this page: https://docs.civicrm.org/sysadmin/en/latest/customize/extensions/

So Eileen is talking about providing extensions in a 'core code location' at 'civicrm/ext' which sounds like the one you can manage via the web interface, but isn't.