What?
CiviCRM core now offers a Continuous Integration service to partner & member organisations on their extensions. The service will run the unit tests associated with an extension whenever a PR is submitted to that extension. This is similar to the process that runs on core and is driven by jenkins and runs on the same test infrastructure. There are free services that do this (e.g Travis) on github in general but they don’t provide the CiviCRM setup and database environment that is needed to run tests on CiviCRM.
Why?
Basically because I wanted it & I paid the core team to do the technical work to make this an offering :-). This is something I’ve pushed for and I chose to fund it personally with financial help from Joe Murray from JMA Consulting to make it happen.
Erm still why?
I was finding that I was spending a lot of my own time dealing with regressions in various extensions that I have written such as Extended Reports and Omnipay - many of them as a result of me accepting community code contributions into them. I felt like it was worth it to me personally to spend some money to make this less painful. If you use these extensions you might like to contribute to the next phases to save me from having to.
What specifically is on offer right now?
Extension CI service on github - you can have up to 6 extensions under CI for $100 per month. (provided you are already a partner or member organisation) with a $500 set up fee . Extra extensions for $15 per month. Whenever a PR is submitted the probot listener will run the unit tests that you have written for the extension. You are responsible for getting the tests working & fixing any issues.
In general the setup fee pays for the development of the CI service to meet the above goals whereas the monthly fee goes towards the server resources and various admin & support overheads incurred. Currently we are in need of more server resources so a couple of sign ups would allow the Core team to purchase more capacity.
What is the road map on this?
There are 3 goals from my point of view
-
Extension CI service (as currently offered) to run CI on PRs submitted to those extensions
-
Extension CI service extended to also run the extension unit tests on latest master when a PR is submitted against master. These will be visually separated from the main core unit tests but if they start failing for an extension for reasons that relate to the extension, not a core change, they will be disabled until the extension author fixes them. I anticipate that once this is developed it would be included in the $100 per month
-
Fork CI service - many organisations have their own CiviCRM forks with patches they are still working on / upstreaming. This service would run the core unit tests against these forks as PRs are submitted to them. Tarballs would be generated for deployment, as happens on the main branch. This would have a separate pricing and would be more expensive than the CI service as it would have higher server resource needs.
Further technical development to be done as funded. The steps below are each expected to cost between $800 & $2000.
-
Update the service to support gitlab as well as github
-
Improve php version coverage - add in php 7.3, possible multiple 7.x versions
-
Update civicrm-core PR testing to go through the same civibot integration
-
Add CI on core PRs to extension service purchasers (depends on step 3)
-
Add support for running CI on forks (depends on step 3)
