Good testing is critical for providing quality software. Some testing is always manual but writing automated tests to do the heavy lifting is a key part of modern coding practices. For developers, the discipline of writing automated tests helps to proactively identify and control edge-cases, and regularly running tests helps to identify problems quickly (before they grow and cause greater harm). Fostering a healthy culture of testing requires easy, consistent, timely test-results which in turn requires an easy, consistent, quick way to setup a new test environment. Easy, consistent setup of a test environment helps both (a) with individual developers who write and run tests on their personal sandboxes and (b) with teams which run tests in a continuous-integration environment.
This need is shared equally by the core team and many other active development teams in the CiviCRM community: many teams need to install and run tests with CiviCRM, a CMS, and a set of modules or extensions. Many of the technical details are identical or negotiable, although several key ones – such as the list of git repos and their placement in the file tree – are different.
With support from Wikimedia Foundation, we recently integrated Github's pull-request process with the CiviCRM's core test suite. Suppose a developer, Alice, submits a pull-request on Github. The "civicrm-builder" bot will automatically download the changes, execute a series of tests, and add an icon in Github (e.g. a green checkmark if the tests pass or a red X if the tests fail). In tandem with the implementation, we've also updated the documentation so that others can replicate the setup.
Testing and Security
There are some security risks in testing unapproved patches, but we have two mitigations. Firstly, tests run in a dedicated testing sandbox. Secondly, Alice must be whitelisted. If Alice is a new developer who hasn't been whitelisted yet, then "civicrm-builder" will post a question: "Can one of the admins verify this patch?" To which an admin should usually respond with "ok to test" or "add to whitelist".
Testing and Completeness
CiviCRM has a large test-suite -- including the web-based Selenium tests, the full suite can take 6 hours to execute, and it's not currently feasible to run the full-suite for every PR. In the current configuration, Jenkins executes tests in three stages:
- A code-review test (CiviCRM-Core-PR) which runs as soon as a new change is proposed through Github. At the moment, this runs a mix of tests (e.g. the APIv3 plus a handful of CRM tests, web tests, and upgrade tests).
- A scheduled test (CiviCRM-Core-Matrix) which runs every 6 hours using the latest, approved, canonical code. This includes the all APIv3 tests, CRM tests, and upgrade-tests.
- A scheduled test (CiviCRM-WebTest-Matrix) which runs every day using the latest, approved, canonical code. This includes only the web-test suite.
Testing and Performance
As mentioned previously, running the full suite currently takes a long time -- which is why we execute the suite in three stages. As part of the work with WMF, we started scoping, discussion, and experimentation on how to improve the responsiveness of the test-suites. (See, e.g., http://forum.civicrm.org/index.php/topic,33608.0.html.) Fortunately, CiviCoop has stepped forward to fund more implementation on this. I'll post again as this matures.
Changes
If you've been using test.civicrm.org, there are a few changes to notice:
-
Job reorganization
- Old: There were separate jobs for each combination of version and test-suite -- which produced about ~12 jobs ({4.2,4.3,4.4,master} x {UnitTests,UpgradeTests,WebTests} =~ 4 * 3 =~ 12).
- New: These jobs are consolidated. Thus, one job ("CiviCRM-Core-Matrix") replaces every combination of {4.3,4.4,master}x{UnitTests,UpgradeTests}, and a second job ("CiviCRM-WebTest-Matrix") replaces every combination of {4.3,4.4,master}x{WebTests}.
- Exception: The tests for v4.2 retain the old pattern because the underlying git structure is different.
-
Git and CiviBuild summaries
- Old: To see a breakdown of the specific git respositories/revisions in a build, one would look at "Changes".
- New: To see a breakdown of the specific git respositories/revisions in a build, one would look at "CiviBuild".
Documentation
Setting up similar testing (but with your own mix of applications, modules, themes, test-cases) can be useful -- but it can be also be difficult. A major stumbling block is automating the build (i.e. the download and installation of Civi+CMS+addons). Within our community, we have access to several great tools (such as Drupal's drush and WordPress's wp-cli), but none is a silver-bullet -- many community members work out their own unique mix of tools/workflows to meet their organization's particular requirements. (And -- thankfully -- some folks share stories about these tools and workflows in the forums, blogs, conferences, sprints, and IRC.)
For civicrm.org's infrastructure, the requirements for automating the build are fairly rigorous -- we need to support builds for automated-testing, demos, sandboxes, trainings, and sprints. We need access to different versions of Civi, Drupal, Joomla, and WordPress. We need to work within different operating systems (such as Linux and OSX) and within different virtualization schemes (such as Ganeti and Vagrant). As I've learned about useful tools/practices (through our community and through my own research), I've tried to incorporate them into civicrm.org's infrastructure and bundle the tools/scripts into a package -- civicrm-buildkit.
If you're interested in automating your build & test process, you should take a look at the documentation for buildkit. It includes tutorials for basic installations and for automated testing.
Fair warning: Buildkit is a bit biased towards civicrm.org's use-cases. It's not appropriate for every use-case, and the documentation is relatively new (with some TODOs). But even so, the documentation may provide useful references/examples/structures. Comments or questions would be greatly appreciated -- feel free to reply in this blog or post in the developer forum.
