Good testing is critical for providing quality software. Some testing is always manual but writing automated tests to do the heavy lifting is a key part of modern coding practices. For developers, the discipline of writing automated tests helps to proactively identify and control edge-cases, and regularly running tests helps to identify problems quickly (before they grow and cause greater harm). Fostering a healthy culture of testing requires easy, consistent, timely test-results which in turn requires an easy, consistent, quick way to setup a new test environment. Easy, consistent setup of a test environment helps both (a) with individual developers who write and run tests on their personal sandboxes and (b) with teams which run tests in a continuous-integration environment.
This need is shared equally by the core team and many other active development teams in the CiviCRM community: many teams need to install and run tests with CiviCRM, a CMS, and a set of modules or extensions. Many of the technical details are identical or negotiable, although several key ones – such as the list of git repos and their placement in the file tree – are different.
With support from Wikimedia Foundation, we recently integrated Github's pull-request process with the CiviCRM's core test suite. Suppose a developer, Alice, submits a pull-request on Github. The "civicrm-builder" bot will automatically download the changes, execute a series of tests, and add an icon in Github (e.g. a green checkmark if the tests pass or a red X if the tests fail). In tandem with the implementation, we've also updated the documentation so that others can replicate the setup.
There are some security risks in testing unapproved patches, but we have two mitigations. Firstly, tests run in a dedicated testing sandbox. Secondly, Alice must be whitelisted. If Alice is a new developer who hasn't been whitelisted yet, then "civicrm-builder" will post a question: "Can one of the admins verify this patch?" To which an admin should usually respond with "ok to test" or "add to whitelist".
CiviCRM has a large test-suite -- including the web-based Selenium tests, the full suite can take 6 hours to execute, and it's not currently feasible to run the full-suite for every PR. In the current configuration, Jenkins executes tests in three stages:
As mentioned previously, running the full suite currently takes a long time -- which is why we execute the suite in three stages. As part of the work with WMF, we started scoping, discussion, and experimentation on how to improve the responsiveness of the test-suites. (See, e.g., http://forum.civicrm.org/index.php/topic,33608.0.html.) Fortunately, CiviCoop has stepped forward to fund more implementation on this. I'll post again as this matures.
If you've been using
test.civicrm.org, there are a few changes to notice:
Setting up similar testing (but with your own mix of applications, modules, themes, test-cases) can be useful -- but it can be also be difficult. A major stumbling block is automating the build (i.e. the download and installation of Civi+CMS+addons). Within our community, we have access to several great tools (such as Drupal's
drush and WordPress's
wp-cli), but none is a silver-bullet -- many community members work out their own unique mix of tools/workflows to meet their organization's particular requirements. (And -- thankfully -- some folks share stories about these tools and workflows in the forums, blogs, conferences, sprints, and IRC.)
For civicrm.org's infrastructure, the requirements for automating the build are fairly rigorous -- we need to support builds for automated-testing, demos, sandboxes, trainings, and sprints. We need access to different versions of Civi, Drupal, Joomla, and WordPress. We need to work within different operating systems (such as Linux and OSX) and within different virtualization schemes (such as Ganeti and Vagrant). As I've learned about useful tools/practices (through our community and through my own research), I've tried to incorporate them into civicrm.org's infrastructure and bundle the tools/scripts into a package -- civicrm-buildkit.
If you're interested in automating your build & test process, you should take a look at the documentation for buildkit. It includes tutorials for basic installations and for automated testing.
Fair warning: Buildkit is a bit biased towards civicrm.org's use-cases. It's not appropriate for every use-case, and the documentation is relatively new (with some TODOs). But even so, the documentation may provide useful references/examples/structures. Comments or questions would be greatly appreciated -- feel free to reply in this blog or post in the developer forum.