Developers: Extension Testapalooza

Automated tests are important when collaborating with other developers in a large project. Even if you focus your attention on a small piece of the puzzle, your piece depends on other pieces, and others may depend on you. There will be inevitable occasions when a change in one causes an unexpected change or break in another. Automated tests form the first line of defense, providing timely feedback so that problems can be addressed while the material is mentally fresh.

Testing CiviCRM is trickier than testing a basic library -- tests may involve system services (from Civi or the CMS), and CiviCRM developers may use different CMS's, file structures, and URLs. This problem can be mitigated by creating more configuration files (for each extension, test-suite, or installation), but that grows unwieldy with multiple extensions.

We've published some updated documentation and tooling to support tests in extensions. The remainder of this post assumes that you have previously used civix to create a module extension.

Install or update developer tools

The dev tools have been updated for better test support, so make sure you're up-to-date.

If you use civicrm-buildkit, you should update buildkit.

If you manage your tools manually, then install or update civix, cv, and phpunit 4.x. Depending on your environment and previous installations, you may need to do some mix of:

• Set up command-line PHP (esp OSX + Windows)
• Disable or remove the old copies of civix, cv, and phpunit.
• Download PHAR executables for civix, cv, and phpunit 4.x
• Create wrapper commands for each PHAR (Windows) (Note: Windows hasn't been tested specifically and isn't officially supported.)
• Setup cv variables ("Setup: Option B: Manual")

Finally, ensure that you have an up-to-date developer installation of Civi (e.g. based on git with the master branch and a recently-generated copy of civicrm.settings.php). These conditions are met implicitly if you do a new build with civibuild.

Create and run a PHPUnit test

In an existing extension, run civix generate:test:

cd org.civicrm.civixexample
civix generate:test CRM_Civiexample_FooTest
# Write (snip)/org.civicrm.civixexample/phpunit.xml.dist
# Write (snip)/org.civicrm.civixexample/tests/phpunit/bootstrap.php
# Write (snip)/org.civicrm.civixexample/tests/phpunit/CRM/Civiexample/FooTest.php

By default, this test was created from a template called headless. A headless test boots CiviCRM once with a headless database, and all work can be executed in-process. These are faster, don't interfere with your live site, and support automatic cleanup, but they provide a less thorough simulation of real-world systems.

To run a more realistic simulation, you can generate an end-to-end (e2e) test by passing the --template=e2e option. An end-to-end test boots the live installation of CiviCRM and the real CMS. This provides a more thorough simulation, and you may spawn multiple requests to Civi using HTTP or cv() (as real users would). However, spawning separate requests will be slower, and data-cleanup may take more effort.

Regardless of which template you use, you can run the new test with phpunit, e.g.:

phpunit4 tests/phpunit/CRM/Civiexample/FooTest.php

To run the full suite, I suggest running all headless tests as a group; and then running all e2e tests as another group.

phpunit4 --group headless
phpunit4 --group e2e

For working examples and more descriptions, see the Testapalooza PHPUnit Example Extension.

Old PHPUnit tests

Previous versions of civix generated tests based on different conventions (e.g. using CiviUnitTestCase and the civix test runner), and a small number of extensions may have used these conventions. They should still work as before, and there's no plan to explicitly break them, but I don't plan to work hard to preserve support in the future. And the new conventions are better.

To update to the new conventions, see UPGRADE.md for civix:

https://github.com/totten/civix/blob/master/UPGRADE.md

Alternative test tools

PHPUnit is the test framework used in CiviCRM (core), Drupal 8, and many other large PHP projects, but there are other, newer test tools which have compelling features and use-cases -- such as Codeception, Behat, Mink, phpspec, and protractor.

If you'd like to use one of these for testing CiviCRM, read https://github.com/civicrm/org.civicrm.testapalooza/ for general, framework-independent commentary.