Not a well known fact until now - during 3.0 release cycle, we've been working hard with DharmaTech to start regularly using unit tests for CiviCRM development and quality assurance. It's time to go out of the closet, since we're nearing the end of the first stage of this project - which was setting up everyone in the team with proper tools and migrating all the tests that we wrote before to our new framework.
So here's what we have right now:
- A list of tests that we have right now is here http://tests.dev.civicrm.org/results/ It's not fully green (meaning - some tests still fail), since we're still in the process of migrating old tests and fixing them, but we hope to get to "all green" state soon. This page is being regenerated every 30 minutes (for now) and we'll be using it to continually check for regression bugs.
- Code coverage report is here http://tests.dev.civicrm.org/coverage/ - we are doing quite good with API tests especially, coverage has been increasing steadily in last weeks. This should make our API more stable and reliable for integrators to work with.
Once we're finally done with current stage (finishing the migration), next steps will be:
- Starting to run tests against 3.0 (stable) code and for trunk (3.1 development) in the same time. This way, we'll see the regression happening on the trunk as new features jump in and will be able to wipe out some of the newly introduced problems long before alpha testing.
- Work on increasing the coverage for existing tests - the goal for 3.1 is API tests code coverage at 80% (at 59% currently), BAO tests code coverage at 50% (at 43% right now).
- Introducing browser tests (aka web tests). We'll be using Selenium and PHPUnit for this and hope for significant time savings on quality assurance we run before each stable release.
I'll be writing about this more, but will start with short request now: if you're using CiviCRM, filing issues and fixing bugs, please consider also helping us out with our unit tests. Filing an issue which has a test case attached speeds up fixing the bug considerably and also ensures that the similar bugs gets wiped out in the future in case they show up again during development of new versions. For now, if you want to get started, here are the instructions on how to set up the unit testing sandbox for CiviCRM.