CiviCRM is a great fit for mission-driven organizations because it delivers what proprietary platforms typically can’t: true ownership of your data and the ability to meet unique business requirements. But, as we know from experience, the more complex the system, the more complex the upgrade process can be. “Complex” isn’t bad, but it should be planned for when looking ahead at projected costs for maintaining system security and functionality.
New releases of CiviCRM core, extensions, and your CMS (Drupal, WordPress, Joomla, or Backdrop) arrive regularly. CiviCRM core releases always go through rigorous testing and review before being released. However, if your system has grown more complex over time, each upgrade can unintentionally impact critical and customized functionality. And while it can save time sometimes to follow a “let’s deploy and see what happens” strategy, we don’t recommend it. It only works if nothing breaks. Nobody is happy when there are surprises.
A successful upgrade protects your stakeholders’ engagement with your organization:
- Donations process flawlessly and receipts include the right tokens
- Memberships renew on schedule with correct statuses, end dates, and financial types
- Event registrations honor price sets, discounts, and capacity limits
- Recurring contributions keep billing without gaps
- Profiles and custom data behave as expected on public forms and staff screens
- Scheduled jobs (like reminders and renewals) run as planned
- Integrations with external systems, SSO, or accounting are intact
Replacing manual testing with automated tests using Behat
Clicking through every contribution page, membership type, event, message template, and custom feature is slow, inconsistent, and expensive. Depending on changes in the version release, it can also delay important security updates from making it to your production site.
To address this, organizations can leverage Behat, a behavior-driven testing framework. Behat expresses expectations in human language and verifies the result.
- Tests are outcome-focused by design: Tests assert that a contribution, membership, participant, or activity was actually created with the correct values, not just that a “Thank you” page appeared.
- Tests are readable by non-developers: Project managers and program staff can validate that scenarios reflect real-world workflows.
- Test early and often: Tests can be executed before and after upgrades, in a developer’s local environment, or on a dev or a staging instance, as well as anytime a change or new feature is implemented.
- Behat can be integrated with BrowserStack’s Selenium grid to run the same scenarios on various browsers and devices (Chrome, Firefox, Safari, Edge; desktop and mobile).
- Tests can trigger emails to verify that they were sent with the correct tokens, subjects, recipients, and language. You can also use a capture tool like MailHog/Mailpit, or your email provider’s sandbox API, so tests aren’t sent to real inboxes.
Tips for Writing Tests
The way the tests are written matters, keeping the tests relevant and stable as UI and configuration changes are made. Here are key things to keep in mind for writing with the right level of detail:
- Assert recorded results (contributions, memberships, participants, activities, messages sent), not presentation details.
- Use stable selectors or data attributes on public forms
- Seed realistic data (membership types, price sets, custom fields) so scenarios reflect the complexity of your configuration.
- Capture screenshots and logs on failure to speed up fixes.
It’s important to consider when having a human at the computer makes more sense. Writing and designing tests takes time. This time should be balanced with the level of effort, accuracy, and risk of manually testing the same functionality. The recommended result for most organizations with complex systems is a hybrid testing process, relying on both manual and automated testing.
Reach out to Skvare if you're interested in learning how automated upgrade testing could help with your future CiviCRM upgrades. Or, share this article with your current service provider.
If your organization has complex needs and you’re still evaluating CiviCRM, or you’re about to get started, keep automated testing in mind. It will help future-proof your system and support the development of a clear management plan for upgrades and ongoing maintenance.
