Recurring payments, advanced functions

2016-01-18 09:00
Written by

I've been working on a few larger CiviCRM migrations lately. Out of that work I've been adding some new pieces to the iATS payment processor extension as well as my more generic recurring payments extension, and here's my story.

The original CiviCRM model for recurring contributions was built around what let's call the Paypal model, which tries as much as possible to give the donor ownership and management of their payments, by managing the recurring schedule and giving donors their own 'account'. This has benefits for donors and also for the organization receiving the donations - e.g. less administration.

But I've found that for many organizations migrating to CiviCRM, this model isn't ideal. For a start, a lot of older regular donors don't want to manage their payments and assume it's the responsibility of the organization. Secondly, having contributions arriving somewhat erratically means it's harder to do reconciliation. And thirdly, larger organizations get used to doing things certain ways, and workflows become central to how they work, and changing them isn't always a good thing in practice, even when it seems like a smart thing to do in principle. Or, sometimes the customer is right, even if we wish they were wrong.

One part of the solution here is the 'token' model for recurring contributions, where the recurring contributions are managed by CiviCRM, instead of the payment processor. So now CiviCRM has more control over when and how much these recurring contributions come in, and for example - an organization can setup a phone campaign and ask donors to increase their monthly donation. We did this recently with an organization that had access to a phone banking system, to considerable success.

But of course, change inevitably brings unexpected new responsibilities and the "token" model requires some more tools of CiviCRM for things that were previously not part of its function. So I'm happy to report here on four new features that will be released shortly in the next (1.5) release of the iATS extension. Two of the features that are not specific processor-specific (i.e. the Additional Editable Fields and Fixed Schedules) are also available in the Recurring Contribution extension, though they are limited without any payment processor specific code (e.g. the Fixed Schedule feature will still take the initial payment when the payment is first submitted, whereas the iATS version will verify the card and wait until the first allowable date to make the first contribution).

The four new features are:

A) Card on File

This feature incorporates three separate functions related to management of a donor's card information. Of course, card information is not and should not be stored in CiviCRM - the whole point of the token system is to keep that sensitive data only with the payment processor. The token is the unique string that links a recurring contribution with the corresponding client information on the payment processor, and is used to trigger the sequence of donations with the use of a system job.

But that token can now also be used to:

1. View card information

Donor's sometimes call about their recurring donation. Having access to information like their first and last four digits of their card, the issuer, the name on the card and the expiry date, will inspire confidence in your relationship with the donor and will help you with the next task of updating card details as well.

2. Update card information

Cards will expire, and will also stop working before their expiry date for a variety of reasons. When that happens, you'll want to contact the donor and ask them if they want to continue their donations and if so, to give you their new card information. In the past, you'd have to do this via the iATS interface, but now you can do it directly from CiviCRM. If you're chasing down 100 expired cards each month, this is nice.

3. Ad hoc processing of a card

If a donor has missed a regular donation (e.g. due to an expired card), or if they just want to make a separate one-time donation, it's handy not to have to ask for their card information again. This feature gives you a simple form that allows you   to generate a one-time charge with the card already on file.

B) Fixed Schedules

By default, a recurring contribution schedule will start on the day the initial contribution is made and continue each month (or quarter, or 3 weeks, or whatever periodicity is configured and requested). But there are also good use-cases for setting the scheduling up to only happen on fixed (usually monthly) dates. This is particularly true of larger organizations, so that these 'batches' can be reconciled and accounted for more easily. It also helps larger organizations with scheduling staff time to follow-up with failures (e.g. so it becomes a monthly or bi-monthly task instead of having to chase down expired cards potentially every day).

This new feature allows an organization to configure specific days of the month for all recurring contributions. If enabled, a recurring contribution request will not go through immediately, but will be queued for the next allowed day. In addition, all recurring contributions will get tested and pushed forward if necessary to the next allowed day.

This new functionality opens a bit of a can of worms (e.g. what about allowing donors to choose their next donation? that's a feature not implemented yet), and really requires the next feature below to be useful.

C) Additional Editable Fields on Recurring Contribution RecoRDS

Every recurring contribution schedule has a corresponding record in the recurring contribution table that describes things like the amount, the payment processor, the next scheduled contribution, etc. In the default recurring contribution model, there's not much to do with the recurring contribution except look at it, since the schedule is managed by the payment processor.

But with our new "token" model, you generally would like to edit more things, and now you finally can. Also, by default the edit form can be a bit scary because it doesn't show you whose record you're editing, so it's nice now to get some extra information on the same screen, e.g., the name of the person, to be sure you're changing the right thing.

As an aside - you'll also want to be able to sort and search your recurring contribution records, and there's already an iATS report for this purpose that you can use to generate this, e.g. a list of all cards due to expire this week. A version of that report is also available in the Recurring Contributions extension.

D) Handling Recurring Failures

When the payment processor manages the recurring schedule, failed donation handling takes place with the payment processor, so your options as an organization are limited to their policy and practice.

With the first versions of the iATS processor, we set up failure handling very cautiously, judging that it's better to err on the side of fewer donations than annoying or angering your recurring donors.

But of course, there's no one size fits all, and it's nice for organizations to set their own policies and practices, and with the new release we've got a few new things happening by default, and also some new options for advanced usage.

You can read the documentation here, but the key things are:

1. Optional email notifications of failures (yes, by batch).

2. Different handling depending on the nature of the failure.

A more interesting thing is - you can now determine exactly how/when attempts are made to reattempt failed contributions. There are a couple of new switches in the recurring contribution job that make it appropriate to turn off the default recurring contribution job, and only run it on specific dates/times via the cron api method. This is particularly useful for fixed recurring dates, so you can for example, run one or two retries on specific days for each batch before giving up and then trying again the next month. You can get more details here.

All this, and maybe a bit more will be ready with the new iATS release 1.5, due out in about a week. Advanced testing welcomed! Did I mention you get all this for ACH/EFT (direct debit) as well? Yes, you do.

Click thumbs up if you thought this blog post was useful (login to vote or to comment)


Hi Alan - great blog. Thanks for the work on this and broadcasting about it. I wondered if it might help others to make it a bit clearer which are the developments which are iATS specific and which are part of the more generic extension (eg by making that clear in brackets at end of each sub-heading)

Hey Peter, thanks for the encouragement. All of the features are in the iATS extension, and the fixed recurring dates and additional field editing features are also available in the generic extension, though I'm not sure how useful they might be for other payment processors other than example code. Originally, I'd planned them as generic features for 'token' type recurring payment processors, and I do use the fixed date piece with Moneris in one install, but it doesn't really generalize safely. But your advice is good, I'll put this information in the blog post itself.

These features are now available with the 1.5.0 release of today.

Is iATS compatible with civicrm 4.7

The work you've done here actually mirrors what's been going on on the Drupal space, which I've been involved with due to my work on Aegir Site Subscriptions.

Implementing recurring billing in Drupal currently has two tracks:

  • Recurly, which is a third-party service.  It handles all of the above (including dunning) and allows for multiple payment processors.  So you could use Stripe for credit cards and PayPal for PayPal, for example.  They take an additional payment on top of the payment processor free.
  • Commerce Recurring Framework, which handles everything in Drupal internally, just like the CiviCRM work above.  It also uses the token method for card-on-file tracking, and runs individual transactions through payment processors whenever necessary.

So it's nice to see that CiviCRM is now in the do-it-yourself camp!

I sometimes wonder if this direction was actually a Good Thing. When things go well, then there's not really much to distinguish between a processor-managed schedule vs a civicrm-managed schedule, most or all of the functionality can be implemented in either assuming the processor provides the api for it.

The downside of civi-managed schedules come out when things don't work as planned: e.g. copies of the site generating extra contributions, or duplicate contributions due to server miscommuncation, or missing out on recurring contributions due to cron not running or other unexpected errors.

Of course, these things can also happen on processor-managed schedules, and then you have both less responsibility but also less control.

I think the preference ultimately has to come down to the organization themselves - i.e. if they want to manage the schedules, understanding the risks and responsibliities. The original impetus for the code was in fact a large organization migrating into CiviCRM where they were accustomed to managing the schedule and really didn't want to give that up. In their case, they didn't run the recurring on any automated schedule, but always triggered it themselves, which is something that does require CiviCRM to manage the schedule.