Rethinking payment processing

2014-08-14 22:31
Written by

I have a few payment processing related projects on the go but having just gotten the first one to alpha stage I thought it would be a good time to share some thoughts (& try to get them straight in my mind). The project I am referring to is developing a Cybersource Secure Acceptance POST payment processor. For this payment processor I had a big challenge and also a big opportunity.

The challenge

The Cybersource Secure Acceptance POST gateway is one of an increasingly common breed known as transactional redirect or sometimes direct post method. The idea is that you gather most data on your site and then present the user with a credit card form on your site which POSTs to the Cybersource site. This way the credit card form is presented on your site and the customer never knowingly (depending on internet speed) leaves your site but none of the credit card information ever hits your server in any way. This gives you PCI compliance without the ugly redirect (& risk of the offsite form being fugly). The only problem is that CiviCRM isn't built for it.

The opportunity

In the not so distant past I discovered an exciting library of payment processor gateways - omnipay written by

Adrian Macneil

. Omnipay normalises your interactions with payment processors so that you need less code to integrate them and you can leverage contributions from an open source community that extends far beyond CiviCRM. In this case there was not an existing Cybersource gateway but someone called Rafael Diaz-Tushman had a shell of a repo which I submitted code to and now he is actively working on too. So, already we can see this has enable collaboration outside our community.

What works?

The 2 processors I tested are Paypal and Cybersource - although I tried a few others like BitPay, and Payment Express. Bitpay worked locally but not on a second site and I didn't figure out the problem with that or the others. Generally I didn't have the credentials / time to explore non-relevant processors too much. However, Paypal & Cybersource both have very different flows and use the same code. If you check the processors list there are about 35 processors there. In some cases they have endpoints for functionality like recurring and refunds that we haven't fully developed in CiviCRM yet.


The first thing I discovered working with Omnipay is that the maintainer has decided that thinking about processors in terms of off-site processors and on-site processors is not useful.

"Generally most payment gateways can be classified as one of two types:
  • Off-site gateways such as PayPal Express, where the customer is redirected to a third party site to enter payment details
  • On-site (merchant-hosted) gateways such as PayPal Pro, where the customer enters their credit card details on your site
However, there are some gateways such as Sage Pay Direct, where you take credit card details on site, then optionally redirect if the customer's card supports 3D Secure authentication. Therefore, there is no point differentiating between the two types of gateway (other than by the methods they support)."

This is quite a different approach to that adopted by CiviCRM which has very different flows for on-site and off-site processors. However, I quickly came to the conclusion that the different flows are not actually that helpful and only capture a small subset of payment processing possibilities. In fact the doTransferCheckout in my implementation of the OmniPay library looks like this:

function doTransferCheckout(&$params, $component = 'contribute') {
  $this->doDirectPayment($params, $component);
  throw new CRM_Core_Exception('Payment redirect failed');

The 2 key differences in how Core processes Onsite and Offsite payments are

1) Form Building differences

2) Pre-handover processing differences

Form Building differences

CiviCRM says 'if you are an onsite processor you need to have address fields + credit or debit card fields and I will create a billing address for you & for your contribution otherwise you don't need these fields'. As it turns out the form fields requirements of payment processors vary quite a lot and in the case of Cybersource - which in the end I had to get CiviCRM to 'think of' as an offsite processor you still need specific address fields (which are then signed). Conversely we get fairly frequent requests from people wanting us to remove billing fields as their processor doesn't require them and they prefer less fields. Basically core needs to 'ask' the processor which fields to display rather than decide based on the processor's 'type' combined with it's payment type. I achieved some of this within the extension (in particular replacing BillingBlock.tpl with a block that has no hard-coded fields in it & which displays fields based on variables assigned to the template rather than template logic. However, I feel like I'm only starting to think through the metadata involved in 'instructing the form layer' and this is something I'm going to write more about - probably in a second blog.


Pre hand-over processing differences

If you love a contribution set it free - if it comes back, change it to completed. If it doesn't - hey who knows maybe it's gone off-site & there will be an IPN later.

The above is my revised version of how CiviCRM should interact with the payment processor. In other (rather less opaque) words CiviCRM should build the pending contribution & related assets prior to handing over to the payment processor regardless of what 'type' it is and then update to completed based on 'what happens next'. How does this compare to current action? Well - at the moment CiviCRM creates a pending contribution if the processor is 'of type notifiy' but if it is of type 'form' it doesn't create one unless it is confirmed by the doDirectPayment function. This has a couple of effects. Firstly it makes the code more complex because the form layer is iffing & thenning all over the place. Secondly it makes it hard to create payment processors that don't fit standard flows.

What are the downsides of this flow change? Well, firstly this change shouldn't hurt existing processors (which is our main block to making some other changes I'm mulling) but there are 2 potential gotcha areas

1) Probably this would leave more failed transactions in the database and arguably this is a bad thing. At the moment failed credit card transactions are either not stored or rolled back out of the database - I've only been on the 'why do we lose our audit trail' side of this but there are others who prefer it this way. My suggestion would be to have a cron that cleans up failed transactions rather than not record them or introduce another setting as to whether they are desirable

2) Double transactions. Some processors create the option of a separate transaction for memberships vs contributions. I can't see this being able to be decoupled from payment processor type 'category' in the short term. In the longer term I think we could think about metadata & capabilities. I'm not going to go further into metadata & capabilities this blog but I do want to publish a blog on it

What does Omnipay Do

Basically it normalises your data & the interactions with the gateways. It doesn't matter what your gateway calls the First Name field - you set the same fieldname everytime and Omnipay takes care of that for you. So, here is my doDirectPayment function - that handles onsite processor Paypal, offsite processor Bitpay & transactional Redirect Processor Cybersource.

function doDirectPayment(&$params, $component = 'contribute') {
  $this->_component = strtolower($component);
  $this->gateway = Omnipay::create(str_replace('omnipay_', '', $this->_paymentProcessor['payment_processor_type']));
  $this->setTransactionID(CRM_Utils_Array::value('contributionID', $params));

  try {
    $response = $this->gateway->purchase($this->getCreditCardOptions($params, $component))->send();
    if ($response->isSuccessful()) {
      // mark order as complete
      $params['trxn_id'] = $response->getTransactionReference();
      //gross_amount ? fee_amount?
      return $params;
    elseif ($response->isRedirect()) {
      if ($response->isTransparentRedirect() || !empty($this->gateway->transparentRedirect)) {
        CRM_Utils_System::redirect(CRM_Utils_System::url('civicrm/payment/details', $response->getRedirectData() + array(
            'payment_processor_id' => $this->_paymentProcessor['id'],
            'post_submit_url' => $response->getRedirectURL(),
    else {
      //@todo - is $response->getCode supported by some / many processors?
      return $this->handleError('alert','failed processor transaction ' . $this->_paymentProcessor['payment_processor_type'], (array) $response, 9001, $response->getMessage());
  } catch (\Exception $e) {
    // internal error, log exception and display a generic message to the customer
    //@todo - looks like invalid credit card numbers are winding up here too - we could handle separately by capturing that exception type - what is good fraud practice?
    return $this->handleError('error', 'unknown processor error ' . $this->_paymentProcessor['payment_processor_type'], array($e->getCode() => $e->getMessage()), $e->getCode(), 'Sorry, there was an error processing your payment. Please try again later.');


Omnipay features

Omnipay gateways support different methods and it is possible to query the gateways to find out what methods they support. The most relevant methods are Purchase, Create Token, Refund and Complete Purchase. Purchase is obvious. Complete Purchase actually means 'handle IPN'. Once again I was able to use a very generic function - although there is some argy bargy with the $_REQUEST which was because it's possible to run this from the api. (I should check if that's required)

public function processPaymentNotification($params) {

  $this->gateway = Omnipay::create(str_replace('omnipay_', '', $this->_paymentProcessor['name']));
  $originalRequest = $_REQUEST;
  $_REQUEST = $params;
  $response = $this->gateway->completePurchase($params)->send();
  if ($response->getTransactionReference()) {
  if ($response->isSuccessful()) {
    try {
      civicrm_api3('contribution', 'completetransaction', array('id' => $this->transaction_id));
    catch (CiviCRM_API3_Exception $e) {
      if (!stristr($e->getMessage(), 'Contribution already completed')) {
        $this->handleError('error', $this->transaction_id  . $e->getMessage(), 'ipn_completion', 9000, 'An error may have occurred. Please check your receipt is correct');
  elseif ($this->transaction_id) {
    civicrm_api3('contribution', 'create', array('id' => $this->transaction_id, 'contribution_status_id' => 'Failed'));
  $_REQUEST = $originalRequest;

Refund and Create Token

These are the 2 areas I think offer most potential for development focus at the moment. At the moment we can't call an api to refund a payment & have that passed to the payment processor. With Omnipay the most difficult part of creating that api and having it work on multiple processors will be agreeing what to call it (anyone who has been party to API discussions will know that discussion is a forte).


Create token is the only option Omnipay offers for recurring payments. From the author

"At this stage, automatic recurring payments functionality is out of scope for this library. This is because there is likely far too many differences between how each gateway handles recurring billing profiles. Also in most cases token billing will cover your needs, as you can store a credit card then charge it on whatever schedule you like. Feel free to get in touch if you really think this should be a core feature and worth the effort."

My feeling is that we are seeing most growth in the area of tokens at the moment and a great focus would be to build an Omnipay cron for processing recurring contributions (based on tokens stored during checkout)


Omnipay forever?

The Omnipay library is well-written and well tested. It uses modern coding standards and continuous integration.  It's being built into Drupal 8 and there is Wordpress integration along with a fleet of others such as Silverstripe.The only other library I have found which seems compelling is Payum. Payum has a few more gateways, and some more features around logging and they have integrated Omnipay into their own code. It also looks well written. I didn't perceive a big feature difference between the two - not features that seem needed. So, to my mind it comes down largely to maintenance. Omnipay has greater penetration but seems to have a single maintainer with relatively simple needs "My main use case is a shopping cart, so most of the existing gateways have been built at a lowest common denominator level of interoperability" (!topic/omnipay/lcgkiKg7rt4) whereas Payum has less penetration but they seem to be building a business around developing Payum (it IS open source) so probably have more resources committed to it. So, the risk is that the Omnipay maintainer could get overwhelmed by what seems to be a pretty steady trickle of support requests. 

I'm trying to be even handed here but I've now spent a lot of time getting used to Omnipay and I like it and feel like it's a case of 'convince me why not Omnipay'.


The areas that I still need to work through is the aforementioned metadata because we are dealing with a situation where we are integrating multiple gateways and need to know things like 'which fields to I show for this processor', 'will this processor support 2 payments in one transaction'? and in the case of transparent redirect fields it needs to know things like whether to show a single field for expiry date or 2 fields. Some metadata options are features of the processor and others are features of your account (currency for example could be either). Another variant is Stripe which requires a javascript inclusion (and probably specific classes or similar).

Some other libraries that I didn't think were as promising

How do I add a processor (that has a Omnipay gateway) to Omnipay Multiprocessor extension

Basically add your processor to the .mgd file & see if it works ....

Note that I am using payment_type = 3 to denote that billing address fields should be displayed (but not credit card fields). The url fields are not used and the label fields need to reflect the gateway implementation - ie. the password label is 'Access Key' - which will translate to $gateway->setAccessKey('blah'); The name field is also important as it is omnipay_GatewayName

  'name' => 'OmniPay - Cybersource',
  'entity' => 'payment_processor_type',
  'params' =>
      'version' => 3,
      'title' => 'OmniPay - Cybersource',
      'name' => 'omnipay_Cybersource',
      'description' => 'Omnipay Cybersource Payment Processor',
      'user_name_label' => 'Profile ID',
      'password_label' => 'Access Key',
      'signature_label' => 'Secret Key',
      'class_name' => 'Payment_OmnipayMultiProcessor',
      'url_site_default' => '',
      'url_api_default' => '',
      'billing_mode' => 4,
      'payment_type' => 3,

Filed under


Could this or will this replace, then, the built-in Paypal payment processor?

Yes - eventually I think the built in processors would be replaced by this approach - would need quite a cross-over though

I like the idea of describing payment-processors with more fine-grained metadata instead of pushing them into 3 buckets.

Storing a token for follow-up transactions seems pretty flexible (e.g. you can define custom schedules for recurring payments; and you also use tokens to improve UX for manual renewals and upsells), and architecturally it seems simpler to understand the overall system if we have one token-based scheduler that works with many payprocs rather than fragmented scheduling rules. I'm not sure if what the flipside (e.g. what are the advantages of using the payproc's scheduler?).

It does seem better in the long-run to build on a payment-processing library like Omnipay rather than maintain our own.

Re the 'flipside' - I'm not sure that many processors offer both scheduler & tokens. Currently Paypal & use the scheduler - but I'm not sure if they offer tokens. More recent implementations seem to offer tokens only.


The upside and downside of tokens is you are reasonable for your own schedule - if you can't manage your cron you have a problem 

Eileen - I think the use of a payment processing library sounds like a good strategy, and should increase the number of payment processors available to CiviCRM. Esp. if the library chosen is widely used. SInce OmniPay is built into Drupal 8 (in core? Or as a community module?) and used with WordPress, that sounds promising. 

Regarding, they offer both a scheduler and tokens. Their API for tokens is called "CIM" and is described at: Customer Information Manager (CIM)


This is a very interesting post, touching on several of the issues we experienced with the existing Payment Provider mechanism while building our SEPA Direct Debit extension:

1. The need to ask the Payment Provider implementation what fields to display. This is crucial for SDD, as the fields are very different not only from the Credit Card fields, but also from the existing Direct Debit ones.

Indeed there is not much point in having a "Direct Debit" payment type at all: unlike for Credit Card payments, there is no more-or-less standard set of fields used for all card types and by all providers; but rather, each direct debit scheme uses different fields.

Implementing the SEPA-specific fields on the online (fronted) Contribution Pages was not too tricky, as CRM_Core_Payment has its own special buildForm() method to override -- so only little hackery was needed so form validation still works etc. This mechanism doesn't work for the various back-office contribution forms though: instead we need to hack the various forms in a host of ugly ways through the general hook_civicrm_buildForm().

(I tried to raise this issue on our SEPA project list, constituted of the people implementing the SEPA extension, as well as a number of people working with other DD schemes, and some core team members... But there was no resonance at all, so I didn't pursue this further.)

2. Different flows for different billing types. This is actually even more confounded, as there are even differences between single and recurring Contributions for the same billing type! (Quite arbitrary ones as far as I can tell.) Some standardisation would be greatly appreciated.

3. Cron Job for scheduling recurring contributions. There is one (rather basic) in the Offline Recurring Payments extension; and we created another one for our SDD extension. A generic way to handle this kind of scheduling for any payment method that needs it would be very useful.

Oh, yes please!

This all sounds excellent. Getting the right abstractions for a payment processor is hard, using someone else's library completely makes sense, and the current CiviCRM system is not good.

Some brief notes:

1. tokens vs. IPN - I'm obviously a fan of the tokens approach, and implementing the logic in CiviCRM not only gives you more options about how to set up a recurring schedule, it also gives you more flexibility in changing a schedule. As a simple example - you may get a monthly donor who's only willing to do it for a year, initially. If you can phone them up and convince them they should keep going, it'll be easier if they don't have to refill out a form. In other words, exactly what Tim said.

On the flip side, there is a particular case that I'm working on now that is much easier if the payment processor handles recurring payments, and that's the UK recurring direct debit. The reason is that there is a lot of legal-type overhead in each payment, and letting the payment processor do that work is nice. I should note that this recurring payment processor (iATS Payments) doesn't use IPN, but instead provides a reporting system for the recurring payments, so it doesn't actually need anything special from the abstract payment processor.

So maybe the right answer is just to ensure that the code offers, but doesn't require a token system for those recurring payments, without making any further assumptions.

2. The form building issue. Although the direct debit form is really unuseable in it's default form, it's not a terrible idea to have - just consider it a kind of interface that needs to be implemented. And the form hooks work well.

3. Pre-handover processing. Seems to me there's a compromise in there to make both developers and bookkeepers/administrators happy - some kind of queue of potential payments that get converted to real payments once there's something in return, or send them off into a slowly decaying graveyard otherwise.

Just a quick note to say thanks for the great responses - I'm still filtering them through the mush in my cranial cavity & will follow up with another blog shortly

Hi!  Where is the documentation for how to configure the IPN for Cybersource with this extension?

You need to follow the standard CiviCRM instructions for

( is a brand owned by Cybersource, I am assuming that is what you are looking to details on.)  The extension only impacts, PayPal and eWay recurring. 


Here's clarification on a few items:

These are mandatory fields under card absent acceptance rules, so it's not up to processors or merchants to decide what they want to provide. How they respond to the data may vary, but they all need to submit it.

  • The card account number
  • The name as it appears on the card
  • The card expiration date as it appears on the card
  • The cardholder’s statement address

RE: separate transaction for memberships vs contributions: Merchants must send the correct processng code indicator (sale, recurring, installment, etc- Reference MasterCard or Visa Processing Rules).  Will the Gateway manage it automatically?  Will the specific form trigger it? This is something you'll need to explore, even if Civicrm manages triggering recurring payments.

What to call refund? It's important to recognize the differences. A return can be linked to a prior sale and use the prior authorization, or unlinked and there is no prior auth code or the amount of refund exceeds original sale amount. 

Some gateways call the first a refund, or return, and the latter a credit; others call it them both a credit and just omit prior sale data for unlinked transactions. Why does this matter? Because PCI 3.0 requires merchants limit transaction functions by user. Very few people should ever be allowed to perform a credit (unlinked). I suggest you use REFUND (requires prior auth) and CREDIT (unlinked or exceeds).  This will support gateways that manage limiting by account set up as well as by user roles.

Some processors are providing physical card swipe devices that work with their CiviCRM extension. ( such as the latest version of the iATS extension). Seems like this could be treated as a "card present" transaction.