We've started to look into changing how objects get persisted in CiviCRM and what can we do to make things easier for people extending CiviCRM. Part of our approach is to try and integrate Doctrine into CiviCRM.

Background

We get a lot of requests for customization and feel that the current CiviCRM customization infrastructure doesn’t meet most of our customization needs. We get a lot of requests to customize event checkouts and one of the things we frequently want to do is modify what happens at the end of the checkout process. A recent example is a subscription system where subscribers get a number of free passes to an event. We did this by manipulating some of the internal form variables to not do the payment part of checkout. The main problem with this approach is that we don’t really expect it to survive updates over a long period because it uses some internal mechanisms.

We’d like to fix this situation by separating out a lot of the object configuration work (creating participants and contributions) from the persistence work (saving the participants and contributions to the database). Currently, a lot of the configuration code is interspersed with the persistence code. This makes it difficult to reuse code. For example, if I call CRM_Event_BAO_Participant::create(...) to create a participant record and I want it to do everything in there, but one small part, I need to add some arguments to that function or copy it and not include the part I don’t want. This is because the function persists the participant (as well all some other objects). If we take the persistence part of that function and just returns the objects that we want to create, the caller can then manipulate the objects before they get persisted making the one small change.

Approach

We talked with some members of the core team and decided to try incorporating Doctrine into CiviCRM. The main thing that Doctrine buys us is that it handles associations between objects very nicely. This makes it easy to build up a tree of objects in lower level generic routines, and then let upper level routines manipulate the objects, and then have the objects get persisted.

We wanted to do this with a representative, but small, part of the code so that we could get it done in a reasonable amount of time and provide a realistic model for this approach going forward. Because we work with the event checkout system a lot, we decided to try and rewrite CRM_Event_Form_Registration_Confirm::postProcess to use doctrine because that’s where most of the persistence in the event checkout process happens.

The rewriting should follow this general pattern:

1. Create Doctrine objects instead of using the DBObject objects and static functions.

2. Set all the properties and relationships between the Doctrine objects in memory without persisting.

3. Call appropriate hooks for the action being taken passing the Doctrine objects and accepting a list of objects back.

4. Persist the objects returned from the hooks.

5. Call appropriate hooks for actions that need to happen after persistence.

6. Run any standard after persistence actions (send emails, show page, etc…)

Results

We rewrote about 25% of the CRM_Event_Form_Registration_Confirm::postProcess function so far and we wanted to share our progress so far before the Dalesbridge sprint was over. The current code is up at https://github.com/giant-rabbit/civicrm-core/tree/doctrine right now. The checkout process isn’t fully working, but we have a unit test setup for that function that at least gets through all the parts we’ve rewritten.

Here’s a look at what happens to  CRM_Event_Form_Registration_Confirm::updateContactFields when we change to this approach:

It gets a lot simpler. Partly this is because some of the code moves elsewhere. The code that checks to see if this contact is already in the group gets handled in some code I added to addToGroups method in the Contact entity that Doctrine generates.  That looks like this:

 public function addToGroups($groups)
   {
       $new_group_contacts = array();
       foreach ($groups as $group) {
           $found = FALSE;
           foreach ($this->getGroupContacts() as $group_contact) {
               if ($group_contact->getGroup() == $group && $group_contact->isActive()) {
                   $found = TRUE;
               }
           }
           if (!$found) {
               $group_contact = new Civicrm\GroupContact();
               $group_contact->setStatus('Added');
               $group_contact->setGroup($group);
               $this->addGroupContact($group_contact);
               $new_group_contacts[] = $group_contact;
           }
       }
       return $new_group_contacts;
   }

The nice thing about moving code into the Contact entity method is that this fairly general purpose code that prevents subscribing more than once to the same group is now easier to use in more than one place. Also it’s now decoupled from a lot of other code that did other things like send confirmation emails.

The other thing that happens to the code that was in updateContactFields is that it is going to move to the end of the postProcess function like this:

/* After all of the objects created with this event registration have been persisted */

foreach ($this->new_group_contacts as $new_group_contact) {

 CRM_Group_Emails_Confirmation::send($new_group_contact);

}

Another spot to look at is CRM_Event_Form_Registration_Confirm::processContribution. There’s not quite as much code reduction in this part, but I do think it gets easier to read.

One thing that I think will be really nice when using Doctrine is that we can setup methods to handle a lot of the standard text to lookup value stuff to make it easier to use.

This:

$contribParams['contribution_status_id'] = array_search('Pending', $allStatuses);

Becomes:

$contribution->setStatusByName('Pending');

Doctrine Implementation

Doctrine has a bunch of ways to work with it, so we’ll layout some details about how we set it up to work with CiviCRM.

All the Doctrine files live in ‘src’. Doctrine files are split into two main categories: metadata and entities. The metadata lives in src/Civicrm/metadata and the entities live in src/Civicrm.

Metadata

I chose YML for the format of the metadata files because I like YML. No real technical reason there. The YML metadata files define the structure of your data. Doctrine has a tool which can generate the metadata from an existing database and I used that. It isn’t perfect though. It doesn’t seem to pick up one-to-many relationships. So we have to add those manually, here’s an example of how I added phones to contacts in src/metadata/Civicrm.Contact.dcm.yml:

  oneToMany:
       phones:
           cascade: ['persist']
           mappedBy: contact
           targetEntity: Civicrm\Phone

Entities

Once you have the metadata setup, you can use that to generate entities. One big difference between the generated DAO objects and Doctrine entities is that you don’t split the entity code into the generated file (DAO) and the file you add your additional methods (BAO). Doctrine leaves any code you put in the entity file alone when you regenerate your entities. Continuing the example from above, once I added that code to the metadata file, I ran ./vendor/bin/doctrine orm:generate:entities which added the addPhone, removePhone, and getPhones methods to the Contact entity. I then added this line to the addPhone method:

$phone->setContact($this);
That line ensures that when you persist the contact, the phone also gets persisted with a reference to this contact.

Regenerating the entities won’t mess with that code I added to the addPhone method. Because of this sometimes if you make certain changes to the metadata, you need to remove methods from the entity manually. I’ve had to do this a few times and it’s no problem.

Repositories

Repositories are another major component of Doctrine that we are making use of. Repositories are basically collections of queries. Instead of putting queries inside the entities, Doctrine puts them in Repository classes. I have an example one that I did to make it easy to get a specific option value out of the civicrm_option_value table:

https://github.com/giant-rabbit/civicrm-core/blob/doctrine/src/Civicrm/OptionValueRepository.php

You can see how it gets used in the Basic Usage section below.

Basic Usage

I created a unit test to test the work I’m doing on the CRM_Event_Form_Registration_Confirm::postProcess function and it provides a good example of how to use Doctrine in the CiviCRM code base. This code creates an event with the minimum associated objects to do a credit card registration.

   $entity_manager = CRM_DB_EntityManager::singleton();
   $config = CRM_Core_Config::singleton();
   $domain = $entity_manager->getReference('Civicrm\Domain', 1);
   $civi_event_component = $entity_manager->getRepository('Civicrm\Component')->findOneBy(array('name' => 'CiviEvent'));
   $payment_processor_type = $entity_manager->getRepository('Civicrm\PaymentProcessorType')->findOneBy(array('name' => 'Dummy'));
   $payment_processor = new Civicrm\PaymentProcessor();
   $payment_processor->setBillingMode(1);
   $payment_processor->setDomain($domain);
   $payment_processor->setIsActive(TRUE);
   $payment_processor->setIsDefault(TRUE);
   $payment_processor->setClassName('Payment_Dummy');
   $payment_processor->setName('Test');
   $payment_processor->setPaymentProcessorType($payment_processor_type);
   $entity_manager->persist($payment_processor);
   $entity_manager->flush();
   $price_set = new Civicrm\PriceSet();
   $price_set->setDomain($domain);
   $price_set->setName("Test");
   $price_set->setTitle("Test");
   $price_set->setExtends($civi_event_component->getId());
   $price_field = new Civicrm\PriceField();
   $price_field->setName('Test');
   $price_field->setLabel('Test');
   $price_field->setHtmlType('Text');
   $price_field->setIsActive(TRUE);
   $price_set->addPriceField($price_field);
   $price_field_value = new Civicrm\PriceFieldValue();
   $price_field_value->setName('Test');
   $price_field_value->setLabel('Test');
   $price_field_value->setAmount(22);
   $price_field_value->setDeductibleAmount(0);
   $price_field_value->setIsActive(TRUE);
   $price_field->addPriceFieldValue($price_field_value);
   $participant_role = $entity_manager->getRepository('Civicrm\OptionValue')->findOne('participant_role', 'Attendee');
   $financial_type = $entity_manager->getRepository('Civicrm\FinancialType')->findOneBy(array('name' => 'Event Fee'));
   $event = new Civicrm\Event();
   $event->setTitle('CRM_Event_Form_Registration_ConfirmTest');
   $event->setIsActive(TRUE);
   $event->setIsOnlineRegistration(TRUE);
   $event->setIsMultipleRegistrations(FALSE);
   $event->setIsMonetary(TRUE);
   $event->addPaymentProcessor($payment_processor);
   $event->addPriceSet($price_set);
   $event->setDefaultRoleId($participant_role->getValue());
   $event->setIsEmailConfirm(TRUE);
   $event->setFinancialType($financial_type);
   $entity_manager->persist($event);
   $entity_manager->flush();

Problems

We’re happy to report that Doctrine works well alongside the existing CiviCRM code base. It even handles very nicely some difficult areas like the civicrm_price_set_entity which has a foreign key to different tables based on the entity_table column (see Civicrm.PriceSetEntity.dcm.yml). The only problem I’ve had with so far are fields that store foreign keys in a text field like the payment_processor column in civicrm_event. I think that’s possible to do using some of Doctrine’s hooks, but it’ll need some more time as working inside the hook I think would work well is a little bit complicated.

What’s Next

Assuming the CiviCRM community thinks that this is a good way to go, the first step would be to get the Doctrine code integrated into the code base. What I have in the Giant Rabbit github repository is nearly there. The only thing that needs to  be done is that I’m using a new class called CRM_DB_Settings to store the database settings and I think we need to get that loaded from the CIVICRM_DSN constant to keep the settings file backwards compatible. Once that is done, people can just start using Doctrine most places inside CiviCRM.

Because we rushed our plan a little bit to get some of the work done inside the postProcess function before the Dalesbridge sprint, we want to go back to the tests and get tests written that cover all the branches in the existing postProcess code. Once that’s done we’ll pick back up and finish converting the postProcess method to use Doctrine.