CiviCRM Multi-Org: Merging Groups with Relationships

CiviCRM Multi-Org refers to the ability to support multiple owner organizations (usually a hierarchical structure of umbrella orgs and sub-orgs or local chapters) and keep some data siloed in each org while allowing other users to aggregate data across orgs too. It has been been a tough nut to crack. But I and the organization I work for (The Public Interest Network) are committed to getting it working in CiviCRM 2.3. At the recent developer camp in San Francisco, we had 2 great brainstorming sessions on multi-org, which resulted in this wiki page: http://wiki.civicrm.org/confluence/display/CRM/Multi-Organization+Support+in+CiviCRM. For many multi-org use cases, one of the currently-possible options listed on that page could very well work. However, I need a little more flexibility, and I also want better multi-org support "out-of-the-box" (the current options involve custom fields and ACL hooks), especially with regards to usability. The current multi-org options and several of the implementation ideas that have been discussed suffer from very poor usability. The main problem is that they rely on custom fields, ACL hooks, and/or relationships w/ ACLs in order to model the multi-organizational structure. This becomes an issue because it requires that you model your organizational structure in 2 different places (once in groups so you can use CiviMail and other tools that expect groups and again in the custom field values or the relationships). Yes you can create smart groups for the non-group properties, but we're already struggling with "smart group explosion" since they are used for so many things in CiviCRM. We need to be able to model our organizational structure in one place so as people use the system, they just put contacts into the right place w/ the right properties once and everything falls into place. After the dev camp sessions and further brainstorming with Jason Bertolacci of IMBA (who is also here in the Denver/Boulder area), we came up with a potential solution. This is option #6 on the wiki page referenced above, "Get rid of groups as they currently exist in the data layer and use relationships instead. Keep the groups interface / views intact." Jason and I realized that groups and relationships were actually redundant at the data model level. In fact, groups could be thought of as relationship types and vice versa. So, for example, the "Administrators" group could be modeled as "everyone with the 'Administrator of' relationship to that particular Organization contact" (this is where the multi-org piece comes in, since you can have many different Organization contacts, and the groups they "own" would be the contacts w/ relationships with them). We did NOT want to get rid of the UI for using and managing groups and replace it with the equivalent relationship tools. Instead we wanted to keep the groups concept as it exists now in the interface, but model it at the data layer as relationships. This gives us a foundation upon which we can build a much more intuitive, powerful, and flexible multi-org implementation (as well as some other nice benefits). The downsides are that it could potentially have performance issues due to relationship type and relationship explosion. However, this can be mitigated by implementing caching for relationships similar to what groups already use. Relationships could also be further table-sharded by their type, for example (i.e. Individual-Individual, Individual-Organization, and Organization-Organization could each have their own table). It could also have some interface usability issues in cases where you don't want a relationships to show up as a group and vice versa. Our ideas for dealing with that were: 1. Only relationship types where the contact type on the right side (the B contact) is Organization would show up as groups. Or... 2. Introduce a new concept of "root contact" for each user in the system, and then only relationships where the B contact is this root contact show up as groups for that user (will also require a way to drill down into others' groups for users w/ permission to do so). Relationship types that have no relationships to your root contact would not show up in your group list until or unless you or someone else created such a relationship. This has some interesting side effects that could be very useful to many users. One example is that, by default, you could have everyone's root contact be their own Individual contact record. Then if you have a volunteer coordinator who has a set of volunteers with the "Volunteer Coordinated by" relationship to them, those people will show up as a group for that user (so they could use CiviMail to contact them, for example). For most employee users, the root contact would be set to the organization's contact record. Thus anyone w/ a relationship to the org would show up as a group to those users, which is probably what you want. This seems pretty powerful and elegant. Or... 3. Since we'll probably want the "Mailing List" and "Access Control" type options added to relationship types under this scheme, we could just piggyback on that. We could replace "Mailing List" with "Group" and have those relationship types show up everywhere a group would, and then just collapse all ACL handling to relationships (which would clean that up as well). This is all a bit brain-bending to absorb, so I have created a proof-of-concept implementation on the multiorg branch. You can try it out here: http://multiorg.demo.civicrm.org/standalone/. That is a very basic proof-of-concept implementation, and you will find bugs in it all over the place. But it does have groups implemented as relationships, and ACLs hook into that in order to determine who has access to what. I'm presenting this to the core development team on May 28, 2009 to see if we can sponsor its full implementation for 2.3. If you have questions, concerns, or feedback for this approach, please do let me know by commenting below or contacting me directly.

Comments

It would be excellent (should this idea fly) for group-contacts to be able to have custom data attached (in the same way that relationships can). I suspect this might be for-free side-effect of your proposal, but if it is not, I would like to take this opportunity to lobby for it.

(In my usage, our users are using groups to represent "teams" of contacts, not only because of the CiviMail support, but also because "group" and "team" are synonymous in our organisational culture. This is despite CiviCRM's current inability to add custom data to group-contacts. We add this custom data to relationships to measure their strategic fit.)

Nice proposal!
Ken

This is an interesting proposal, but are there implications for the integration with Organic Groups? The site I'm building requires OG/CiviCRm group integration and waiting for that functionality to stabilise has been a factor in delaying full build and deployment of the site.

Logically what is proposed makes a lot of sense, but I would be concerned if the OG relationships were lost in the process.

Shouldn't be too much of an issue as long as the CRM_Bridge_OG classes get updated, which shouldn't be terribly difficult.

In so far as the UI would stay they same, I imagine most of the parent functions would still exist by their old name anyway...?

"Jason and I realized that groups and relationships were actually redundant at the data model level. In fact, groups could be thought of as relationship types and vice versa."

I've been saying this since last year, discussed extensively with kurund at BADCamp/CiviCamp.

Further developed the idea of merging "categorization" objects here:
http://civicrm.org/node/512#comment-3391

And this has started to make its way into the 3.0 Brainstorm document:
http://wiki.civicrm.org/confluence/display/CRMDOC/CiviCRM+3.0+Brainstorm

So we are likely to need to implement a ACL hook so that the 'main contact' for a company can have controlled access to the other 'employees' of that company. This seems to be another situation where the boundary between Group/Relationship could be blurry. It would be easy to base this on fact that 'Main Contacts' have a particular relationship - all other employees have 'employee of' relationship - and we then create a group based on the latter and use this for the ACL

Just a +1 vote for the recommendation, and a +1 vote for custom data at Group level.