Toward CiviConnect

CiviCRM sits in the middle -- exchanging data with your CMS, payment processor, email service, SMS service, spreadsheets, ad nauseum.  CiviCRM is also extremely flexible -- supporting multiple CMSs, multiple payment processors, multiple email providers, multiple SMS providers, ad nauseum.  These are great power features, but they also come with a cost -- complexity.  The on-boarding process for a new organization requires evaluating and configuring a plethora of integration options, and the configuration is not always easy. I'd like to talk a bit about this problem and a nascent project to solve it.

Integrations today

The cron service is a good example of an integration. Every site running Civi needs a set of cron jobs to keep the system moving -- to deliver mail blasts, to update membership statuses, to send scheduled reminders. Cron is actually a fairly simple service, but there are a lot of small variations (which are almost trivial... but which prevent universal automation).

Unfortunately, a new administrator often doesn't understand the cron situation at the start.  A typical story for a new administrator (call him Tim) might go like this:

1. Blissful ignorance - Tim successfully installs Civi. As soon as he sees that dashboard, he gets excited and starts playing with the app.  But he doesn't know anything about cron.  He's only skimmed the Civi documentation, and he never really *needed* cron with his other web-applications (Drupal, Joomla).  So he's working on an incomplete system.
2. Discovery - Tim writes an email blast and schedules it for 2:00pm. 2:00pm comes...  and goes...  with no emails! He spends 20 minutes searching Google, the forum, or Stack Exchange and discovers the missing element - cron!  He eventually finds the wiki page with instructions.
3. Scavenger hunt - The wiki page is not the end -- it is just the beginning of the technical scavenger hunt. Tim needs to find the right credentials from the right places and put them together.  This might include:
   • Create a new username and password in the CMS. (Use a secure, random password. How do I make secure, random password?)
   • Create a new role in the CMS. Grant special permissions to the role. (Wait, which permissions?)
   • Lookup the site key in civicrm.settings.php. (Wait, where is civicrm.settings.php? What were my SSH credentials?)
   • Add a new entry to cron tab using a funny URL (including the username, password, and site key). (Wait, where is the crontab? Did I make any typos in the URL?)
   • Navigate to the "Scheduled Jobs" screen and enable the most common jobs. (Wait, which ones are normally needed?)
   • Cross fingers, watch, and wait. Eventually, the original email blast will go out.

Hooray! Tim now has a working cron integration.  That whole process was a bit of surprise, but sometimes it's fun to do a scavenger hunt -- you learn more along the way!  Happy ending.

Or is it? Tim hasn't configured the email integration, so the mailings still won't go out...  until he goes through another discovery and another scavenger hunt.  More generally, if CiviCRM sits in the middle, then it will need several integrations -- and many of them will require a technical scavenger hunt.

CiviConnect

The CiviConnect initiative aims to provide an alternative to the scavenger hunt while preserving Civi's core values (Free/Libre Software, data ownership, on-site deployment, etc).  How?  By providing a simple, secure mechanism for finding and connecting supplemental services.

A supplemental service is one that runs outside Civi but which enhances Civi. Civi users today work with many supplemental services (such as cron, email, or SMS), and they're necessary for basic technical and economic reasons: supplemental services often involve precise software requirements, special expertise, external regulations, and (yes) a scavanger hunt. In an ideal world, all Civi users have the freedom to run these services themselves -- but if we insist that every user actually do so, then we gratuitously drive up the cost and complexity.

CiviConnect aims to provide the technical glue so that our community can deploy and connect supplemental services in a simpler, more cost-effective fashion.

Technical Trials

On the technical front, CiviConnect must support administrative approvals, fine-grained permissions, general vetting, and revocations so that site-administrators can have confidence in the security of supplemental services.  CiviConnect should securely work on the complex range of shared, dedicated, and clustered hosting environments (which may be well-configured or poorly-configured).

The good news is that we have a working prototype (CRM-16173).  In the coming months, the next steps are to develop a couple services which would be useful for a new administrator -- and then to begin real-world technical trials. If you've joined Civi's membership program, you'll be the first to get access.

It's important that the technical trials begin with real but small services.  Real services will allow developers to discover unexpected issues.  Small services pose lower risk to users (if something goes wrong) and don't require as much business planning. Before we can pursue broader menu of services, our community must discuss the business issues that they'll raise.

Business Issues

CiviCRM is a codebase, and it is a community -- with many shared values and a few distinct constituencies.  In the long-term, I believe that everyone in our community -- users, administrators, core developers, consultants, service providers, and full-solution firms -- will benefit from replacing the Scavanger Hunt Problem with a simpler connection system.  It will enable all of us to spend less time on configuration and more time on specialized, high-value projects.  It will improve satisfaction of new users and boost competitiveness, efficiency, and user-experience.

However, in the short term, CiviConnect represents new terrain. New terrain can open new business models for providing services, but it also requires new ground-rules, and we need to figure out what those rules will be.  Which supplemental services should be available through CiviConnect?  How do we decide?  Who runs them?  Do all services need to be open-source?  Should services be offered as a wild-west appstore, where any fly-by-night company can drive down prices (and quality)?  Would we prefer a monoculture, where corporate overlords dictate the options?  How do we find a middle ground that represents the goals and values of our community?

I don't claim to have an easy answer. However, based on conversations at April's events in Colorado, I believe these are worthy goals:

• Guarantee user freedom. We do not want services to become an "end-run" around the freedoms of open-source and data-ownership.
• Maintain our non-profit character. We do not want to commercialize our community. We serve causes - not shareholders.
• Provide a simple, clear on-boarding process for new organizations -- without the Scavanger Hunt Problem.
• Leverage our strengths. Some members of the community are great at building software; some are great at scaling server clusters; others are great at crafting newsletters.
• Encourage solid business models -- not speculation. Users need services that will be around for the long-haul. Service providers need to earn a living without fear that someone will commandeer their product.
• Deliver good value to users. Services should be high-quality with simple, competitive prices.

A tall order? Yes. Hopefully, we can talk more about this in the coming months. What services do you think should be offered through this medium? On what terms? Please feel free to comment on the blog or send email (totten at civicrm.org or josh at civicrm.org).