Upcoming Events
San Francisco CiviCRM Meetup - February 8th, 2012
February 8th, 2012
Come meet others from the Bay Area who are interested in, using or developing (more...)
UK usergroup - London meetup
February 8th, 2012
Come and meet others from the UK that are using CiviCRM or are interested in (more...)
London user and administrator training
February 23rd, 2012
A comprehensive two day hands on training course covering the configuration, (more...)
CiviCRM London sprint Feb 2012
February 27th, 2012
Following the CiviCRM training here in London, we will have a CiviCRM code (more...)
Philadelphia - CiviCRM Meetup for Q1 2012
March 13th, 2012
UK South West - CiviCRM Meetup
March 20th, 2012
Come meet others from the Area who are interested in, using or developing for (more...)
[Bristol, UK] user and administrator training
March 21st, 2012
A comprehensive hands on training course covering the configuration, (more...)
San Francisco user and administrator training
March 29th, 2012
A comprehensive two day hands on training course covering the configuration, (more...)
CiviCRM Usability, Test and Code Sprint - San Francisco (March 2012)
March 29th, 2012
This usability, code and test sprint is targeted at CiviCRM users and (more...)
CiviCon 2012 San Francisco Bay Area - April 2nd 2012
April 2nd, 2012
CiviCon is THE annual event bringing together the people who use, develop, (more...)
CiviCRM Documentation, Test and Code Sprint - after CiviCon San Francisco (April 2012)
April 4th, 2012
This sprint is targeted at CiviCRM users and developers who want to work on (more...)
CiviCRM Components
Tools for engaging your supporters...
Proposing a New Architecture for CiviCRM
- Not Just a Contact Database
-
These optional components give you more power to connect and engage your supporters.
civiCASE
Case management for clients and constituents.
civiCONTRIBUTE
Online fundraising and donor management.
civiEVENT
Online event registration and participant tracking.
civiMEMBER
Online signup and membership management.
civiMAIL
Personalized email blasts and newsletters.
civiREPORT
Report generation and template management.
Here at raSANTIAGO we are entering our third year with CiviCRM and still find ourselves struggling to make desired changes to the codebase. Too often we have expressed desired to re-architect and re-factor the CiviCRM. Recently we have completed two major projects that had us deeper in the codebase then before and realizing that we had to stop complaining. So we set ourselves to figuring out a good target architecture and a migration path that would not interfere with current development but give us the opportunity to engineer in patterns which have produced stable, extensible and maintainable systems for us. In the process, we wanted to deal with serious issues like advancing the API, advance our abilities to test and radically increase code reuse.
After a lot of work and prototyping we think we may have found an architecture that will work and a path to that architecture. From this work we offer a proposal for a new CiviCRM architecture. The proposal covers our analysis of the current architecture, some criticisms and details on a proposed architecture. At the end it lists out some activities for getting to the new architecture, many of which we have already begun and will blog about.
In short, we propose using PHP Doctrine for the ORM layer in place of DAO/BAO. Fortunately, both of these layers can exist in parallel using the same database without stepping on one another. PHP Doctrine provides a good implementation of the ActiveRecord pattern. Instead of QuickForms and Smarty we propose using thin controllers which speak XML and JSON. All UI is subsequently pushed directly to HTML, JS and jQuery. The controllers and the UI are connected through authenticating RESTful interface.
We will be posting and blogging about some of our prototypes and working code to show the pieces of the architecture in action. The most exciting news right now is that we have a 90%+ working RESTful API as a natural byproduct of our efforts so far.
This is a work in progress and appreciate any feedback, criticism and contributions. We hope this effort will serve the community well.
Peace
Roberto
Comments
Yes
+1 for this. I think you've got some great ideas here that are well thought out. I think the current architecture has been great at helping the application move as quick as it has, but is now holding back a few key areas.
After working with Drupal's elegant FormsAPI, QuickForms is anything but quick and infuriatingly difficult to use due to zero documentation. Furthermore it doesn't play well with Smarty in some very important areas. I'll be glad to see it go.
But that begs the question, what will create forms? And how will we programmatically alter them (i.e. hook_form_alter)?
And will there be some sort of equivalent to Smarty where we could override the HTML of a page?
I have similar feelings about Pear DB. As you point out it is a bit limited. My cursory review of Doctrine is that it is very well documented and has an active development community. Combined with your proposed model there's the potential to eliminate a load of duplication while making things generally much more consistent. +1
My one concern about the JavaScript dependency is that the application not loose any accessibility (or if possible that we become more accessible). And by that I mean things like screen readers, mobile devices etc. Dojo had some usability issues and moving to jQuery has helped, but I think if we don't keep bringing up the topic, we tend to forget about it.
Another thing that we need to keep in mind is self-documenting code. Currently you can open any file and you'll see a whole lot of
foreach ( $dpml as $p => $n ) {
Which means that it takes an order of magnitude more time to figure out how something works. Your proposal incorporates some really elegant tools that could markedly increase the maintainability of the code. However this can easily be made naught if the code is not understandable on a micro level.
Lastly I think you've made a good move with the idea of incremental migration. Release-early-release-often has served CiviCRM well and I wouldn't want to see a 2yr re-architecting plan kill off the community.
agree rest + js is a bit too extreme
Hi,
Waiting on the code to be sure, but for what I understood, you plan to throw out any regular html form and replace the usual post by ajax to a rest api ?
That's a bit of a usability concern here. It should work by default (or as fallback) with the good ol' PUT ;)
Couple of comments:
I'd suggest JSON over xml at anytime, especially to talk to the browser
and I'd use a templating jquery solution (eg pure)
X+
But all the cool kids are doing it....
I am not sure I agree that REST + JSON/XML is really extreme. Its a great dev pattern in use on a lot of big sites and one that has served us well. I am not sure what the complaint is here. Can you describe a scenario where this would pose a problem? Perhaps that would help clarify your concern.
The suggestion for JSON is a good one and we are planning to support it since Doctrine already natively supports XML and JSON (and a couple more serialization patterns)
hmm, can u give a few examples of ..
a few BIG sites that use a pure REST + JSON/XML model. Also links to other articles/websites that implement it would be appreciated
lobo
I generally prefer JSON as
I generally prefer JSON as well. PHP's native JSON functions work much better than PHP's native XML functions. I would think that things would be more performant on the JavaScript side too.
I'm not too concerned about requiring a fallback for no JavaScript. The iPhone was the first mobile device to be somewhat bearable to surf the net and it has JS support. Future phones will follow suit. Screen readers do have limited support for JS, they generally do better with the "renderthe JS when the page is loaded, clicking a link sends you to a new URL" approach rather than the "load a huge JS library upfront, clicks make AJAX calls and you always remain at the same URL". As I understand it this new architecture uses the more compatible method.
updates?
Roberto,
Have there been any updates regarding your proposed changes to the CiviCRM architecture? Did you prototype this for standalone or is it riding on Drupal/Joomla? Do you have a blog/wiki for developers w/instructions on where we can download the prototype code, dev environment requirements, etc.?
Posted first example and code.
Take a look at http://civicrm.org/node/597. The blog entry posts to the repository at GitHub. Sorry for the delay on it.
We are using it for Standalone, Drupal and Joomla.
supporting bulk operations on large datasets
I'd like it if more data integrity and business logic could be pushed down into the dbms. As MySQL 5.1 is now out in stable GA release, we could be looking at expanding the use of stored procedures, functions and triggers. This independent of whether the main PHP code is using the existing libraries or this proposal's.
My concern is that I'm looking at a couple of very large CiviCRM implementations (500k - 25M contacts) which will require regular synchronizations with external data sources that do not have shared contact identifiers or consistent name or addressing for the same person. As a result bulk operations will need to be performed at a scale that is not really practicable with a record by record PHP approach.
The Right Tool for the Right Job
So there are a number of huge architectural disasters which come from business logic in the DBMS. So I would avoid this like the plague. Been down that road a few times and while in the short term you gain huge performance you lose all long term maintainability and reuse. Bleh!
But, this is not to say that you cannot take advantage of the DBMS for optimizations of simpler CRUD operations especially in situations of Bulk updates involving related objects. That is a good thing but for the type of problem you are facing I think you are choosing the wrong tool.
Data Synchronization can be handled in number of ways. I would not recommend using the API to do very large bulk operations record by record. We have used Talend and other open source toold to do synchronization on large data sets distributed across numerous heterogeneous databases. This type of issue is not relevant to CiviCRM, though. So it should not be a consideration in the architecture.
Sounds heavy
It sounds like you're almost going to require JS support on the part of the client to run CiviCRM, seems like a lot of effort. Personally I'd suggest porting it to something like Joomla! which would make stand alone versions easy (you ship a stripped down Joomla! with the CiviCRM in it) and you can relatively easily then embed it into Drupal (unlike the other way which requires you to hack apart a tonne of Drupal's files just to get something to work because you put it in a different directory). But thats just me and I'm slightly biased towards Joomla!.
I've proposed the exact opposite
haha. :-)
Drupal Core is lean enough, no stripping needed.
DAO/BAO -> nodeapi / FieldAPI (drupal7) (+ DB:TNG)
QuickForm -> FormAPI
Smarty -> PHPTemplate
It's worked for Ubercart.
(I'm curious how Joomla would map like this. I don't really think of Joomla as a Framework at all like Drupal is, but maybe I can be persuaded?)
Anyway, I like a lot of what's been proposed, although I'm not especially impressed by my first glance at PHP Doctrine.
Even apart from this idea of using a CMS Framework as the framework, the proposal does seem to miss a lot in regards to CMS integration. It seems you would need a View that produced HTML to, for example, put profile data collection into CMS registration forms.
Hmmmmm.... Sounds like Turducken
Not sure embedding Joomla in CiviCRM in Drupal makes a lot of sense. The new architecture is meant to reduce the number of places where you can place business logic and application logic. Javascript seems like a minimal and very reasonable requirement.
can we see the code too please ...
while the document does give us a fair amount of detail, seeing a working example would be even better. So looking forward to seeing the release of the code under an OSI license in the near future
lobo
Code Coming Next......under AGPL
No problem. The code will come next. I always prefer to let analysis and documentation lead code.
release date ?
Is your code available in public subversion?