Pubblicato
2009-06-22 17:56
Here at raSANTIAGO we are entering our third year with CiviCRM and still find ourselves struggling to make desired changes to the codebase. Too often we have expressed desired to re-architect and re-factor the CiviCRM. Recently we have completed two major projects that had us deeper in the codebase then before and realizing that we had to stop complaining. So we set ourselves to figuring out a good target architecture and a migration path that would not interfere with current development but give us the opportunity to engineer in patterns which have produced stable, extensible and maintainable systems for us. In the process, we wanted to deal with serious issues like advancing the API, advance our abilities to test and radically increase code reuse.
After a lot of work and prototyping we think we may have found an architecture that will work and a path to that architecture. From this work we offer a proposal for a new CiviCRM architecture. The proposal covers our analysis of the current architecture, some criticisms and details on a proposed architecture. At the end it lists out some activities for getting to the new architecture, many of which we have already begun and will blog about.
In short, we propose using PHP Doctrine for the ORM layer in place of DAO/BAO. Fortunately, both of these layers can exist in parallel using the same database without stepping on one another. PHP Doctrine provides a good implementation of the ActiveRecord pattern. Instead of QuickForms and Smarty we propose using thin controllers which speak XML and JSON. All UI is subsequently pushed directly to HTML, JS and jQuery. The controllers and the UI are connected through authenticating RESTful interface.
We will be posting and blogging about some of our prototypes and working code to show the pieces of the architecture in action. The most exciting news right now is that we have a 90%+ working RESTful API as a natural byproduct of our efforts so far.
This is a work in progress and appreciate any feedback, criticism and contributions. We hope this effort will serve the community well.
Peace
Roberto
After a lot of work and prototyping we think we may have found an architecture that will work and a path to that architecture. From this work we offer a proposal for a new CiviCRM architecture. The proposal covers our analysis of the current architecture, some criticisms and details on a proposed architecture. At the end it lists out some activities for getting to the new architecture, many of which we have already begun and will blog about.
In short, we propose using PHP Doctrine for the ORM layer in place of DAO/BAO. Fortunately, both of these layers can exist in parallel using the same database without stepping on one another. PHP Doctrine provides a good implementation of the ActiveRecord pattern. Instead of QuickForms and Smarty we propose using thin controllers which speak XML and JSON. All UI is subsequently pushed directly to HTML, JS and jQuery. The controllers and the UI are connected through authenticating RESTful interface.
We will be posting and blogging about some of our prototypes and working code to show the pieces of the architecture in action. The most exciting news right now is that we have a 90%+ working RESTful API as a natural byproduct of our efforts so far.
This is a work in progress and appreciate any feedback, criticism and contributions. We hope this effort will serve the community well.
Peace
Roberto
Filed under
Comments
while the document does give us a fair amount of detail, seeing a working example would be even better. So looking forward to seeing the release of the code under an OSI license in the near future
lobo
No problem. The code will come next. I always prefer to let analysis and documentation lead code.
Is your code available in public subversion?
It sounds like you're almost going to require JS support on the part of the client to run CiviCRM, seems like a lot of effort. Personally I'd suggest porting it to something like Joomla! which would make stand alone versions easy (you ship a stripped down Joomla! with the CiviCRM in it) and you can relatively easily then embed it into Drupal (unlike the other way which requires you to hack apart a tonne of Drupal's files just to get something to work because you put it in a different directory). But thats just me and I'm slightly biased towards Joomla!.
Not sure embedding Joomla in CiviCRM in Drupal makes a lot of sense. The new architecture is meant to reduce the number of places where you can place business logic and application logic. Javascript seems like a minimal and very reasonable requirement.
haha. :-)
Drupal Core is lean enough, no stripping needed.
DAO/BAO -> nodeapi / FieldAPI (drupal7) (+ DB:TNG)
QuickForm -> FormAPI
Smarty -> PHPTemplate
It's worked for Ubercart.
(I'm curious how Joomla would map like this. I don't really think of Joomla as a Framework at all like Drupal is, but maybe I can be persuaded?)
Anyway, I like a lot of what's been proposed, although I'm not especially impressed by my first glance at PHP Doctrine.
Even apart from this idea of using a CMS Framework as the framework, the proposal does seem to miss a lot in regards to CMS integration. It seems you would need a View that produced HTML to, for example, put profile data collection into CMS registration forms.
I'd like it if more data integrity and business logic could be pushed down into the dbms. As MySQL 5.1 is now out in stable GA release, we could be looking at expanding the use of stored procedures, functions and triggers. This independent of whether the main PHP code is using the existing libraries or this proposal's.
My concern is that I'm looking at a couple of very large CiviCRM implementations (500k - 25M contacts) which will require regular synchronizations with external data sources that do not have shared contact identifiers or consistent name or addressing for the same person. As a result bulk operations will need to be performed at a scale that is not really practicable with a record by record PHP approach.
So there are a number of huge architectural disasters which come from business logic in the DBMS. So I would avoid this like the plague. Been down that road a few times and while in the short term you gain huge performance you lose all long term maintainability and reuse. Bleh!
But, this is not to say that you cannot take advantage of the DBMS for optimizations of simpler CRUD operations especially in situations of Bulk updates involving related objects. That is a good thing but for the type of problem you are facing I think you are choosing the wrong tool.
Data Synchronization can be handled in number of ways. I would not recommend using the API to do very large bulk operations record by record. We have used Talend and other open source toold to do synchronization on large data sets distributed across numerous heterogeneous databases. This type of issue is not relevant to CiviCRM, though. So it should not be a consideration in the architecture.
Roberto,
Have there been any updates regarding your proposed changes to the CiviCRM architecture? Did you prototype this for standalone or is it riding on Drupal/Joomla? Do you have a blog/wiki for developers w/instructions on where we can download the prototype code, dev environment requirements, etc.?
Take a look at http://civicrm.org/node/597. The blog entry posts to the repository at GitHub. Sorry for the delay on it.
We are using it for Standalone, Drupal and Joomla.
+1 for this. I think you've got some great ideas here that are well thought out. I think the current architecture has been great at helping the application move as quick as it has, but is now holding back a few key areas.
After working with Drupal's elegant FormsAPI, QuickForms is anything but quick and infuriatingly difficult to use due to zero documentation. Furthermore it doesn't play well with Smarty in some very important areas. I'll be glad to see it go.
But that begs the question, what will create forms? And how will we programmatically alter them (i.e. hook_form_alter)?
And will there be some sort of equivalent to Smarty where we could override the HTML of a page?
I have similar feelings about Pear DB. As you point out it is a bit limited. My cursory review of Doctrine is that it is very well documented and has an active development community. Combined with your proposed model there's the potential to eliminate a load of duplication while making things generally much more consistent. +1
My one concern about the JavaScript dependency is that the application not loose any accessibility (or if possible that we become more accessible). And by that I mean things like screen readers, mobile devices etc. Dojo had some usability issues and moving to jQuery has helped, but I think if we don't keep bringing up the topic, we tend to forget about it.
Another thing that we need to keep in mind is self-documenting code. Currently you can open any file and you'll see a whole lot of
foreach ( $dpml as $p => $n ) {
Which means that it takes an order of magnitude more time to figure out how something works. Your proposal incorporates some really elegant tools that could markedly increase the maintainability of the code. However this can easily be made naught if the code is not understandable on a micro level.
Lastly I think you've made a good move with the idea of incremental migration. Release-early-release-often has served CiviCRM well and I wouldn't want to see a 2yr re-architecting plan kill off the community.
Hi,
Waiting on the code to be sure, but for what I understood, you plan to throw out any regular html form and replace the usual post by ajax to a rest api ?
That's a bit of a usability concern here. It should work by default (or as fallback) with the good ol' PUT ;)
Couple of comments:
I'd suggest JSON over xml at anytime, especially to talk to the browser
and I'd use a templating jquery solution (eg pure)
X+
I generally prefer JSON as well. PHP's native JSON functions work much better than PHP's native XML functions. I would think that things would be more performant on the JavaScript side too.
I'm not too concerned about requiring a fallback for no JavaScript. The iPhone was the first mobile device to be somewhat bearable to surf the net and it has JS support. Future phones will follow suit. Screen readers do have limited support for JS, they generally do better with the "renderthe JS when the page is loaded, clicking a link sends you to a new URL" approach rather than the "load a huge JS library upfront, clicks make AJAX calls and you always remain at the same URL". As I understand it this new architecture uses the more compatible method.
I am not sure I agree that REST + JSON/XML is really extreme. Its a great dev pattern in use on a lot of big sites and one that has served us well. I am not sure what the complaint is here. Can you describe a scenario where this would pose a problem? Perhaps that would help clarify your concern.
The suggestion for JSON is a good one and we are planning to support it since Doctrine already natively supports XML and JSON (and a couple more serialization patterns)
a few BIG sites that use a pure REST + JSON/XML model. Also links to other articles/websites that implement it would be appreciated
lobo