Published
2013-10-29 06:50
Hi,
Fair warning, this post is intended to the technical part of our community, if you don't care about the architecture of civi, please skip this one, I'll come back to you soon with awesome datavisualisation and an interview of Micah about security and privacy (you'll like it).
And if you read anyway, I'm a bit of a drama queen and some of the mountains I describes are probably hills, at best.
Various components of civicrm were chosen at the start of the project, when the top of the art in php component was the PEAR repository and dinosaurs were still roaming the earth. Most of them didn't age that well and there is a broad consensus we need to replace them at one point or another. We've already replaced our javascript framework and moved from dojo to jQuery and some ajax heavy part of the interface are benefiting from backbone+marionette for instance.
Replacing the database abstraction layer was discussed recently with very promising results, and doctrine seems to be the obvious choice to get rid of DataObject.
However, despite all the issues we have dealing with the existing libraries, I'm not sure replacing them with more modern ones will really solve my (and I suspect most) problems: boilerplate and duplication.
To add a simple "whatever that is stored in a new table" in civi you need to (at least):
- describe the schema (xml)
- have a sql upgrade to create the table
- write a minimal bao
- write a minimal crud api
- write a php form
- write a page to list the whatever
- write a template for the form
- write a template for the list
Some steps can be skipped (eg. by not providing an api) or combined (eg. the sample template for the list and the form) but it's not best practice nor does it save much time.
Adding a new field in this whatever implies at least
Adding a new field in this whatever implies at least
- update the schema (xml)
- have a sql upgrade to alter the table
- alter the php form
- alter the template for the form
Quite a few steps, isn't it? and plenty of options to miss one and create issues for some users, or (ab)use one size fits all OptionValue/Group even if it means no more DB enforced coherency. Or skip unit tests. Or inject business logic in the template....
I'm sure that replacing quickform and pear classes are going to help, but will it be really worthwhile if, at the end, we still have to touch half a dozen files spread all around to implement simple changes?
I'm sure that replacing quickform and pear classes are going to help, but will it be really worthwhile if, at the end, we still have to touch half a dozen files spread all around to implement simple changes?
I know of a framework that makes it easier: with django, you can write a single file and it manages to provide the rest and have a decent looking interface that stores and retrieves from the db. Moreover, does a pretty good job letting you override any of these parts if you need something more advanced than what is provided out of the box.
Can we have in civi something like that that makes the trivial automatically done and the complex easier to customise and maintain too?
I've seen some examples of symfony where the db+form layers are one step, others with form+templates, but I haven't found my holly grail of "change a single file and get everything updated automatically".
Are you aware of such an example? ideally, integrated with backbone and that handle sql schema upgrade automatically too.
As for the way forward, we are probably not going to change everything at once so we won't benefit directly from that integration, but wouldn't it be good to know it will be possible before we continue this journey?
Xavier, for the committee of the better tomorrows ;)
Filed under
Comments
I like the direction of your thinking, Xavier. It would be good to have a more streamlined development process, implement a DRY approach (http://c2.com/cgi/wiki?DontRepeatYourself), and maybe even put all the info about that 'table' in a single place.
Civix, in extending Symfony, has gone a fair ways towards auto-generating standard code for scaffolding, etc. It might be one good place to start, eg making it so that more of the auto-generation occurs easily.
This isn't quite the same as a different programming paradigm where 'the single place' contains all of the unique things about the 'table', and there is a common pattern for handling all of the standard things about tables. But maybe we should explore more how to do that.
I've previously pushed for a single definition of simple data validation code that could be run on form inputs, on data received via REST, and in-browser to improve the UX. Maybe it would be side-by-side places storing php code for the server and js/jQuery for the browser. Next step would be handling all validation.
What parts of the selection criteria or evaluation of Frameworks that we went through a year or two ago don't make sense now? If Django isn't a reasonable new framework for reasons like the maintainability of a project with even more independent technologies, are there easier steps to a programming paradigm like Django that wouldn't involve switching from PHP to python?
Validation is a good one that would greatly benefit of more automation indeed.
I suspect we'll need to extend a bit more the datamodel, but most should be there already.
yes, the django stuff is quite nice and i looked around to see what can be done in S2. Stumbled on this one and played with it a wee bit, seems quite nice for the simple cases (and in all honesty, i suspect 66% of the table in civi need a simple interface only)
http://sonata-project.org/about
lobo
Hi,
Has anyone tried it? sounds interesting indeed.
Agree, most of the tables are "basic CRUD", if we can get them working without too much work (we probably need to extend a bit more our datamodel but we made great progress already in that direction) we could then focus on doing by hand the complex ones (contrib/membership...).
X+
A few points which I liked/agreed with:
I'm not sure we can completely get rid of the "html only" for the public facing pages (eg. contribute/register to event...) but having js enabled and a modern browser is more or less mandatory anyway already for the back-end so could work there.
However, I don't think one framework on the client side is a clear winner (as symfony in the php world is). Not sure if whatever framework we chose on the client side isn't going to go the dodo way. Not sure it matters is most of the JMVC is automatically done using the metadata of the schema either, but backbone doesn't completely feels right to me.
Or maybe we can have a middle ground like slickgrid+api that seems to be a versatile solution to build CRUD interfaces that are fast and pleasant to use?
For as long as I can remember, we've been looking for technological solutions to bridge the gap between two key objectives. One is building the ultimate UX where everything makes sense and all buttons that save stuff are labeled ‘save’, and the problem space semantics dictate how you navigate from one concept to the next. It’s deployable on anything from a mainframe to your coffee machine and never goes out of style. The second is to avoid having programmers write code they are not interested in and therefore only do half right : CRUD code copied from another scene but badly or partly adapted, and basic form/view processing to allow carbon-based life forms to maintain data.
PHP (in this case) frameworks lean towards the second objective, but frankly do a really food job at that. It does not matter which one you use (Zend, Symfony, …) as long as you do not roll your own (as if we never do that) and use only one. But a lot of the Civi code which can be considered crappy or obfuscated sits in the C part of MVC which is not the center of attention of these frameworks, but rather in the application glue code or the overall controllers. In Civi, that’s Page and Form stuff, and yes, we could do with a serious overhaul of the semantic/meta layer (DAO und API) but I am afraid the *design* of the Form/Page controllers is, as a separate layer, a whole different ball game.
UI frameworks tackle this from the other end. They are intended to be lightweight, very meta/service-oriented and intend to optimise use of the client platform in terms of flexibility and visual possibilities (grids, popups, …). Moving from HTML with a dose of jQuery to an MVC-like UI framework is not a gradual process : it’s a radical change, which many developers will find extremely hard to get their heads around in practice.
I think the lower layer of MVC (BAO/DAO/API etc.) is definitely a good thing to start on. As long as we do not have that, we'll never get around all the hooks not firing, custom fields not being returned, API signatures being slightly different etcetara. I do wonder how many parts of the ‘data logic’ currently sitting in the processing of Contributions etc. needs to be carved out of the page controller part and moved to the data business object itself, but this is triple bypass surgery … Perhaps a good approach is to build out the base layer again and *replace* module by module in a new architecture.
We've discussed this issue a bunch at Giant Rabbit. We definitely think the best approach would be to pick a new framework that pushes most of the code into client side Javascript with the server side code mostly limited to persistence, security, and scheduled tasks. We think a new framework would be a good choice because some of thing things that we would like to build into the current framework would be pretty hard to do. Our main goals for a new framework would be:
Young-Jin Kim and I attended a very well reasoned talk at SymfonyLive in Portland last spring with the premise that working in parallel, developing new features on the new framework and moving existing services to the new framework incrementally was far more successful than "ground up" rewrite of legacy projects. The Nov. 2012 version of the slides are here:
http://www.slideshare.net/fabrice.bernhard/modernisation-of-legacy-php-applications
SensioLabs has put up a screencast of the talk on YouTube:
http://youtu.be/kZocRPreQ5g
If others find this presentation compeling, then let's make a plan!