Fair warning, this post is intended to the technical part of our community, if you don't care about the architecture of civi, please skip this one, I'll come back to you soon with awesome datavisualisation and an interview of Micah about security and privacy (you'll like it).
And if you read anyway, I'm a bit of a drama queen and some of the mountains I describes are probably hills, at best.
Replacing the database abstraction layer was discussed recently with very promising results, and doctrine seems to be the obvious choice to get rid of DataObject.
To add a simple "whatever that is stored in a new table" in civi you need to (at least):
- describe the schema (xml)
- have a sql upgrade to create the table
- write a minimal bao
- write a minimal crud api
- write a php form
- write a page to list the whatever
- write a template for the form
- write a template for the list
Adding a new field in this whatever implies at least
- update the schema (xml)
- have a sql upgrade to alter the table
- alter the php form
- alter the template for the form
I'm sure that replacing quickform and pear classes are going to help, but will it be really worthwhile if, at the end, we still have to touch half a dozen files spread all around to implement simple changes?
Can we have in civi something like that that makes the trivial automatically done and the complex easier to customise and maintain too?
I've seen some examples of symfony where the db+form layers are one step, others with form+templates, but I haven't found my holly grail of "change a single file and get everything updated automatically".
Are you aware of such an example? ideally, integrated with backbone and that handle sql schema upgrade automatically too.
As for the way forward, we are probably not going to change everything at once so we won't benefit directly from that integration, but wouldn't it be good to know it will be possible before we continue this journey?
Xavier, for the committee of the better tomorrows ;)
I like the direction of your thinking, Xavier. It would be good to have a more streamlined development process, implement a DRY approach (http://c2.com/cgi/wiki?DontRepeatYourself), and maybe even put all the info about that 'table' in a single place.
Civix, in extending Symfony, has gone a fair ways towards auto-generating standard code for scaffolding, etc. It might be one good place to start, eg making it so that more of the auto-generation occurs easily.
This isn't quite the same as a different programming paradigm where 'the single place' contains all of the unique things about the 'table', and there is a common pattern for handling all of the standard things about tables. But maybe we should explore more how to do that.
I've previously pushed for a single definition of simple data validation code that could be run on form inputs, on data received via REST, and in-browser to improve the UX. Maybe it would be side-by-side places storing php code for the server and js/jQuery for the browser. Next step would be handling all validation.
What parts of the selection criteria or evaluation of Frameworks that we went through a year or two ago don't make sense now? If Django isn't a reasonable new framework for reasons like the maintainability of a project with even more independent technologies, are there easier steps to a programming paradigm like Django that wouldn't involve switching from PHP to python?
Validation is a good one that would greatly benefit of more automation indeed.
I suspect we'll need to extend a bit more the datamodel, but most should be there already.
yes, the django stuff is quite nice and i looked around to see what can be done in S2. Stumbled on this one and played with it a wee bit, seems quite nice for the simple cases (and in all honesty, i suspect 66% of the table in civi need a simple interface only)
Has anyone tried it? sounds interesting indeed.
Agree, most of the tables are "basic CRUD", if we can get them working without too much work (we probably need to extend a bit more our datamodel but we made great progress already in that direction) we could then focus on doing by hand the complex ones (contrib/membership...).
A few points which I liked/agreed with:
- Civi has (for a long time) required one to touch several layers. Civix provides a short/mid-term solution to the problem by providing a boilerplate, but ultimately boilerplate still takes a lot space. Modern frameworks have the same number of layers, but they reduce the problem by providing smarter defaults and automatic behaviors in each layer.
- The solution for us should focus more attention on the client-side framework (Backbone, Angular or similar) and the data-modelling and less attention on PHP/server-side logic. We can't eliminate PHP/server-side logic (because we need to provide security and integrity), but I'm starting to doubt whether PHP MVC provides as much value as JS MVC. I wonder if it's feasible to build most of the application on JS-MVC and a JSON API? Such an architecture would have the benefit of forcing us to design good APIs from the start.
I'm not sure we can completely get rid of the "html only" for the public facing pages (eg. contribute/register to event...) but having js enabled and a modern browser is more or less mandatory anyway already for the back-end so could work there.
However, I don't think one framework on the client side is a clear winner (as symfony in the php world is). Not sure if whatever framework we chose on the client side isn't going to go the dodo way. Not sure it matters is most of the JMVC is automatically done using the metadata of the schema either, but backbone doesn't completely feels right to me.
Or maybe we can have a middle ground like slickgrid+api that seems to be a versatile solution to build CRUD interfaces that are fast and pleasant to use?
For as long as I can remember, we've been looking for technological solutions to bridge the gap between two key objectives. One is building the ultimate UX where everything makes sense and all buttons that save stuff are labeled ‘save’, and the problem space semantics dictate how you navigate from one concept to the next. It’s deployable on anything from a mainframe to your coffee machine and never goes out of style. The second is to avoid having programmers write code they are not interested in and therefore only do half right : CRUD code copied from another scene but badly or partly adapted, and basic form/view processing to allow carbon-based life forms to maintain data.
PHP (in this case) frameworks lean towards the second objective, but frankly do a really food job at that. It does not matter which one you use (Zend, Symfony, …) as long as you do not roll your own (as if we never do that) and use only one. But a lot of the Civi code which can be considered crappy or obfuscated sits in the C part of MVC which is not the center of attention of these frameworks, but rather in the application glue code or the overall controllers. In Civi, that’s Page and Form stuff, and yes, we could do with a serious overhaul of the semantic/meta layer (DAO und API) but I am afraid the *design* of the Form/Page controllers is, as a separate layer, a whole different ball game.
UI frameworks tackle this from the other end. They are intended to be lightweight, very meta/service-oriented and intend to optimise use of the client platform in terms of flexibility and visual possibilities (grids, popups, …). Moving from HTML with a dose of jQuery to an MVC-like UI framework is not a gradual process : it’s a radical change, which many developers will find extremely hard to get their heads around in practice.
I think the lower layer of MVC (BAO/DAO/API etc.) is definitely a good thing to start on. As long as we do not have that, we'll never get around all the hooks not firing, custom fields not being returned, API signatures being slightly different etcetara. I do wonder how many parts of the ‘data logic’ currently sitting in the processing of Contributions etc. needs to be carved out of the page controller part and moved to the data business object itself, but this is triple bypass surgery … Perhaps a good approach is to build out the base layer again and *replace* module by module in a new architecture.
- Every feature and bug fix comes with an automated test or tests.
- Tests are easy to run and write.
- Flexible, simple customization and extension.
Young-Jin Kim and I attended a very well reasoned talk at SymfonyLive in Portland last spring with the premise that working in parallel, developing new features on the new framework and moving existing services to the new framework incrementally was far more successful than "ground up" rewrite of legacy projects. The Nov. 2012 version of the slides are here:
SensioLabs has put up a screencast of the talk on YouTube:
If others find this presentation compeling, then let's make a plan!