Slow pages are annoying and they can cost $$$. But improving performance often means re-writing chunks of code and is often expensive. Bad performance hurts us all. So the question is how can we improve it.
The main mechanism we have used in the community so far for banding together to fund these things is the MIH - the tricky thing for performance is framing the MIH in a useful way.
There are some specific areas of known slowness in CiviCRM which it would be great to address and we could raise specific MIH for. Some that I am aware of are:
- Too many search fields (e.g contribution dashboard) - the query that runs when gathering data for screens like the contribution dashboard gathers a bunch of fields that aren't required for that purpose. We have found that on a medium sized database (80,000 contributions) a temp tables of over a GB of RAM is created (we know this because the temp directory was 1GB). By removing fields like 'product' from those returned we got past the problem. The estimate I have on this is about 15 hours which seems like a great win
- Activity searches - searches involving activities are slower that they need to be because of the table structure. Some time ago we agreed a re-structure plan for these - about 50 hours
- ACLs - Users whose permissions are limited in some way by an ACL hit a piece of code that is pretty sluggish. The current acl cache can be slow to rebuild - and apparently in some (extreme) cases people can actually access things they shouldn't in the meantime! Generally this is a performance hit for anyone who uses ACLs. The approach for tackling this seems to be known and has been estimated at about 20 hours.
- Searches involving joins on option values. In some specific queries we were seeing more then ten-fold improvements by eliminating option value joins. The site I was seeing this on was 3.4 quick search (with prefix id being the offending field)
- Cases where the database is being queried rather than using constants.
The last two of these are a series of small fixes rather than a big one - which leads to my thinking that performance is not a project - it's an ongoing maintenance task. At Fuzion we do regularly submit fixes and improvements to the core code - but we find that it's extremely difficult to make performance improvements within our normal client business. Performance really does matter and we do have to put up with complaints at times. It seems to me that the best way for us and other Consultants/ significant users of CiviCRM to get ongoing improvements in performance and code quality is to commit to a monthly contribution. In our case we are thinking $50 per month each (NZD sorry). If we can get another 20 people / organisations to put in, say, $20 per month we have $6,000 a year - not really enough - but enough to start chipping away at some of the problems.
So, I am launching 2 pages - an MIH for the ACL groups & one for an ongoing performance and code maintenance fund.
Comments
CiviCase related queries are also a significant performance issue. The case queries join with activities and relationships and become quite nightmarish.
I think the plan to have an ongoing maintenance fund is great. But many of the performance weaknesses are found in very specific areas/ways and with live systems stressing a particular area of the system -- so it seems like we should also be working to identify those pain points which seem to be in need of evaluation.
Yes, we should find a place to start to come up with a bit of a list. I had an idea that with the performance fund we could try PCPs - where people could effectively 'vote' for which bugs get high on the list..
I did identify the activity queries as one that we need to tackle. But others on your radar - ideally we should either on the WIKI or JIRA or? flesh them out.
CivicActions has been using CiviCRM with some of our non-profit clients for years and we have always been happy with the level of support available from the community. In particular, Eileen has done wonders with the API, test framework and generally supporting the project to become more solid from release to release, which is something every developer and user of this package benefits from. Thank you!
We are pleased to support the Ongoing Performance improvements and code maintenance MIH that Eileen has suggested and will continue to do what we can to support the project in future releases. (I'd also like to see smart groups optimisation included in the mix!)
This probably goes without saying, but here goes anyway: it seems to me that most of what you've described above are the result of pushing out new features too quickly. I can certainly understand there's always a tradeoff, but my experience of CiviCRM is that new features get pushed forward and out the door too quickly, resulting in extra code that's not really useful at any scale - the ACL is probably a good example of a scary big complex chunk of code that isn't reliable or scalable enough to use in any useful production environment [imho].
I know this is also not an original observation and that it's an ongoing discussion, but I do think that any initiative for an ongoing maintenance and peformance initiative needs to be integrated into the whole code development process - i.e. before any new features come out in the core code, it needs to be examined from a scalability and maintenance perspective before being included in a release. Now that the new extensions architecture seems to be stabiliizing, maybe that's a good way of allowing early adopters (and funders of new initiatives!) to get their code working and available, but still keep it separate from the core code until it's been more thorougly production tested - kind of like Drupal's process of how many modules start of as contributed modules and only move to core later.
I do agree with extensions that folks can package and ship things seperate from the release cycle, which will slow down the number of new features / release. Not a bad thing
ACL's have been around since the 1.x series, and you can get it to scale quite nicely if you implement your own acl's via hooks.
However while having constraints like "the code should scale" is good in theory, a few issue arise:
lobo
I agree with all the points mentioned above, especially #5. Querying a database for no reason is just inefficient coding. But I am not sure your "fund raising" idea would work because it just seems that the amount raised would not be sufficient.
In fact the money from this MIH was used to make the pseudoconstant work (that did #5) viable