The past few days we've been focussed on improving the user experience of CiviCRM. This also included optimizing page size and number of database queries invoked per page load. One of the things that has bothered me for some time was the inefficient implementation of the
quickform hierarchical select widget for our case. That form element is quite awesome and is used quite extensively within CiviCRM. However for our "mapping" case it generates a lot of javascript, which increases page size and duplicates a lot of data.
We can optimize this significantly based on our use cases.
The mapping (import / export / search builder / profile field selector) has a few characteristics:
- We generate an array of hierselect elements, each of which have the same functionality
- We have lots of elements for each of the select elements, however the number of unique select arrays that we use is very limited (typically 4-10). Most of the times we are just dealing with Individal fields, location types and phone types.
- Most of the select elements are hidden by default.
Our first attempt was to replace this with an AJAX driven selector. We made awesome progress with this, but soon realized that this would not be very efficient if each select element made one/more ajax requests to the web server. We poked around Dojo to see if we could deploy some local client side javascript to return the select element options based on the select element values. While dojo has got a flexible data store abstraction, we did not find a relevant abstraction that we could use. At this point, kurund realized that we could improve on our current hierselect which is basically a client javascript driven implementation. We looked into it and realized there were quite a few things we could fix.
The default export has 10 fields. This means that page has 10 hierselect elements, i.e 40 select elements. Since the select element displayed in the second selector is a function of the value chosen in the first selector, there is a multiplicative effect with regard to the number of arrays needed for display. For a 10 field export hierarchical select generates 760 arrays. However in this specific case we only have 5 (individual,organization,household,phone type,location type) unique arrays. Thus we can optimize it significantly by generating 5 arrays and then assigning the other 755 variables to the 5 unique arrays. This immediately reduces the page size by a significant factor. This optimization is quite generic and can be applied to all hierselect.
The export has 10 identical hierselect elements. Rather than generating 76 elements per hierselect, we can optimize it by generating just the first copy and allowing all the other hierselect to point to the arrays generated by the first element. Doing this gives us another significant page size reduction. This optimization is only applicable if you have an array of hierselects. It will do the wrong thing if you have two different hierselects on the same page.
Finally we generated a fair amount of javascript to hide the various select elements. On first page load only 10 of the 40 select elements are shown. i.e. 30 of them are hidden. We had 30 statements similar to: "document.Map['mapper[1][0][1]'].style.display = 'none';". The statements would vary in terms of the indices. We now store the indices in an array, and then perform the above using a simple for loop. This reduces the page size a wee bit, not a lot compared to the above two changes, but still a fair amount. It is also more elegant and cleaner, and hence needed to be done.
With the above 3 changes, for a 50 field export the size of the generated page was reduced from 900K to 90K. i.e the above changes managed to reduce the size of the page by a factor of 10. Not bad for a day of work :) The next major change would be to figure out how to share the option value pairs for the first set of select elements (which are in html). I'm not sure html allows you to do that easily, so we'll skip that for now.
I'm a bit disappointed that we did not figure and clean this up in earlier versions of CiviCRM. We will be more aggressive in fixing such major issues much earlier in future.
A future blog post will focus on the database query optimization.
