21 February, 2007
By shot
Filed under Architecture, CiviCRM
I got myself a new development machine last Thursday, and the old one spectacularly failed two hours later (talk about timing). This means I ended up with a clean install of Ubuntu (I went with the development version of Feisty Fawn, but the below should work for the stable Edgy Eft as well) and can share with you how to setup a CiviCRM development sandbox from scratch. This tutorial is very Ubuntu-centric, but should be easily adaptable to other (especially Unix-based) operating systems. You can of course skip all the parts that are already working in your install. Webserver and PHP I’m a big fan of lighttpd, but of course Apache would be a good choise as well. First, install the lighttpd package. This will get the server installed and running; you can verify it by opening http://... Read more
17 February, 2007
By lobo
Filed under Architecture, CiviCRM

I've spent a fair amount of the weekend attempting to install FishEye from Cenqua. Its awesome that companies like Atlassian (wiki and issue tracking software) and Cenqua (coincidentally both these firms are Australian!) give away free licenses to open source projects like CiviCRM.

FishEye helps you analyze, search, share and monitor your source code repository (in our case svn). We've always wanted something a bit more fancy than what subversion offers out of the box (a vanilla http interface to the code). We also wanted better integration with JIRA and link issues to the appropriate revision of the code. FishEye promised to deliver on both these cases. We had also heard pretty good reviews of the product.

Getting a license was incredibly easy. I filled out a form and gave them all the information and in 24 hours received the license. Installation was quite easy and I had...

Read more
14 February, 2007
By lobo
Filed under Architecture, CiviCRM

An interesting discussion spawned on the civicrm-dev list recently regarding our implementation of custom groups and fields. We have been super cautious about this and have advised people not to create more than 20 custom fields per object (contact, activity, group, relationship etc). However creating a custom field is relatively easy and people have been doing it at a fast and furious pace. We've heard of implementations with more than 100 custom fields. This has worked for some people, but a large number of custom fields has the following problems:

  • Custom values are stored in a thin table, one per value. This table will grow significantly for large number of custom values
  • Primary Export is not possible since it will exceed the mysql JOIN limit of 61 tables
  • You cannot make a large number of these...
Read more
29 December, 2006
Filed under v1.7, Architecture

I just checked in a few changes that allows a CiviCRM install to use a memcached server if available. We use the php memcached integration to make this possible.

To use this feature you'll need to do the following

  • Compile php with the --enable-memcache flag
  • Download and install memcached. Get the memcached daemon running on localhost and the default port (11211)
  • Define the CIVICRM_USE_MEMCACHE constant in your civicrm.settings.php file

Thats all there is to it. Currently we are storing the Config object and all the PseudoConstants in memcached and are seeing some performance improvements. As we figure out more bottlenecks, we'll migrate a few more core objects to the memcached partition.

We'll need to tweak things a bit to move this to production, but this looks like a pretty good first step. Overall, I'm quite happy with the integration...

Read more
27 December, 2006
Filed under v1.7, Architecture

I've spent a fair amount of time in the past two weeks figuring out how we could optimize and improve CiviCRM. Its been an interesting few days and I suspect will become more interesting over the next few days as we start implementing a few things. All this is in preparation for doing a pretty major load test for the Branner project.

I did read a few articles on the web about optimizing LAMP. The article was fairly helpful, but I disagree quite a bit with some of the conclusions. I think there is also a big difference between building something for one site vs building an open source product. I do agree that PEAR DB, QuickForm and Smarty add a lot of overhead to CiviCRM. On the other hand, I also think we get an incredible increase in productivity because of these components. If I was building stuff...

Read more
14 December, 2006
Filed under v1.7, Architecture, CiviCRM

One of our major goals for this year is to optimize CiviCRM to handle load in a graceful manner. This is extremely important for us with the Branner Project. One of our main goals for the project this year is to optimize CiviCRM to be able to handle the last weekend load nicely (similar to the slashdot effect, the application process has most of the applications being filed and completed the weekend it is due)

The application is a multi-step form, made up of approx 25-30 forms. We have data from last year that we can scramble and potentially get a good random data set that is fairly accurate. Writing a functional test for this is potentially quite painful, but there is the wonderful testing tool Selenium. We've been using it for testing CiviCRM with pretty decent results. We hope to make it part of our...

Read more
06 December, 2006
Filed under CiviMail, Architecture, CiviCRM

CiviMail, described in general previously, is our component for mass-mailing the CiviCRM contacts. In this entry, we’d like to get a bit more into the details on how CiviMail exactly works ‘under the hood’.

Recipients List Building

Getting the final list of recipients is not as easy as it may seem. If you look into the getRecipients() method of the CRM_Mailing_BAO_Mailing class, you’ll see that we’re building some temporary exclude (X_*) and include (I_*) tables, based on

  • the user’s selection – excluded/included CiviCRM groups,
  • the user’s selection – excluded/included previous mailings’ recipients,
  • whether the contacts already received this mailing in...
Read more
17 November, 2006
Filed under Architecture, CiviCRM

It is important for CiviCRM to have a full fledged un-structured search engine in addition to the current structured query. I don't think MySQL full text searching (MFTS) is a good model for a couple of reasons. Firstly MFTS is restricted to myisam tables and CiviCRM uses innodb tables. Secondly MFTS is still a table level search and i don't think it can handle hierarchical data. CiviCRM contacts are hierarchical data sets.

Would be great to integrate something like Lucene into CiviCRM. A potential work flow could be as follows:

1. Publish an xml specification of the CiviCRM data model. We have done a fair amount of this work for the Branner project. We could extend and automate this quite nicely using our code generator. Also xml fits quite nicely since we can represent hierarchical data

2. Extend the logging functionality so we are aware of all modifications to any...

Read more
14 November, 2006
Filed under Architecture, CiviCRM
This installment of our architecture series will introduce the templating system used by CiviCRM as the presentation layer (e.g. to actually render forms and pages). Every CiviCRM screen is "composed" from one or more template files. These files contain a combination of HTML tags, text, variables and (often) some code to control presentation logic. CiviCRM uses an open-source templating engine called Smarty. If you are planning on examining, debugging and/or and modifying CiviCRM screens - you'll want to spend some time reviewing Smarty's online documentation.

Finding the Template(s) Used by a Form or Page

The easiest way to find the base template used to render a particular CiviCRM screen is to do a "View Source" while on that screen. Then search for the string '.tpl file'. This will take you to a commented HTML line that looks like this (this is the Edit Contact form):... Read more
14 November, 2006
Filed under Architecture, CiviCRM

CiviCRM Forms and Wizards (multi-page forms) are based on PEAR's HTML_QuickForm_Controller. (QFC). QFC in turn is based on HTML_QuickForm (QF). It was easier for us to model a single form as a one page wizard, and hence all CiviCRM forms are instances of QFC

The basic Form object is CRM_Core_Form. All forms are derived from this class. Each derived class is expected to implement the following functions

  • function preProcess( ): This function is called before a form is built. All objects needed to build the form should be built in this function.
  • function buildQuickForm( ): This function builds the form. There are some helper functions in CRM_Core_Form to build some elements (Radio, Select, Yes/No etc). Classes typically call a mixture of these helper functions and QF functions directly
  • function setDefaultValues( ): Default values when the form is first rendered. Typically we use the same form...
Read more