Full-text Search for CiviCRM: Initial Thoughts ...

It is important for CiviCRM to have a full fledged un-structured search engine in addition to the current structured query. I don't think MySQL full text searching (MFTS) is a good model for a couple of reasons. Firstly MFTS is restricted to myisam tables and CiviCRM uses innodb tables. Secondly MFTS is still a table level search and i don't think it can handle hierarchical data. CiviCRM contacts are hierarchical data sets.

Would be great to integrate something like Lucene into CiviCRM. A potential work flow could be as follows:

1. Publish an xml specification of the CiviCRM data model. We have done a fair amount of this work for the Branner project. We could extend and automate this quite nicely using our code generator. Also xml fits quite nicely since we can represent hierarchical data

Announcing CiviCRM v1.6 Alpha ...

We are happy to announce that our 1.6 Alpha release is now available for preview and testing on our servers. We'd like to get as many folks trying out the new features as possible over the next week. This will help us move quickly to a downloadable Beta release - which is currently scheduled for November 29.

You can login to the sandbox at:


User - demo
Password - demo

Highlights of the release - and things we especially like folks to try out are:
* Ajax-based suggestive search in the Contact Search box (left column)
* Dynamically loaded sections in Advanced Search and Contact View Screens

Users, Organizations and Contributions in 1.6

There's been a lively discussion on the mailing list about requirements for linking CMS (Drupal / Joomla) Users to CiviCRM contacts and handling contributions from both types of contacts. We recognize that there is still work needed to handle the variety of use cases out there - but we've made decent progress in the upcoming 1.6 release:

  • CMS (Drupal / Joomla) users can be "linked" to either an Individual contact (default behavior) OR and Organization contact.
  • Sites will be able to provide a separate user registration link which will expose Organization profile(s) and create a user record, a new contact Organization record - and "link" them.

Team Goals for the Week of November 13

Closing in on the 1.6 release...

We made our goal for last week - reducing the existing queue items to 4! Most of the team has switched to unit testing mode. Each team member is taking on a group of Resolved issues from the master 1.6 queue and retesting the functionality. This has surfaced 2 new problems - bringing the total open count to 6 as of this morning.

In general, we are trying to make sure that a person other than the original assignee does the unit tests - so there is another set of eyes. After testing, team members are "signing off" on the issue by marking it Closed.

CiviCRM Architecture - Templating System

This installment of our architecture series will introduce the templating system used by CiviCRM as the presentation layer (e.g. to actually render forms and pages). Every CiviCRM screen is "composed" from one or more template files. These files contain a combination of HTML tags, text, variables and (often) some code to control presentation logic. CiviCRM uses an open-source templating engine called Smarty. If you are planning on examining, debugging and/or and modifying CiviCRM screens - you'll want to spend some time reviewing Smarty's online documentation.

CiviCRM Search Demystified

A common question on the mailing list has been about CiviCRM search, scalability and speed. Currently basic/advanced search has been designed to try to get as many matches as possible from your contact data. However trying to get as many matches as possible results in some major inefficiencies which does not work for data sets of reasonable size

Currently the default search does the following:

1. For any value typed in the name field, it searches the civicrm_contact table and the civicrm_email table. It connects the two tables using two LEFT JOIN's via the civicrm_location table. LEFT JOIN's are not very efficient sql operators

CiviCRM Architecture - CiviCRM Forms and Wizards

CiviCRM Forms and Wizards (multi-page forms) are based on PEAR's HTML_QuickForm_Controller. (QFC). QFC in turn is based on HTML_QuickForm (QF). It was easier for us to model a single form as a one page wizard, and hence all CiviCRM forms are instances of QFC

The basic Form object is CRM_Core_Form. All forms are derived from this class. Each derived class is expected to implement the following functions

  • function preProcess( ): This function is called before a form is built. All objects needed to build the form should be built in this function.
  • function buildQuickForm( ): This function builds the form. There are some helper functions in CRM_Core_Form to build some elements (Radio, Select, Yes/No etc). Classes typically call a mixture of these helper functions and QF functions directly

Installing CiviCRM 1.5 onto Drupal 4.7 WITHOUT using Command Line

This is what worked for me using Control Panel, FTP, File Manager and PHPMyAdmin.

Before you begin make sure you have a working Drupal 4.7 already installed. Have the name of the database, database user and password handy as well as the name of your mysql host. If you can’t find the name of your mysql host anywhere on your host’s site then your mysql host is probably localhost. Save yourself the aggravation and get all these things written down beside you before you even start installing CiviCRM.

1. Use FTP to upload the zip/tar CiviCRM directly into the drupal/modules folder.

2. Go to your Control Panel or whatever method you use to create new databases and create a new database called civicrm. You will already have a database user name and password from your Drupal installation. Go to your list of database users and add a user to the new civicrm database you just created. Make sure to give the user full privileges.

CiviCRM Architecture - PHP Class Structure

We now move onto the more interesting stuff of what really happens when a request is made and the objects that are responsible for building the response. The top level CiviCRM objects are:

  • Form (CRM_Core_Form): CiviCRM forms are based on HTML_QuickForm. For drupal folks, this is more of an object representation of the much loved form api
  • Wizard (CRM_Core_Controller): A set of CiviCRM Forms that collectively make up an action (Import Wizard, Mailing Wizard etc). For simplicty sake a one page Form is represented as a Wizard (CRM_Core_Controller_Simple)

CiviCRM Architecture - Object Structure and SQL/DAO Generation

CiviCRM uses the PEAR package DB_DataObject to access the mysql database. Doing so reduces the amount of sql we need to write to fetch / store values for a single table. For queries involving more than one table, we write them manually since since it was fairly painful to try to force complicated queries using DB_DataObject (we did use DB_DO for multiple table queries when we wrote EmailNow, in the end we felt the hoops we had to jump through was not worth the effort and hence the current scheme)

The basic PHP classes that represent a table are called DAO's (Data Access Objects). These are generated automatically by (xml/GenCode.php) which also generates the sql structure for that specific table. The xml files representing the schema is stored under CIVCRM_ROOT/xml/schema. The sql file generated (civicrm_41.mysql and civicrm_40.mysql) are stored under CIVICRM_ROOT/sql (note that the generated files are not present in the svn repository). The DAO files are stored under CIVICRM_ROOT/CRM/Core/DAO (and under appropriate directories based on what table it is, this is configured in the xml representation.