18 May, 2008
By lobo
Filed under v2.1, Architecture, CiviCRM
So i've been looking a bit closely at performance for 2.1 (both database and usability) and am attempting to boost it up significantly compared to 2.0 (and prior). Here are some of the highlights
  • We've introduced a new database cache table (civicrm_cache) to cache a few database queries that are repeated a lot. Some of the specific queries include listing all the fields available for the contact types (individual/org/household). This is a combination of the built-in fields (name, address etc) and the custom fields added by the user. This reduces the number of queries invoked from 5 complex queries to 1 simple cache query (and an un-serialize)
  • We've added a column (group_type) to the profile table (civicrm_uf_group), so we know the profile type rather than recompute it every time we need it.
  • Thanx to Dave Lange who reported and did some analysis on this, we've reduced the number of LOWER( dbColumnName) LIKE 'value' to skip the LOWER part. Email is now stored as lower...
Read more
13 May, 2008
By lobo
Filed under v2.1, Architecture, CiviCRM
One of the core features of CiviCRM is the ability to store a query as a group (smart group). This allows folks to create groups of contacts that share the same attribute(s). For e.g. A group of all the people in California. Thus contacts can be added/edited/deleted from the database, and the smart group will always give you all the contacts who live in CA at that point in time. I suspect this feature is heavily used by quite a few installs. Hierarchical organizations / political parties can separate their contacts based on electorate / voting segment / branch etc by using smart groups. For the NZ green party, this results in approx 80 smart groups. More details on their setup can be read in this blog post While smart groups are very convenient, the current implementation within CiviCRM is quite inefficient. The complexity of the queries increase quite significantly as... Read more
07 May, 2008
By lobo
Filed under v2.1, Architecture, CiviCRM
Today we spent a fair amount of time with Pete and Chris from the NZ greens. We got a pretty good overview of how the NZ Greens are using the system and some of their pain points. We saw some of the cool integration that Chris has done with the NZ voter database and linking the electorate ID to a CiviCRM contact ID, ability to merge the address from the voter database etc. Its kinda cool how folks can extend and integrate the systems with other db's and we need to make it significantly easier to enable folks to build such system within CiviCRM. I'll focus on one of their major issues. Restricting who can see what information on whom is a big issue with most political parties. There is one contact database for all their contacts. Some large percentage of the contacts are members. Members can belong to one of nine provinces. Each province is split up into smaller branches. There are approx 70 branches in the NZ green party system. There are users at every level. A branch user can see... Read more
28 April, 2008
By lobo
Filed under v2.1

A few members of the CiviCRM core team (dave, yashodha, michal and kurund) are visiting us in Nelson, NZ. We get together a couple of times a year which is quite helpful when we all return to various parts of the globe. This also gives the team a chance to visit and explore new places.

For this meeting, we decided to focus on improving a couple of things for the CiviCRM v2.1 release. The projects we decided to focus on are: AJAX widgets, Unit Testing, Component Architecture and Events. We focussed on Dojo and AJAX for the first week. The widget that my group tackled were:

  • WYSIWYG editors. Most folks were not too happy with the Dojo editor. In v2.1 we decided to give folks a choice between a couple of editors. In the 2.1 release we will give folks an option between TinyMCE and FCKEditor. There was a QuickForm extension for FCKEditor. This seemed a nice...
Read more
24 April, 2008
By shot
Filed under v2.1, CiviCRM

As hinted previously, I’ve been working on dedupe improvements for CiviCRM 2.1. The first thing I wanted to handle is to move as much of the dedupe search from the PHP code to the database side.

I created a wiki page describing the plan; it would be great if any interested parties gave it a read and commented. Thanks!

18 April, 2008
By lobo
Filed under v2.1, CiviCRM, Drupal, Joomla
Drupal 6 introduced a new menu system. CiviCRM had modeled its menu system after drupal 4.x which meant we needed to upgrade the menu code significantly to upgrade CiviCRM to Drupal 6. We took the opportunity to learn and understand more from drupal's new system and also simplify the interface between CiviCRM and Drupal with regard to the menu hook. The CiviCRM menu system has been based on the Drupal menu system, so all credit for the below goes to chx and the Drupal folks (all blame should be assigned to us). We have simplified and extended it a bit to meet our needs. Similar to Drupal menu system, the CiviCRM menu data now resides in the database. civicrm_menu stores all the information for a menu item.
CREATE TABLE civicrm_menu (
     id int unsigned NOT NULL AUTO_INCREMENT  ,
     domain_id int unsigned NOT NULL   ,
     path varchar(255)...
Read more
14 April, 2008
Filed under v2.1
Brian Shaughnessy is working on some layout improvements for a client - which may potentially result in some core code contributions. He posted this question on the forums today: I notice there's some inconsistencies regarding how some of the forms are laid out. Specifically, I'm running into forms that use a mix of table/tr/td tags with dl/dt/dd tags to layout the form labels and fields. This makes it more difficult to have consistency laying out the page using css, because those tags are structured differently. My personal preference is to use table tags exclusively. Because of the built-in structure of dl/dt/dd tags, I find them hard to layout on a consistent and growable/shrinkable manner. I know that in a strict-css-world we shouldn't use table tags for layout, but for a long list of form labels and fields, many of which have option contingencies, tables seem like the most logical way to handle layout. Since this is an issue we've struggled with, and because we're... Read more
09 April, 2008
By shot
Filed under v2.1, CiviCRM

We’re currently planning on various improvements to the duplicate contact finding (and merging) engine for CiviCRM 2.1. Among others, we plan to have a more responsive mechanism by caching the dedupe search results in a more effective way, add the ability to restrict deduping to a certain group, as well as move at least parts of the dedupe out of PHP and into MySQL (now that we require MySQL 5 anyway).

Thus, a plea: if you have a large real-world database that you could share with us, please do so – either by sending it to shot@civicrm.org encrypted with my GPG key (0xD128F14A) or mailing me to co-ordinate some other way to share it. Your database will not be disclosed to anyone, will be used solely for CiviCRM 2.1 dedupe profiling and will be destroyed once the profiling is done.

09 April, 2008
By lobo
Filed under v2.1, CiviCRM

Some members of the core CiviCRM team are getting together for a design and code sprint towards 2.1 in Nelson NZ from April 24th - May 8th. As part of improving CiviCRM, we'd love to get together with some CiviCRM users and learn a bit about how we can improve and make CiviCRM more effective.

Since most of that period is within the NZ school holidays, we figured that May 7th might be a good meeting date. We'd prefer to meet in Nelson, but are open to traveling to either Blenheim or Christchurch if more convenient to other folks. Please send us email or reply to this post and let us know if you can make it

07 April, 2008
By kurund
Filed under v2.1

Continuing the database cleanup that we started with v2.0, we are planning to drop some of the unused fields from civicrm_address table.

CiviCRM v2.0 ERD

List of fields that will be dropped.

1. street_number
2. street_number_suffix
3. street_number_predirectional
4. street_name
5. street_type
6. street_number_postdirectional
7. street_unit
8. supplemental_address_3

These fields were originally included with the intent that they would be useful for walk-lists or other use cases where address segments were needed. However, as far as we know - no one has taken advantage of these pieces of data. Hence we are planning to drop it in v2.1. Currently you cannot add/update these fields through UI.

So if anyone wants these fields please let us know asap.