Blogs

While doing a deployment for a large organization, our good friends at Rayogram hit upon a pretty severe export scalability issue. The primary export was failing for approx 40K rows. They contacted us to see if we can figure out whats happening and why. They first assumed that the big issue was using a custom feature (merge same address / merge household address) and wanted us to look at it and potentially optimize and fix the issue. In general dealing with memory issues in PHP are not something i look forward to (it comes a close second to debugging core dumps in PHP). Basically you cannot afford to leak a lot of memory in every iteration. For the export, if you leak 10K for every iteration, u will need 400M for a 40K export not including any of the memory used before export was called. On my initial tracing of the code we were leaking between 15K of memory for every iteration. On my local labtop, i could process approx 5K contacts before running out of memory :( It took me some time to get my head around the code and figure out what the potential problems were. I first eliminated our good friend DB_DataObject (which has a tendency to store things in a static global variables). I also ensured that we were free'ing various objects as soon as possible. Spent a few hours on this and realized that i was not making any significant progress.

We will be live streaming the CiviCRM Toronto Meetup at June 15, 6:30-8:30pm EST.

To join us, go to

https://my.dimdim.com/rnao/

I'll be presenting on How to Plan a Successful CiviCRM Implementation. We're hoping that remote participants will be able to join in our discussions.

Relationship are a natural way of storing relations between contacts. However, it doesn't work so well if you have several hundreds of related contacts as the realationship tab becomes unreadable quite quickly. One of our client needed to associate each individual in their base to a local branch (we implemented a nice geo lookup based on the postal code to identify the local branch, but that's another story). It means that each local branch has 1000th of individuals. This could happen in other situations, for instance to keep a relationship between a "main teacher" and each pupil or who is the latest volunteer that contacted each person in a GOTV/ Canvassing campaign...

This year at the first CiviCon we had a great session lead by CiviAction's Ian Rhett about taking CiviCRM into the mobile space. It was a very engaging brainstorm session with a lot of great ideas thrown out there. My time this week at WWDC provided a little inspiration and time spent in a hotel room to create some mockups of what this could look like. I have posted some images below. I think one of the big conclusions at CiviCon is that the app could be built to do a lot of things but at this point working to achieve a CRUD (Create, Read, Update and Delete) app is probably the best starting place. This is a project that I would like to start to work on along with my team at emotive. I think the best approach is to develop the app using the Appcelerator Titanium platform. It's an open source development platform that translates javascript into native iPhone/iPad, Android and coming soon Blackberry apps. I figure that most of us know Javascript much more than we do C or C++ and this would provide us with a single codebase for multiple mobile platforms.

We've been hard at work implementing an online set of forms to collect family information for the school module. This is one of our final projects for the year and eliminates the tedious summer ritual for the admin staff of sending paper forms to the 200 school families and for the parent to fill out the same information every year. This also saves the admin staff from entering that same information into the SIS We've built this work on the parent portal that we launched late last year. Parents can update information on themselves and their children anytime via their drupal account. The form is composed of 5 sub-forms: Household Information: Name, Email, Phone and Address of the household. We currently support 2 household and 4 contacts. These are stored as CiviCRM contacts with a relationship link of type Parent / Child to the student. We created a custom group to store which household a parent belonged to. We did not use CiviCRM's household functionality.

A couple month ago I raised the question about CiviCRM importing scalability, and received mixed answers.

CiviCRM should be mostly used for data import, not for data cleaning. Most importing scalability issues stem from people's reliance on the system to perform both data clean up and import. CiviCRM has a great data cleaning process that should be taken advantage of when doing a large import.

Folks at the Electronic Frontier Foundation (EFF) are currently evaluating options for replacing their current suite of online tools. Their requirements include membership management, fundraising, email blasts and activism - and they would like to use open source solutions if possible (and they are already using Drupal for parts of their web presence). Their team has spent some time exploring CiviCRM's feature set, and Lobo and I met with them last week to discuss their findings, answer questions and evaluate fit and gaps.

I am writing this post to take community feedback on porting CiviCRM to PostgreSQL, the best way to do it, and to team up with any possible members willing to contribute to this effort. Here are my efforts until now. I have not been able to make it 'all' work on PostgreSQL, but certainly to a degree where I can see light at the end of the long tunnel. I am linking 3 PostgreSQL compatible files here:

structure.sql - this has the columns, primary keys, indexes, unique constraints data.sql - this is the sample data included with CiviCRM

So, amongst all the discussion of import methods lately I just wanted to flag another possible approach - creating a CiviCRM hook module for the Drupal migrate module There are a bunch of great blogs out there on how to use the table wizard module with the migrate module to import data from various mysql tables or views into Drupal nodes / users / taxonomies / content types - for example: http://www.lullabot.com/articles/drupal-data-imports-migrate-and-table-wizard The migrate module has a bunch of hooks to allow you to use it for other forms of migrations. I rattled up the module / code pasted at the end of this blog in a couple of hours as a proof of content for using this approach to CiviCRM imports. The code I threw together just offers up civicrm_contact table fields but I think there must be some clever ways to use existing import tools rather than this rudimentary approach.