While doing a deployment for a large organization, our good friends at Rayogram hit upon a pretty severe export scalability issue. The primary export was failing for approx 40K rows. They contacted us to see if we can figure out whats happening and why. They first assumed that the big issue was using a custom feature (merge same address / merge household address) and wanted us to look at it and potentially optimize and fix the issue.
In general dealing with memory issues in PHP are not something i look forward to (it comes a close second to debugging core dumps in PHP). Basically you cannot afford to leak a lot of memory in every iteration. For the export, if you leak 10K for every iteration, u will need 400M for a 40K export not including any of the memory used before export was called. On my initial tracing of the code we were leaking between 15K of memory for every iteration. On my local labtop, i could process approx 5K contacts before running out of memory :(
It took me some time to get my head around the code and figure out what the potential problems were. I first eliminated our good friend DB_DataObject (which has a tendency to store things in a static global variables). I also ensured that we were free'ing various objects as soon as possible. Spent a few hours on this and realized that i was not making any significant progress.
Blog posts by lobo
Keep up-to-date with blogs from the core team, working groups, developers, users and champions worldwide. Subscribe to our newsletter to receive regular updates by email. We also have an RSS feed.
June 9, 2010
By lobo Filed under Architecture, Drupal, SchoolsWe've been hard at work implementing an online set of forms to collect family information for the school module. This is one of our final projects for the year and eliminates the tedious summer ritual for the admin staff of sending paper forms to the 200 school families and for the parent to fill out the same information every year. This also saves the admin staff from entering that same information into the SIS
We've built this work on the parent portal that we launched late last year. Parents can update information on themselves and their children anytime via their drupal account. The form is composed of 5 sub-forms:
Household Information: Name, Email, Phone and Address of the household. We currently support 2 household and 4 contacts. These are stored as CiviCRM contacts with a relationship link of type Parent / Child to the student. We created a custom group to store which household a parent belonged to. We did not use CiviCRM's household functionality.
May 20, 2010
By lobo Filed under CiviCRMAt CiviCon SF 2010, we met the folks from Emotive LLC (chang and matthew b) who expressed an interest in helping scale CiviCRM and in specific help with building an import system that can handle large data sets.
I started discussing this with chang on the forums and IRC. You can read our forum thread here: http://forum.civicrm.org/index.php/topic,13630.0.html
May 16, 2010
By lobo Filed under CiviCRM, Drupal, JoomlaThe past few weeks have been just an amazing ride through the CiviCRM universe. We've had some excellent training sessions, an awesome CiviCon and then the sprints which produced a nice new book and some major improvements to our translation process. We've also been spending a lot of time thinking about what we want to accomplish with CiviCRM 4.0.
May 10, 2010
By lobo Filed under CiviCRMWe've received a small amount of funding to help start our work on Canvassing and GOTV. We are still looking for sponsors, if this work is important to you, please consider supporting the project by making a contribution.
April 29, 2010
By lobo Filed under CiviCon, CiviCRM, Documentation, Drupal, SprintsThe past 8 days have been an amazing period for the CiviCRM community and core team members. Its been incredibly intense, extremely fulfilling and mind-blowing. A huge thank you and tip of the hat to the members of the community who participated in the event and came together from various parts of the world (asia, europe, north america) to push the project to greater heights, from a usability, documentation and localization viewpoint.
April 21, 2010
By lobo Filed under CiviCRM, DrupalWe had another super awesome day at DrupalCon SF 2010 today. Was great to watch and hear Cricket from Jane Goodall Institude proclaim: "I love CiviCRM!" to the audience :) To keep the momentum going, we've scheduled two BoF sessions for today. Please check the BoF notice board for the rooms.
April 19, 2010
By lobo Filed under Drupal, SchoolsWe had a great day at DrupalCon SF 2010 today, especially in our BoF sessions :) To keep the momentum going, we are planning to have a few more BoF sessions:
April 14, 2010
By lobo Filed under CiviCRM, Drupal, JoomlaThe FSF just put out a press release recommending CiviCRM for non-profits. We've been working with the FSF very closely over the past 4 months to ensure that CiviCRM is licensing compliant (i.e. all the packages that we use and ship within CiviCRM are all compatible). This is an important milestone and a great endorsement for the project.
Some snippets from the FSF press release
March 11, 2010
By lobo Filed under CiviCRM, DrupalPolitical groups and campaigns have been some of the earliest users of CiviCRM. We've had quite a few political parties using CiviCRM: Green Party of New Zealand, Green Party of Canada, Oregon State Democrats, Vermont Progressive Party and even the Pirate Party of Germany! One of the features missing from CiviCRM has been Canvassing, GOTV, PhoneList and WalkList functionality. This has been long requested and there have been various specifications on the wiki for this.
Earlier this year we worked with Progressive Technology Project (PTP) on CiviEngage, a Drupal Module that brings address parsing, walklist and phonelist support into CiviCRM. You can read more about this work here: Canvass and Phonebank.
At the same time Will Brownsberger, a state legislator from Massachusetts, started using CiviCRM to support his office and campaign operations. As part of his campaign he wrote a drupal/civicrm module to do voter canvassing. Will was kind enough to attend and demo his module at the CiviCRM Boston Developer Camp in February.
