In this blog I want to explain the round up we have done around the refactoring of the acl_contact_cache. In the previous sprints we discovered that a lot of the performance was slowed down by the way the acl_contact_cache was used (or rather not used at all). See also the previous blog post: https://civicrm.org/blog/jaapjansma/the-quest-for-performance-improvements-5th-sprint
At the socialist party they have 350.000 contacts and around 300 users who can access civicrm. Most of the users are only allowed to see only the members in their local chapter.
In the previous blog we explained the proof of concept. We now have implemented this proof of concept and the average performance increase was 60%.
We created a table which holds which user has access to which contacts. We then fill this table once in a few hours. See also issue...Read more
When email was first designed, security was not considered important and up until fairly recently it was still possible to send an email from any address and get away with it.
However, as spam, phishing and spoofing attacks by email have become increasingly common there have been various attempts to make email more secure. In the last year or so the major providers (AOL, Google, Microsoft etc.) have all seriously tightened their security and authentication requirements for validating and receiving email. The result of this is that a lot of legitimate email is now being classified as spam or rejected by those providers. In order to ensure that your email continues to be marked as legitimate and received by these larger providers it is now almost essential that you implement SPF, DKIM and DMARC on your domains otherwise many of your...Read more
The last two days we spent another sprint at socialist party to improve performance. And again we used a guy with database knowledge to analyze our queries. We have optimized a few things so far one of the major areas to improve next was the ACL part of queries. That is the part of the query to decide which contacts you are allowed to see. So at the socialist party they have local chapter administrator who are only allowed to see the active members in their local area.
To decide which contacts you are allowed to see a search query is rebuild to include this ACL part. So there is a join on the membership table and a join on the chapter-structure table and then a where expression is to filter on the chapter and to filter on active memberships. Meaning that the MySQL will scan all records in the table civicrm_contact which at the socialist party is around 350.000.
We have worked on a proof-of-concept solution so far which is to create a table with the user_id and which...Read more
JMA Consulting is pleased to welcome Jon Goldberg as our new Director of Operations effective today.
After a brief stint as a political organizer, Jon spent 13 years working in various capacities at a non-profit legal organization, primarily in IT. In 2010 he co-founded Palante Technology Cooperative and started their CiviCRM department, where he worked for 7 years. Outside of work, Jon can be found engaging in queer community organizing, (dis-)assembling electronics, and training parrots.
"I'm really excited to have Jon join us given his keen appreciation of how to help progressive organizations achieve their missions using CiviCRM. He's got a deep and wide knowledge of CiviCRM. I appreciate how he gives back to the community like through StackExchange, where he is the top ranked CiviCRM contributor," said Joe Murray, President of JMA Consulting and co-author of...Read more
Last week we had a fourth sprint to improve CiviCRM performance at the socialist party.
During this sprint we started with looking at why the screen for adding and editing memberships loaded slow. The issue reported was that it took some time before the end date field jumped from the right side of the screen to the middle of the screen. It turned out that as long as the field was displayed at the right side the screen was still loading. Timing this gave a time of about 18 seconds before the screen was fully loaded.
We discovered a few causes:
- Every request in CiviCRM was also logged in Google Analytics and in Piwik.
- The PHP function getGroupsHierarchy in CRM/Contact/BAO/Group.php took around 900ms to execute (see the issue: https://issues.civicrm.org/jira/browse/CRM-19831)
Since the previous sprint a guy with a lot of database knowledge has done some analysis of the queries and he came up with the following observations. Most queries are build in CRM_Contact_BAO_Query class and that class adds a join on the tables civicrm_group_contact and civicrm_group_contact_cache and a where clause with an or on both tables. See example query below.
SELECT contact_a.id AS contact_id FROM civicrm_contact contact_a LEFT JOIN civicrm_group_contact `civicrm_group_contact-2304` ON (contact_a...Read more
Three weeks ago I wrote about our quest for performance at the Socialist party. This week we had a follow up sprint and I want to thank you for all the comments on that blog.
During this sprint we have been looking into the direction of the amount of groups (+/- 2.700) and whether the amount of groups slowed down the system. We developed a script for deleting a set of groups from all database tables and we deleted around 2.400 groups from the system and we saw that this had an positive impact on the performance.
Before deleting the groups adding a new group took around 14 seconds. After removing 2.400 groups, adding a new group took around 3 seconds. So that gave us a direction in which we could look for a solution.
We also looked what would happened when we delete all contacts who have not a membership from the database and that also had a positive impact but not as huge...Read more
After the socialist party upgraded civicrm to version 4.6 a month ago they are experiencing performance issues. In this blog I will round up our quest for performance improvements. But before that some facts about their installation.
- +/- 350.000 contacts
- +/- 300 users who can access civicrm and see members in their local chapter
- +/- 2700 groups
- There are several campaign websites linked to CiviCRM and one of their successfully campaigns leads to 7500 new contacts per week in the system
- Running on a VPS with 4 CPU cores and 8GB of RAM
- around 40 active extensions
Since yesterday we have added New Relic as a monitoring tool. With New Relic we can monitor and look back in the history. We can also see all the details in each request. So we can analyze the performance.
Someone asked me to post this here - so that he can give it a try!
I've posted the details in a QA format including some of my slides from my CiviCON Lightning Talk on CiviCRM's StackExchange site:
Give it try!
Anyone who has tried to login to their CiviCRM database via their phone knows the feeling: utter helplessness. You would even be forgiven for thinking CiviCRM is actively hostile to the small screen.
This initial experience of the un-initiated CiviCRM user on the phone will probably remain until the eventual adoption of the Bootstrap framework (a CSS framework with built-in mobile/responsive elements).
What may be surprising to many, however, is that CiviCRM today is quite mobile friendly in all the important areas. You just have to put the pieces together.
After a long discussion on the Civi Partners list of the myriad ways CiviCRM developers have been integrating Boostrap into their projects, Allan Dixon started a page documenting them.
The Bootstrap page includes both the...Read more