1. Címlap
  2. Blogs
  3. Blogs

MongoDB as civicrm cache with new config

Közzétéve
2013-03-16 23:00
Written by
deepak.srivastava - member of the CiviCRM community - view blog guidelines

After my previous blog post, i have been working on making progress on working model w.r.t NoSQL and config. Starting with civicrm cache was a good idea. Keeping in mind NoSQL, new config system and what Eileen has already done with settings, here is what i planned and accomplished  :

Placed cache and cache-type (mongodb) settings file on disk with default values
Sticking to plans in previous post, and eileen's work on richer metadata, created two files in settings directory of civicrm. Richer metadata does help in creating web configuration forms without any extra coding for the config-items, and i was able to use it for building the new simple config UI.

deepak@bfc:civicrm$ ls settings
....
system.cache.php
system.cache.mongodb.php
....

deepak@bfc:settings$ cat system.cache.php
return array(
  'class' => array(
    'group_name' => 'system.cache',
    'group' => 'cache',
....
    'default' => 'ArrayCache',
    'title'   => 'Cache Class',
....
    'help_text'   => 'default is mongodb if not set',
  ),
);

deepak@bfc:settings$ cat system.cache.mongodb.php
return array(
  'host' => array(
    'group_name' => 'system.cache.mongodb',
    'group' => 'cache',
    'name' => 'host',
    'default' => 'localhost',
....
  ),
  'db' => array(
    'group_name' => 'system.cache.mongodb',
    'group' => 'cache',
    'name' => 'db',
    'default' => 'civicrm',
    'title'   => 'Database',
....
  ),
....

 

Default setting is read and stored in db - civicrm_settings table
During initialization or when a setting is asked for, setting table is loaded with values from config files - system.cache and system.cache.mongodb. To what's already available in civicrm v4.3, it also allows users to override default values.

Every time a setting is stored in db, a new config file is created on disk
When a setting is altered / saved in setting table, a new file with same values is also written to disk in .../files/civicrm/ConfigAndLog/ directory. The file structure is exactly same as that residing in civicrm/settings directory with an additional attribute 'value' holding the latest value for the config item.

deepak@bfc:~$ ls /var/www/mongo.loc/sites/default/files/civicrm/ConfigAndLog/
CiviCRM.db145e70b362f94fbfe4c5a865ca538c.log
system.cache.mongodb.php
system.cache.php

These files when copied to settings directory could serve as default files. Or when copied to a different installs would help to migrate config settings.

Provide a UI to allow user to tweak default setting, reset to default and restore settings from files on disk
Needed a single interface to update cache settings, reset to default values at any point of time and be able to restore settings from files in ConfigAndLog directory also helping in migration.


 
UI here allows to update cache settings, though has been coded in a way that any config item (present in settings directory) could be specified against "gname" url parameter.

Make cache code pickup cache settings and work with cache-type (mongodb here)
For me plugging in the mongodb wasn't so difficult, as it was to make caching work with new config. The cache now picks up settings from db to figure out the configured cache. And how mongodb works as a cache is driven by CRM/Utils/Cache/Mongodb.php file. For now it only works for mongodb but wouldn't be hard to provide support of other already supported caches per new config style.
 

The implementation serves as a prototype for new config approach that uses mongodb as cache.
The code is all available on github - https://github.com/deepak-srivastava/nosql-config.
 
Filed under
CiviCRM
Architecture

Comments

That looks really nice & tidy!

 

There are a couple of things in the UI I did that I find handy (edit in place on Select, text, boolean fields), being able to see all settings on one page, being able to see multiple domain settings. There might still be a place for a page that does those things.

 

Did you also look at the Drupal Variable module? It provides an interface to variables that includes the individual variable groups and the navigation between the groups.

 

One thing I expect you noticed is that the division of settings between groups currently is pretty broad. ie. 80% are 'Core' (that's not something I determined - I just 'documented' what was there).

From UI perspective, agree that other options - select / text / boolean still has a valuable place. And one page to see all is also a very useful thing, which i noticed in your extension. I think of it as tabbed ajaxyfied page like that of drupal config migration screen where all tabs extend a group. For example all "system.*.php" settings on one page with each config item a tab.

 

Drupal variable module i checked. Its very similar to what you trying to do. At the same time i don't think its going to survive in drupal 8. Check this - http://drupal.org/node/1882526 which is part of http://drupal.org/node/1637614 (another intresting  thing).

 

Agree with groups being broad, and thats what would become very clear once we have settings classified properly. Its a good project. Plus would help going beyond core settings that is components, extensions etc. I feel very tempted to clean things up but its a big project.

Seems like there are several enhancements in here. To make sure I understand, the scope of this includes a few distinct enhancements:

  • 1. Settings System -- Define a mechanism for storing/loading settings from a file (including a file-naming convention and an import/export system)
  • 2. Cache System (with Settings) -- Update the cache system to configure itself through the "Settings" layer (instead of PHP constants)
  • 3. Cache System (with MongoDB) -- Add a driver for storing caches in MongoDB (i.e. as an alternative to memcache or sql)

The three enhancements are demonstrated together in some specific use-cases. (The use-case might be "Site administrator configures CiviCRM to store caches in MongoDB by loading a config file.")

Some questions/comments:

  • [General] Are the enhancements otherwise independent -- if one wanted, could one break them out and do each enhancement separately? Or is there a deeper link between them?
  • [Setting System]:  How much of the metadata is exported and imported? If it exports "help_text" and "description", then wouldn't we have a problem with those fields growing stale? I feel like it makes sense to export/import the *value* of the setting but not all the metadata.
  • [Setting System]: restoreIntoDB() tries to read from a file named "$metaDataFolder/.../$group.php", but I don't think this will always restore the default settings. Firstly, it assumes that $group is the file-name. (I don't think that's enforced, and I think it was an intentional decision, but I can't remember the full rationale.) Secondly, third-party code (like extensions) may manipulate default settings -- at least, I remember Eileen working on that but don't know where it ended up. To restore defaults, have you considered using the "Setting.revert" API?
  • [Cache System (with Settings)] The changes to CRM_Utils_Cache allow it to determine the preferred cache-driver using the Setting System -- i.e. it reads settings from the setting export file (@include $config->configAndLogDir . "system.cache.php"), but that appears different from the normal "Setting::getItem" call. Is this snippet something that other developers should emulate/copy/paste? Or could you explain why this snippet is special?

Re: General - Yes enhancements could be done independently. Just that the scope in my enhancement was limited to using mongodb as cache.

Re: metadata import/export - agree that exporting metadata is redundant. For now in order to provide flexibility to also use / transfer config files (stored in log dir) as default files (civicrm/settings dir) i decided to also export metadata and not do any differentiation.

Re: restoreIntoDB - my idea here is that like xml menu files, all settings file (named based on new naming convention) serve as defaults i.e even if we truncate civicrm_setting table we could pull all info from these files. To limit the scope and in order to use whats already available 'group' column was best fit. Agree that something like GroupConfig or something would be a better choice and use 'group' for grouping one or more config-groups. I did check / investigate Setting.revert API, but as my assumptions were different (file based) didn't use it.
 
Re: cache system - The reason i didn't do an getItem() and read from file directly because, going into db was making another nested call (through getItem) to initialize the cache and getItem would never work. Its only for caching i think, that its a special case and i decided to use settings file.
 

Deepak - you obviously don't need to use the existing revert API - but I hope you are replacing the API with an appropriate api rather than moving the logic to the form layer?