Taking Config Management To Next Level

As part of my course i have been doing research on what would it require plug an external storage engine into CiviCRM, and how other open source systems doing it. Answer lies in a better config system which allows doing it in a scalable pluggable manner. As i make progress i'll be showing more reasons to get excited and curious about building a better config system. Drupal 8 has spent a fair bit of time on configuration management to make things easier. And we shouldn't shy learning from them and others.     
 

Standardizing config with hierarchical storage

The second layer of drupal's proposed configuration system is to store config data from file system to database for fast access. That is storing all config data in one place. CiviCRM has already been doing this decently for quite some time with civicrm_settings table. Whats new things to learn here are -

1. Standardising access to config

So we could have things like:

$config->system->cms->framework_version
$config->system->profile->double_optin
$config->component->civimail->replies

2. Storing information hierarchically in a tree of configuration objects

If we try to categories civicrm configuration and put them in tree kind of structure, this is how "group_name column of civicrm_settings table" or "config branches" may look like:

Please note these naming conventions are not fixed. I'm just throwing ideas on how things might look like. And this is how configs could be accessed or worked with.

$config = civicrm_config('civicrm.system.cms');
echo $config->userFramework;
echo $config->userFrameworkVersion;
echo $config->userFrameworkClass;
echo $config->userHookClass;
$config = civicrm_config('civicrm.system.profile');
$config->double-optin = true;
$config->add-to-group-double-optin = false;
$config->save();
// loads all config with 'system.' prefix
$config = civicrm_config('civicrm.system');
echo $config->profile->double-optin;
echo $config->cms->userFrameworkVersion;
echo $config->custom->templateDir;
echo $config->custom->phpDir;
// when we interested in core settings only
$config = civicrm_config('civicrm.system.core');
echo $config->base-url;
echo $config->root-dir;
echo $config->compile-dir;
// component name space
$mailConfig = civicrm_config('civicrm.component.civimail');
echo $mailConfig->workflow;
echo $mailConfig->replies;

3. Storing new revisions of config everytime its changed, helping doing rollbacks incase of failures.

4. Not loading the entire config for every page request.

So far thats about primary config. How about other site-wide configs like event / contribution pages, custom sets, profiles, custom sets .. ?  They could extend the same config system. Since storing is in the form of serialized arrays, it's possible to create new configs from existing configs, for example how cool would it be to create a new profile by merging two existing profiles, something like:

civicrm.system.profile.page3 = array_merge(civicrm.system.profile.page1, civicrm.system.profile.page2)

Like drupal we will also need an interface, drush / cli, apis to convert / interpret these special configs.

Managing Environments with configs on disk

Drupal likes to store all active configs on disk, under a special files/config directory.

A lot more discussion on what other file formats (xml, json, ini) could be with each having its own pros and cons can be seen here - http://groups.drupal.org/node/159044.

In case a config gets corrupted, is re-stored from disk. So a new config is always written to disk + database, with db storing all revisions. Drupal's idea of storing configs on disk is to make configuration portable and version control friendly. So configs could be migrated from say development to production environment - via SFTP, $vcs commit/push/update/pull.

Individual config file like block.block.bartik.search.yml looks like:

If you tried installing drupal 8, you would see new directories like:

Say you have two drupal sites setup example.com and dev.example.com at two different machines, and we interested in deployment of site name from dev to production. Both sites have their own data stores:

1. Change the site name at http://dev.example.com/admin/config/system/site-information.
2. Copy system.site.yml from ../dev.example.com/sites/default/files/config_*/active to ../example.com/sites/default/files/config_*/staging. (This can be done manually or via git).
3. At example.com visit admin/config/development/sync and click "Import all".

Another interesting thing here is to be able to store these configs to pluggable storage backends in addition to file systems, like apc, chdb, mongodb or some other system for increased performance. It seems drupal is going to extract benefit from symfony framework here.

I have been trying to figure out the way configurations are going to look like for these pluggable storage backends but haven't had any success. Considering that the primary settings.php is not going to change, seems like it could still be $conf or $database with a bit more generalised form of (in case of mongodb for example):

........
  $conf['mongodb_connections'] = array(
    // Connection name/alias
    'default' => array(
      // Omit USER:PASS@ if Mongo isn't configured to use authentication.
      'host' => 'localhost',
      // Database name
      'db' => 'drupal_mongo',
    ),
  );
   $conf['cache_default_class']         = 'DrupalMongoDBCache';
   $conf['page_cache_without_database'] = TRUE;
   $conf['page_cache_invoke_hooks']     = FALSE;
........

Managing Environment in CiviCRM

When doing config changes probably is the best time to think about providing better support for managing environment - dev / staging / production.

Considering the new config tree structure, civicrm.settings.php file format could be changed to - xml, json, ini or something else. Assuming xml for now here is how the file may look like:

civicrm.system.xml (civicrm.config.xml sounds better ?)

Interesting aspect of this zend type of style is to provide 'overrides' to specific values, and inherit rest of of the properties.
This is how initialization code may look like :

// this builds & stores config tree and say could return civicrm.system.core branch
// APPLICATION_ENV environment variable configured in virtual host, could alternatively be given preference
$config = civicrm_config_init('path/to/civicrm.config.xml', 'Staging');
// working with multiple databases
$config = civicrm_config('civicrm.system');
crm_db_activate($config->database->extra);
// execute query here to work with additional db with dsn named 'extra'
crm_db_activate();
// start working back with default db - $config->database->default

Whole idea of this post is looking at configs from new perspective, evaluating what drupal has already done and how could that help civicrm. When it comes to implementation there is still a lot to discuss and think about, for example converting nested array in nested objects, import/export/sync/parsing, writing to disk, api implementation. As we plan to move forward and make progress we'll be doing more detailed posts on a particular implementation.

You can read more about drupal's config initiative here incase you haven't already - http://groups.drupal.org/node/155559.