New in 4.6 Dev: Composer and NodeJS

Summary

Beginning with CiviCRM v4.6, developing code for CiviCRM (civicrm-core.git) will require two additional tools: Composer and NodeJS. If you develop CiviCRM, then you should install them in order to continue development.

Who is affected?

• NOT AFFECTED: Site administrators who install CiviCRM (from .tar.gz or .zip files) do NOT need to install these tools.
• NOT AFFECTED: Developers who write modules or extensions do NOT need to install these tools.
• AFFECTED: Developers and maintainers who work with the CiviCRM git repositories (esp. the "master" branch -- v4.6) DO need to install these tools.

What to do?

If you are a developer who works with the CiviCRM git repositories, then please ensure that these tools are installed:

• Composer: The "composer" command downloads PHP libraries. You can check if it's already installed by going to the command-line and running "composer --help". For install instructions, see http://getcomposer.org/ .
• NodeJS: The "node" command executes Javascript code on the command-line. You can check if it's already installed by running "node --help". For install instructions, see http://nodejs.org/ .

Note: civicrm-buildkit is a bundle of tools useful in CiviCRM development. If you use buildkit, then you already have a copy of composer. However, NodeJS must be installed separately. (Buildkit only includes portable programs written in languages like PHP or bash. NodeJS requires native binaries that are tailored to your operating system. Fortunately, nodejs.org distributes pre-compiled binaries for most platforms, and recent Linux distributions -- like Ubuntu 14.04 -- provide a NodeJS package.)

Do I have to use these new tools?

If you've installed the tools, then you shouldn't need to do anything more. Whenever you run "setup.sh", it will automatically run the most important commands.

However, there are two times you might use the new tools:

• If you want to add, upgrade, or remove dependencies, then you may need to edit a .json file and/or run a command.
• If you work on the Javascript tests, then you may need to run "npm test" or "karma".

What if there are problems?

Civi has used the civicrm-packages process and phpunit for a long time -- so we've collectively learned a bit about how to support them. As with any newly introduced tool or process, there may be some quirks in adopting Composer or NodeJS. If you encounter problems with using these in Civi, please share your experience on the developer forum so that we can improve the process and make it easier for future developers.

In Depth

If you've installed Composer and NodeJS, then you can stop reading -- we've covered the urgent stuff. In the rest of the blog post, we'll dig a bit more into the why's and how's of this change.

In Depth: Why? Reason 1: To improve testing of Javascript code.

CiviCRM includes automated testing based on PHPUnit and PHPUnit-Selenium. PHPUnit is great for writing tests that cover server-side PHP code and database code. With the addition of Selenium, one can also write PHPUnit tests which perform end-to-end testing.

In recent development cycles, CiviCRM increasingly implements major features using Javascript logic and APIv3. For example, v4.5 introduces a CiviCase configuration GUI based on AngularJS, and v4.6 introduces a revised CiviMail GUI also based on AngularJS. As more code is written in Javascript, Javascript testing becomes more important. Unfortunatey, neither PHPUnit nor PHPUnit-Selenium is well-suited to testing client-side Javascript code.

The Javascript community has produced some great tools for Javascript testing. Generally, these tools require NodeJS. Specifically, in AngularJS the standard practice is to write tests with Karma and Jasmine -- which are better suited for testing Javascript code (more concise, more performant, better documentation). And they require NodeJS.

In Depth: Why? Reason 2: To improve management of libraries.

If a developer wants to write Javascript along with some tests, he must first download the NodeJS runtime -- and then download several other tools and libraries written in Javascript (Karma, Jasmine, AngularJS, etc). For any new developer (and many experienced developers), understanding and downloading all these dependencies can make the head spin. Forutnately, one can download dependencies automatically with a package-manager (such as "npm").

Dependencies are not a new problem for CiviCRM. With CiviCRM v1.0 - v4.5, CiviCRM's approach to dependencies was based entirely on the "packages" folder - which currently works like this: if CiviCRM needs a dependency "Foo", then a Civi developer downloads "Foo", copies it into the "packages" folder, and sends a pull-request to the "civicrm-packages.git" repository. This poses some trade-offs:

• Pro: It works with almost anything (PHP, JS, CSS, PNG, Java, etc -- basically anything except native binaries).
• Pro: It doesn't require any extra tools to perform a download. (Which was important in 2005 -- because the tooling sucked!)
• Pro: If you have the CiviCRM source code, then you have all the dependencies - there's nothing else to download.
• Pro: If one of the dependencies needs a small patch, you can put the patch in civicrm-packages.git -- without any administrative overhead or negotiation with the original developer.
• Con: It removes the incentive to collaborate with the original developer.
• Con: It's difficult to determine the list of dependencies and their versions. (The VERSIONS file isn't always up-to-date, and browsing the file-system doesn't reveal embedded dependencies.)
• Con: It's difficult to determine which dependencies have been patched - and why.
• Con: It's difficult to upgrade a dependency (because you need to determine if it's been patched).
• Con: If a developer wants to use a new library in some code, he must submit two PRs. The first PR (civicrm-packages.git) adds the library, and the second PR (civicrm-core.git) uses the library. Split PRs make it harder to manage development over a long period of time and interfere with automated testing of PRs.
• Con: PRs for new or upgraded libraries are so big that they can't be meaningfully read.
• Con: civicrm-packages.git grows large over time -- because it includes a copy of each version of of each dependency ever shipped with Civi.
• Con: It's not a well-received practice in the broader PHP open-source community. The recent "PHP renaissance" (circa 2011+) has been based on the broad adoption of "composer" as the standard solution for dependencies.

"npm" and "composer" are package-managers which follow a different approach. (There's actually a number of similar tools, each targetted at slightly different programming communities: CPAN for Perl, Gem for Ruby, Maven for Java, Composer for PHP, NPM for backend Javascript, and bower for frontend JS+CSS.) In each case, one creates a document (such as composer.json, package.json, or bower.json) which lists each of the required dependencies. Any developer who works on the application can download the dependencies with a small command (e.g. "composer install" or "npm install").

The trade-offs are basically the reverse:

• Con: Each package-manager works with different dependencies. ("composer" works with PHP; "npm" works with headless Javascript.)
• Con: You have to download extra tools ("composer" and "node/npm").
• Con: Dependencies are downloaded separately. If the distribution channel for any one dependency breaks, then you can't install a new developer site. (This is not as bad as it sounds -- package-managers include caching, and most dependencies rely on the same distribution channel as CiviCRM -- i.e. github.com and sf.net. If those go down, then you'd be SOL in any case.)
• Con: It's more difficult to use forked/patched code (but it is possible).
• Pro: It preserves the incentive to collaborage with the original developer.
• Pro: The list of dependencies (and their versions) is readily browsable and necessarily accurate.
• Pro: It's readily apparent when the application uses clean/official code from upstream -- and when it uses forked/patched code.
• Pro: When using forked/patched code, you can maintain the fork with git's branch/merge/rebase features.
• Pro: If a developer wants to add or upgrade a dependency, the developer can submit a single, concise PR (e.g. which updates the composer.json and uses the new library).
• Pro: "npm" and "composer" are well-recognized in the broader community of JS and PHP developers (respectively)

In Depth: How to add a new package in v4.6+?

Unfortunately, "composer" and "npm" are separate tools, and there is no grand-unified dependency-manager that will do everything we need for CiviCRM. However, the division of responsibility should be pretty clear among the tools. To add a new package in v4.6+:

• If the package is a PHP library (or tool), then follow the directions from http://getcomposer.org. In brief, this means running "composer require PKGNAME" (or editing "composer.json").
• If the package is a backend library (or tool) written in Javascript, then follow the directions from http://npmjs.com. In brief, this means running "npm install PKGNAME --save" (or editing "package.json").
• If the package is a frontend library (or tool) written in Javascript, then follow the directions from http://bower.io. In brief, this means running "bower install PKGNAME --save" (or editing "bower.json").
• If the package doesn't meet any of the above criteria, or if the upstream author doesn't support the appropriate package-manager, then add the code to civicrm-packages.git.

In Depth: How to upgrade a package in v4.6+?

If Civi includes the package by way of composer, npm, or bower, then simply follow the documentation for that tool. In brief, this generally means running a CLI command and/or editing the .json file.

If Civi includes the package by way of civicrm-packages.git, then consider moving it to composer, npm, or bower. This will involve a few steps:

• Determine current version of the package and determine if the package has been patched.
• If it's been patched, then identify the patch and start a discussion (e.g. on IRC or the developer forum) so that we can figure out if the patch is still needed, how we want to maintain it, etc.
• Otherwise...
  • Prepare a pull-request for civicrm-packages.git which removes the old package.
  • Prepare a pull-request for civicrm-core.git; specifically:
    • Add the package using composer, npm, or bower
    • Search the civicrm-core.git source tree for any references to the package.
    • Update any references which would break.

In Depth: See also...

• https://github.com/civicrm/civicrm-core/pull/4767
• https://github.com/civicrm/civicrm-core/pull/4829
• https://github.com/civicrm/civicrm-buildkit/pull/118