Developer tip: Managing multiple git repositories

When developing with CiviCRM, one often prepares a development-build on your workstation which includes several git repositories, e.g.

• Git repositories for the official CiviCRM code (civicrm-core, civicrm-packages, and one or more of the CMS repos -- civicrm-drupal/civicrm-joomla/civicrm-wordpress)
• Git repositories for some Drupal modules, Joomla extensions, or WordPress plugins
• Git repositories for some CiviCRM extensions

Managing all these git respositories in the development-build can be challenging, and a few techniques have arisen within our community:

• Manage them all manually. This gives precise control over checkouts, branches, merges, etc., but it can be cumbersome, and it's easy to make a mistake or omission when you have more than 2 repos.
• Manage them with "gitify" and "givi" (which are bundled in CiviCRM). These commands help with cloning and updating code (respectively), but they only work with CiviCRM's repositories and (for givi) deviate from the "vanilla" git workflows.
• Manage them with "drush make --working-copy". This is great at cloning the repos and works with any git repo.  However, after hacking various repos for half a day, there's no great way to ensure that they're all clean and up-to-date -- the safe bet is to destroy and rebuild (which is slow and destroys your git remotes, git hooks, local branches, etc).  And it only works with Drupal.
• Manage them with private shell scripts. This approach also works with any git repo, but it requires more scripting skills to adapt to your needs.
• (Note: Other techniques have arisen outside our community; see the footnote if the topic really interests you.)

I'd like to offer another tool -- it's not a full replacement for any of these, but it complements them.  The tool is git-scan. The basic assumptions of git-scan are:

• You have already created a development build by other means (with gitify, drush-make, composer, manual checkouts, etc).
• Your development build may include a dozen repos at once, but you generally focus on one at a time.
• If you're focusing on a particular repo, then you should use normal git commands to do the work -- "git checkout", "git push", etc.  These are the commands described in all the documentation and training materials for git.
• For all the other repos, you don't want to think too much about them.  You just want to know that they've got canonical code -- ie they check out a reasonable branch, don't have any local changes, and are up-to-date.
• You don't want to write any new configuration, manifests, or metadata files that are custom to your system -- after all, that's more learning and more maintenance!

These aren't hard assumptions -- if you're spreading focus among 2-3 repos for a particular project, git-scan will still help you stay organized, but the housekeeping work will grow as you spread your attention across more repos.

Usage

For example, when I start development in the morning:

• First, I make sure that development-build is in a clean state. Were there any changes I forgot to commit yesterday? If so, I need to commit them (or stash them or discard them). Did I checkout an experimental branches? If so, I need to get back to a normal branch. But with so many repos, it's hard to remember which ones I messed around with yesterday. "git scan status" scans for git repos and displays the status of each so that I can decide what housekeeping is needed.
• Second, I want to make sure that I have the latest code -- this helps ensure that my changes work correctly when all the developers' code comes together. "git scan update" scans for git repos and updates each. Note: "git scan update" is pretty conservative. If a repository has uncommitted changes, deviates from the main-line code, etc, then "git scan update" will skip it and remind you to take care of it. (In git-speak, the command will automatically "fast-forward" but will leave you in charge of any "merging" or "rebasing.")

For more details and examples, see the README.

Installation

On most Unix systems with PHP CLI, you can download the standalone Phar executable, e.g.

$ sudo wget https://download.civicrm.org/git-scan/git-scan.phar -O /usr/local/bin/git-scan
$ sudo chmod 755 /usr/local/bin/git-scan

If you're interested in doing development, you can checkout the code and dependencies via composer:

$ composer create-project totten/git-scan --keep-vcs

Please note: The package hasn't been broadly tested yet -- in particular, it's mostly been used with MAMP and some flavors of Linux.  You will need a working and reasonably recent/complete PHP CLI environment. If you try it in another environment, let me know if it works (or how it breaks).

Footnote: Other solutions

There are a few other solutions to the problem of managing multiple git repositories. I omitted them earlier because I haven't encountered people using them within our community, but they merit consideration. They present different trade-offs:

• git-plus includes a feature for updating multiple repos, and it generally has more actions than git-scan, but it's designed to check a single-level in the file-tree. A typical development-build based on Drupal, Joomla, or WordPress has code in many different locations in the file tree.
• mr and repo handle updates to git repos in many locations, but they require extra metadata/configuration for each repo. They also coordinate several aspects of the SCM workflow -- cloning, pulling, pushing. That is neat functionality, but (a) we have more specialized tools (drush-make, composer) which are arguably better at the cloning issue and (b) in practice, the workflows and norms for merging code vary depending on the particular repo/project. It seems unlikely that one could find any single combination of tool+metadata+workflow that allows active participation in PHP's (and CiviCRM's) diverse ecosystem.