Enhancing Greetings Handling

2013-07-15 02:27
Written by

English has been my favourite language for a long time now.

There is more than one reason to this. I feel I should write a long and beautiful essay (in English!) about it... But alas, that's not what I'm going to do now. (Sorry -- I hope you can bear the deprivation ;-) ) Rather, I need to help making the world a better place, by saving some poor German NGOs from starving. Which I mean to achieve by -- can you guess it? -- of course: by improving CiviCRM! Easy one :-)

And that's why I'm going to talk about one particular reason for liking the English language: the beautiful simplicity of (certain aspects of) English grammar. Or more to the point, about the fact that those not having the wonderful luck of living in regions doing all communication in English, often have to deal with more complicated grammatical rules -- including, among others, pretty much the entire continental part of Europe.

(Actually, I will mostly talk about Germany only -- after all, I don't want to offend our dear fellow Continental Europeans by implying that they are just as complicated :-) Though I must say we *did* hear some anecdotal evidence -- i.e. rumours -- that some others might happen to be kinda somewhat in the some boat with this...)

So what does grammar got to do with greetings in CiviCRM? Don't worry -- getting there :-)

We all know there is this nifty greeting template system in CiviCRM. We just need to set up a default greeting template in line with the communication style of our organisation, and CiviCRM will generate according greetings for each contact automatically. We can further set up extra greeting templates, so we can easily select some other standard form for certain contacts who we want to address with a different level of familiarity for example. Only for special cases we need to enter individual greetings by hand.

Now the bad news: while implementing CiviCRM at digitalcourage and other German NGOs, we (that is, us folks from Software für Engagierte e.V.) soon realised that this approach doesn't really scale to German greetings grammar. Drat.

What makes English greetings easier to tackle? First and foremost, there is little gender distinction in English. The only difference between "Dear Mary" and "Dear John" is the name, which we just insert literally as a token -- the "Dear" part is always the same. When going more formal, it's "Dear Mr. Smith" vs. "Dear Mrs. Smith"; so we need an additional token for the gender-specific prefix ("Mr." or "Mrs.") -- which we can also take directly from the contact entry form.

Not so in Germany. We have to distinguish between "Sehr geehrte Frau Müller-Lüdenscheidt" and "Sehr geehrter Herr Müller-Lüdenscheidt" (spot the difference!); or (informal) "Liebe Brunhilde" vs. "Lieber Gottlieb". So it's not just the gender-specific prefix ("Herr" vs. "Frau") that differs, but also the actual greeting word. ("[Sehr] geehrter" vs. "[Sehr] geehrte"; "Lieber" vs. "Liebe".) We can't just insert some contact field as a token to make this distinction -- we need to have different templates for male and female forms, and select the right one manually for every single contact... Ouch.

Not convinced of the superiority of the English language yet?... ;-) Well, there's more. In modern-day English, while the greeting can range anywhere from "Hey slacker" to "Your Majesty", the pronoun used throughout the actual letter is (almost) always "you" -- formal and informal alike. It's not so easy where other languages come into the picture. (Sweden being the only exception I'm aware of...) In Germany for example, there is the familiar "Du" and the formal "Sie". (And also "Ihr", if you feel an urge to cover the majestic plural as well... ;-) )

Now how does that affect greetings? Well, indirectly it does: to properly handle formal and informal modes of communication, we need to vary not only the greeting, but also the message body. We can handle this with CiviCRM of course -- but to do that, we need a separate, explicit data field telling us the mode for each contact. Which means we have to select the appropriate greeting template, *and* the matching pronoun (in a custom field) for each contact. Did I say "ouch" before?...

There is yet another issue with the greeting handling in Germany -- though this one is not really more complicated for a change, just different: the way titles are included in names.

(Well, it's not more complicated, unless we wanted to do it *really* correctly... But the finer rules are so intricate and obscure, that nobody really bothers to have them implemented in software :-) )

Titles are very important to Germans. So much so, that instead of appending them after the name like English titles ("MBA" etc.), in Germany virtually all titles are prepended, between the gender prefix and the actual name. Thus, all contact entry forms in Germany begin with the gender prefix, followed by a title field, and only then the first and last names. (There is actually a national standard for that...)

What's more, the title field in Germany is generally free-form: as titles are so important, there are way too many obscure titles and combinations of titles to make a select box with a number of fixed choices feasible. (How about a "Prof. Dr. Dipl.-Ing." for example? ;-) )

Note that not having a set of predefined choices here -- while it may sound horrible to us database nerds -- is not really a problem in this case: the title is usually not processed, but rather just included as a literal token in addresses and greetings -- so there is no real need for normalisation! (I'm surprised at myself for saying this... ;-) )

In the face of all these almost horrible shortcomings, I must wonder why (to the best of our knowledge) nobody has attacked this issue before? I subtly hinted above that these problems are not all exclusively German -- yet nobody else seems nearly as bothered as we are? (In lack of a better explanation, all I can do is resorting to stereotypes: attributing it to the proverbial German "Gründlichkeit" ;-) )

Whatever the reasons, the fact of the matter is that these waters remain uncharted thus far -- so we were left with no other choice but to set out on a heroic mission to solve this properly once and for all. In the course of this adventure, we first went through the deathly perils of intense brooding. Then through more brooding. Then some empirical research. (Read: failed early attempts at a working solution ;-) ) But most importantly, more brooding. And last but not least -- no doubt the hardest part -- convincing ourselves that we are indeed finally on the right track!

So what have we come up with? At the centre of our elegant and finely crafted solution, resides a simple yet powerful concept: let's process the greeting templates with Smarty, so we can include conditionals! (We already discussed that bit on IRC a while back...) This way, we can create a complex default greeting template, which will automatically adapt the wording to different genders and formal vs. informal addressing modes.

While technically this is not a big change, it has major consequences -- greeting handling will never be the same again... ;-) Which is to say, using this will somewhat change the modus operandi for the greetings mechanism: rather than explicitly choosing between different greeting forms by selecting distinct templates, most variants can now be covered using only a single (complex) default template -- deriving the appropriate form from other data, such as a separate formality level field. Explicitly choosing different greetings will now only be necessary in special cases.

(But fear not, traveller: if this new world order intimidates you, it will still be possible of course to use the original approach!)

Allowing conditionals is only part of the story: we also need appropriate database fields to feed these conditionals. In our cautious adventurers' wisdom, we have already mapped out these parts of the expedition as well :-)

For the gender, let's keep using the conveniently already existing gender-specific name prefix field. Although it was originally conceived by its creators as a token to be directly included in the greeting as text, we can use it as a switch for driving conditionals just as well -- I'm sure the field won't mind ;-) (That's indeed how it is generally handled in German contact input forms... Remember that standard I mentioned?)

For recording the formality level, our grand plan stipulates planting a brand new field in the communication preferences block. We are not sure yet what talents to endow this field with exactly: for digitalcourage -- and probably most other organisations too -- a simple binary choice is fine; but it seem quite fathomable that some organisations -- perhaps in other countries? -- might feel more ambitious about this. Thus it could be preferable to implement this as a mighty configurable option group from the start?

Finally, we need some place to stick our fancy prefix titles. For this we shall introduce another new field, right in the main contact block with the other name components. We will make this one optional, so those lucky organisations only needing to deal with English naming conventions won't be bothered by it; while us others can tune up our happiness by ticking a simple configuration option :-)

(By the way, we contemplated extending our mission by adding Smarty processing to the address format templates as well, while already navigating neighbouring waters... However, there are some other deep perilous considerations lurking there, suggesting a whole new adventure on its own -- we shall follow up in a forum discussion.)

Now having established the best course, it's time to fetch the oars! In fact, we have already plunged ahead with an implementation of the Smarty processing in greeting templates -- you can marvel at our achievements so far in our greetings handling branch :-)

(To shelter the faint of heart, it might be desirable to add a configuration setting enabling this only optionally, the same way as Smarty processing in message templates needs to be enabled explicitly? This doesn't seem nearly as crucial here though, as the greeting templates are generally only set up once during the implementation phase, and not modified in normal usage...)

With none of the other stages of our planned route appearing too challenging either, we are confident we can reach the destination harbour within few weeks, so as to safely land this in time for CiviCRM 4.4. And then we shall see plentiful rejoicing, with one of the major frontiers for broader CiviCRM adaptation in Germany finally taken care of! :-)

But let's not forget our dear fellow passengers. Having previously established hard anecdotal evidence that we are not exactly alone in this boat, it's time now to pass the word to the other travellers. So, fellow Europeans -- or anyone else with a boarding pass for this cruise -- speak up now: do you think our plan can keep your side of the boat afloat as well?



Could you clarify why you couldn't achieve the same by using the custom greeting where you can put the "Geehrter Prof. Dr. Dipl.-Ing." or whatever fancy greeting you want?

To make it easier for the normal user, you could automatically set the greeting based on the gender of the person (eg Hans get Lieber and Hanna gets Liebe). it would avoid a lot of processing at the smarty level


You are entirely right: we could just add some custom hooks in each organisation, to fill the greetings according to any rules they want. Indeed, everyone could do so! So let's do away with the unnecessary greeting template system altogether. (It appears to be broken in master right now anyway...) This will shave off quite some code :-P

On a more serious note, your comment actually helped me understand more fully, why any approach combining the standard template processing with some custom hooks -- which I *did* consider as a potential solution at some point, but it never felt quite right -- would be misguided. Either we put all the rules in custom code; or we work only with templates managed through the UI (which becomes feasible now that we made them more powerful with Smarty processing) -- but not bits of both. Thanks for making me clearly see this fundamental point :-)

BTW, the "Prof. Dr. Dipl.-Ing." example is not about Smarty processing at all... Sorry for being unclear about it.


Why wouldn't it make sense to have an extension "german greeting" or "french greeting" that contains the custom code for each language? Then all the organisations in germany can install this extension. Seems to be easier than having custom template for each language with lots of {if}.

Beside, my users have already screwed up their fair share of templates of event registration and the al by misclosing the various {if event_full...}. Having the logic safely put in an extension that is the common civi way of packing extra features does rather seem to be a more natural way.

I would love to see the template language dumber, I personally like mustache approach, it would allow a better separation between business logic and presentation. That's not because our template language offers advanced logic (or even embedding php code) that one needs to use it. Civi offers already options to add logic in the template directly, via php code in the core, via php code in extensions or via templates in extensions. There is more than one way to do it and I'm afraid we'll still have to deal with the complexity of having to decide on a per case basis what's the best way of adding logic in the forseable future.

To clarify, is your idea to duplicate the gender->salutation calculation logic on every template that can possibly start with a salutation? I'd personnaly rather avoid duplicating code, even more so if it's mixed with "real content" and stored in a db that is UI editable.


I agree wholeheartedly that excess configurability is often a cop-out from providing good standard behaviour... (The address format handling I mentioned in a side remark is indeed such a case.)
However, it seems to me that greeting handling doesn't fall into this category. Organisations do have diverse communication styles -- even within one country. I know that for a fact. I believe it's genuinely useful to have a powerful customisation system for this, which doesn't require every organisation to create custom hooks, or to use some crude workarounds.
(Having useful default choices for each country as a good starting point is desirable too of course... But that's not limited to greetings, and a whole different topic :-) )
Regarding your question about duplicating the logic, I'm not entirely sure what you are asking -- but I think the answer is no. We don't want to include this logic in the actual message templates. (That wouldn't require any changes to CiviCRM. In fact, some organisations are doing it this way right now as a workaround...) What we want is simply to use the {contact.postal_greeting} token in mailings, just as intended. But for this we need to apply conditional logic in the *generation* of this token. And that's where Smarty comes in. (To be clear: when talking about "greeting templates", I'm referring to what is configured at admin/options/postal_greeting )


So the idea is to use the token, but being able to generate/modify it dynamically before using it. So far seems good, but not sure I see the benefit of doing it everytime a contact token is used vs saving/caching it. eg Lieber Olaf today is still going to be Lieber Olaf tomorrow, what's the point of not using the version that has been generated previously and re-generating it?

As for in which language should the token generation logic be written, can be smarty, php, lua, whatever. I'd argue that as your code isn't doing any layout, smarty doesn't seem a very well suited language, beside being less known than php in this context IMO. Have you seen hook_civicrm_tokenValues? seems it'd allow you to have your gender specific greeting with minimal work.

If your aim is to have a UI and save in the db, not sure I appreciate why storing smarty would be easier/better than storing php. I'm in the camp thinking that code should be in file and get all the git goodness, not in db, but I've been proven a minority opinion quite a few times ;). Do you think it would help admin that know how to code smarty but don't know how to use ftp?


It's not about generating the token dynamically -- but rather about generating it according to more complex rules. The caching mechanism is unchanged: the greeting templates are still processed when the contact is saved (or through a batch job), and the results cached in the DB. The only difference is that during template processing, after substituting CiviCRM tokens and before saving the result, it is additionally passed through Smarty.
About storing the rules in the DB vs. in the file system, I don't have a very strong opinion really. There is always a blurry line between configuration data and code -- and I've yet to see a really convincing solution to this fundamental problem... I tend to think though that the greeting rules fall more on the configuration data side: I'm pretty sure the vast majority of organisations will have an easier time managing them through the UI.
Regarding the language used to express the rules: I don't see anything in Smarty making it particularly specific to layout. It's just a generic text template language -- very much like PHP... Quite frankly, I never understood why people feel the need to use a separate template language in PHP applications -- for layout or otherwise -- considering that PHP is itself just an advanced template language. I certainly have no particular desire to use Smarty for greeting processing (or for anything else really) -- I'm just following CiviCRM conventions on this. (If it was up to me, I'd most likely do all greeting processing in SQL, so they come right out of the DB, without any caching...)

so the idea is to keep this:


But on the top of Dear {contact.individual_prefix} {contact.first_name}

You'd get {if contact.gender_id =1}Lieber{else}Lieber{/if} {contact.first_name}?



That's the idea.

(Except that we want to use the `prefix` field rather than `gender`... And the CiviCRM token within the Smarty condtional still needs its own pair of curly braces :-) )