The new de-dupe functionality, introduced in CiviCRM 1.8, is configurable under Administer CiviCRM → Duplicate Contact Rules. This post describes the meaning of the fields and the way their contents impact the de-dupe search engine.
The first decision to make after going to Administer CiviCRM → Duplicate Contact Rules is which rule to edit. For CiviCRM 1.8 we decided to allow one rule per every contact type (individual, organization and household); in future versions this can be extended to arbitrary list of rules.