Deduplication deduplication
The Deduplication activity is a Targeting activity. This activity allows you to delete duplicates in the result(s) of the inbound activities, for example duplicated profiles in the recipient list. The Deduplication activity is generally used following targeting activities, and before activities that allow the use of targeted data.
Configure the Deduplication activity deduplication-configuration
Follow these steps to configure the Deduplication activity:
-
Add a Deduplication activity to your Orchestrated campaign.
-
In the Fields to identify duplicates section, click the Add attribute button to specify the fields for which the identical values allow the duplicates to be identified, such as: email address, first name, last name, etc. The order of the fields allows you to specify those to process first.
-
In the Deduplication settings section, choose how many unique records to keep using the Duplicates to keep field. The default is 1, which keeps one record per duplicate group. Set it to 0 to keep all duplicates.
For example, if records A and B are duplicates of Y and record C is a duplicate of Z:
- If the value of the field is 1: Only the Y and Z records are kept.
- If the value of the field is 0: All records (A, B, C, Y, Z) are kept.
- If the value of the field is 2: C and Z are kept, plus two values from A, B, and Y, randomly or based on your deduplication method.
-
Choose a Deduplication Method, this defines how the system decides which records to keep from each group of duplicates:
- Random selection: Randomly selects the record to be kept out of the duplicates.
- Using an expression: Keeps records with the highest or lowest value based on an expression you define.
- Non-empty values: Keeps records where the selected field is not empty, e.g. keep only profiles with a phone number.
- Following a list of values: Allows you to prioritize specific values for one or more fields, e.g. you can give priority to records with “Country” set to France. Click Attribute to choose a field or create a custom expression. Use the Add button to enter preferred values in the priority order.
-
Check the Generate complement option if you wish to exploit the remaining population. The complement consists of all the duplicates. An additional transition will then be added to the activity.
Example deduplication-example
In the following example, a Deduplication activity is used to remove duplicate records from the target audience before sending a delivery. The audience is first filtered to include only profiles with a non-empty Email field. Then, the Deduplication activity uses the Email address to identify and exclude duplicates.