Which email design works best, or where should the offer banner be added to gauge more interest from the target users? Comparing and analyzing the success and acceptance of different versions of Marketing Offers or Mailing or different Segments is the essence of today’s Marketing.
A popular saying goes, “comparison is the thief of joy,” does not hold good when it comes to comparing the results by different Marketing strategies. In this case, the comparison isn’t necessarily a bad thing. When you compare different variables in Marketing Offers, Marketing Mails, or website or your social media posts, you can uncover which one performs the best. This type of comparison and analysis can be best achieved by A/B testing, also called Split Testing.
HCL’s Unica suit has the capability to carry out the Comparative analysis from its different Inbound and outbound products for:
- Mailing comparison using A/B testing
To find out the most effective email design/message contents / Mailing subject in Unica Deliver.
- Campaign effectiveness using Target VS Control group
To check the success and effectiveness of Campaigning in Unica Campaign by comparing the Target group vs. Control group responses.
- Offer version comparison using A/B testing.
To find out the most effective Offers or Offer versions in Interact.
This article will see each of the above mentioned 3 methods in detail and how it is used for Comparative analysis.
Mailing comparison using A/B testing
In Unica Deliver, the User can configure and conduct A/B testing and automatically send the most effective version (winner) of an HTML email message to the audience. By sending the mailing as an A/B test, responses from the recipient are used to determine the most effective email design and message content. When a user sends a mailing as an A/B test, the system sends different email messages, or different versions of the same message, to between two and five sample groups of recipients.
In the A/B test, the user can compare how recipients respond to various combinations of different messages, layouts, content variations, or subject lines.
Different Criteria to Compare mailing in A/B Testing:
User can compare results for the following criteria:
- Different email layouts,
- Different content elements within the same email layout (Combination of different layouts and different content.)
- Different email subject lines.
Compare email layout in an A/B test:
For example, a user might conduct an A/B test to send a mailing that includes a link to open a new account or receive a special offer.
- Create one email communication that puts the call to action at the top of the message.
- Create another communication that places the link at the bottom of the message.
- Add each document as a separate test in the A/B test configuration to determine the more effective design.
To compare the different versions of the email layout, select Maximum Unique Clicks as the evaluation criterion.
Compare content designs in an A/B test.
In some cases, users might want to determine the effect of changes in content rather than changes in the layout of an email message.
- In email communication, add multiple versions of a sign-up button to a zone in email communication. In the configuration of the A/B test, add each version of the button as a separate test.
- Compare results between using a graphic or a human photograph in a call to action.
- Compare colors and graphics for headlines or sign-up buttons.
- See what happens when you include the word Free in a headline.
- Compare images that show people of different ages.
- Compare a shorter message to a longer message.
- Try different typefaces
- Compare a single-column layout to a multi-column layout.
Select Maximum Unique Clicks as the evaluation criterion.
Compare subject lines in an A/B test.
The objective of the test is to determine which subject line variation compels the greatest number of recipients to open the email.
- Users can create several email documents, each with a different email subject line, and add each document to a separate test split in an A/B test.
- Users might instead create a single email document in the Document Composer that contains multiple email subject lines.
To Compare the most appealing variation of the subject line, select Maximum unique views as the evaluation criteria OR Select Minimum unique complaints as the evaluation criteria.
Criteria to evaluate response in A/B test to decide the winner
As part of the planning for an A/B test, the User must define the system’s criteria to determine the winner. A/B testing provides three criteria for evaluating responses.
- Maximum unique views.
Considers how many different recipients opened the email. For example, you might specify Maximum unique views as an evaluation criterion if you were conducting an A/B test to compare email subject lines. The variation which gets the Maximum unique view will be the winner in A/B testing.
- Minimum unique complaints.
Considers how many different recipients reported the email as spam or asked to unsubscribe. The system records a complaint when it records an unsubscribe request or an ISP indication that the recipient marked the message as an unwanted email or spam. For example, if a user is comparing email subject lines, using Minimum unique complaints as an evaluation criterion can help you to identify subject line variations to avoid the current mailing. A variation that gets Minimum unique complaints will be the winner in A/B testing.
- Maximum unique clicks.
Considers how many different recipients clicked at least one link in the email. For example, users might use Maximum unique clicks as evaluation criteria to compare recipient response to different email designs, the placement. The variation which gets Maximum unique clicks will be the winner in A/B testing.
Target and Control group analysis
Comparing the responses of Target groups, VS Control groups is a powerful analysis tool for measuring a marketing campaign’s effectiveness. In Unica Campaign, Groups that contain audience IDs which you purposely exclude from being targeted by the offer(s) for analysis purposes are called control cells.
Target groups / Cells that contain audience IDs that are targeted by the offer(s). Some portions of Audiences are purposely exclude based on different sampling techniques from marketing campaigns to ensure that they do not receive the offer. After the campaign runs, the marketer compares the activity of the control group against those who received the offer to determine the effectiveness of the campaign.
Target vs. Control In Unica:
In the Unica Campaign, Target and Control can be defined in Target Cell Spreadsheet or in the MailList / CallList processes box of the Flowcharts. The contact history tables are populated when the user runs these types of flowcharts in production mode. The contact history identifies the members of control cells and the offers that were withheld (not sent to controls).
Once Responses are received from audienceIDs from Target group VS responses received from audienceIDs of Control cells, it is populated into ResponseHistory of Unica Campaign. This History information helps analyze and compare the responses from target versus control cell for lift and ROI calculations.
Inferred responses from control groups
All responses from members of control groups (which are always hold-out controls in Campaign) are inferred responses. Since members of a control group did not receive any communication or Offers, they cannot have any tracking codes to return, so their responses are tracked with the matching the Attribute of Interest.
Control groups are always tracked using multiple attributions: every response from a member of a control group receives full credit.
There are different reports available in Unica Campaign to compare the responses from the Control group against the responses against the Target group, i.e., Lift Over Control Group (increase in response compared to the control group).
Analysis and Reports
Say there are 98 Policyholders whose Premium due date is 31st March. Now Company wants policyholders should renew the policy before the due date, so it offers a 5% discount to 78 policyholders (Target Group), and for the rest of the 20 policyholders (Control Group), do not give the offer. This is to compare the effectiveness of the 5% discount offer by checking the Target group’s response rate and the response rate of the Control group without any offer.
Out of 78 unique offer recipients, say 50 responded positively to the offer; in this case, the Response rate will be 64.10%, and out of say 20 customers to whom offers were not given and still 13 customers from this group deposited the Policy premium in that case the response rate will be 61.90%. Thus, there is no significant difference or impact of treating the customers with a 5% discount offer to say that the Campaign is not effective. In this case, the value of Lift Over Control Group is 3.55%, as shown in the below image.
Campaign performance comparison Report
A campaign might target a group of customers who do not have to check accounts with a checking account offer. Members of the control group are tracked to see if they open a checking account within the same time period as the checking account offer.
The higher the value for Lift Over Control Group indicates the high effectiveness of the Campaign. A single control group can be used as the control for multiple target groups. However, each target group can have only a single control group.
Offers or Offer version comparison in using A/B testing
In Interact, A/B testing was done for the comparative analysis for the responses received for Offers or Offer versions. It is done to test the best accepting offer version.
In the Interact treatment rule with all its properties fixed, but only one parameter varies, this parameter is a combination of offer and offers attributes.
In A/B testing of a rule, each variation of offer and/or offer attributes represents one branch with specified audience distribution.
This A/B testing is valid only till within the Effective and Expiration date specified. If the Expiration Date is null, it is in effect forever.
To enable A/B testing, click the Enabled checkbox on the A/B testing tab. This adds the base rule as the first branch in the Branch table. The additional branches can be added by click Add Branch.
That is the Base branch and is related to the Comparative branch.
To analyze A/B Testing results, a column “ABTestBranch” can be optionally added into UACI_CHStaging and UACI_DtlContactHist tables in the Campaign database to record the name of the selected branches when A/B testing is engaged in arbitrating the relevant smart rules.
The information in this column, together with other treatment fields, enables users to create reports on the performance on A/B testing branches for individual smart rules.
To increase Marketing effectiveness and Return on Investment, marketers should engage in a comparative analysis by using different techniques. Different HCL Unica Products uses different techniques like A/B Testing, Target vs. Control analysis to do Comparative analysis. It helps to avoid contact Fatigue, achieve a higher response rate, more acceptance, and greater customer satisfaction. To understand how this works in Unica here is a product guide for the same.