Many marketing managers want to see how different segments respond to different messaging. Psychological studies have shown that the human brain processes pictures first an then words¹. For this reason, it is a best practice to test the message and the creative execution as if they are one and the same thing².

Message testing is usually your last priority not your first. Within a particular target segment differences in messaging/creative execution ARE NOT typically associated with huge changes in responses. For this reason, we typically recommend that marketing managers look to test list sources and offers first before testing the message/creative execution.

Planning
Below is an example that tests has small-and-medium business customers (SMBs) respond when presented with different messaging/creative executions. The test is designed to see whether SMBs respond better to vertical or generalized messages.

The minimum sample size for each test cell is determined by the expected response rate, confidence interval, and allowable percentage error. Click here for a tool to calculate minimum required sample size.

For each campaign, a diagram needs to be constructed that visually depicts the testing we plan within each segment. Click here for an example of what we mean.

Backend Analysis
By properly designing test cells in the planning stage, backend analysis becomes simpler and allows us to learn with a high degree of confidence.

Using a confidence interval worksheet, one can determine whether we can lift response by varying the message/creative execution. Click here for an example.

In this example, at the 95% confidence level, the vertical messaging had an expected response between .72% and .88%. Similarity, the expected response for a generalized message is between .62% and .78%.As a result, we can conclude at the 95% confidence level that there is no statistical difference in response for SMB customers when presented with vertical or general messages.

Business impact
Vertical messaging appears to lift response slightly when compared with general messages. However, this result is not statistically significant. This suggests that the added expense and time involved in fielding vertical messages may not be worth it. Further
testing is warranted to explore this finding in more detail. For example, it may be that vertical messages work to lift response in a more dramatic way for some segments and not others.

Notes
1. Source: How Customers Think: Essential Insights into the Mind of the Market, by Gerald Zaltman, HBS Press, January 2003.
2. For a valid test, you should vary only the message/creative execution and not the type of DM package used.