Statistical significance
What is statistical significance?
Statistical significance is a measure of whether the results from collected data are due to a specific cause, rather than chance or randomness. For brand lift studies, we test the hypothesis that exposure to the ad campaign led to the observed lifts between the exposed and control groups for the study's KPIs.
How does Lucid determine statistical significance?
A traditional two-tailed/sided z-test is used for Measurement campaigns. A z-test is a statistical test used to measure the difference in proportions between two populations of interest. For brand lift studies, these populations are the exposed and control groups, whose main difference is exposure to the ad campaign.
The margin of error is then calculated using the Z-score and the sample's standard error. If the margin of error is less than the absolute lift percentage, the result is statistically significant.
Margin of error
Margin of error is the product of the Z-score (critical value) for a specified confidence level and the standard error of the sample. Impact Measurement supports Z-score calculations at 80%, 85%, 90%, and 95% confidence levels. As the total sample size increases throughout a campaign, the standard error decreases, thereby lowering the margin of error.
Confidence level
The confidence level pertains to the repeatability of a study's results. Measurement defaults to a 90% confidence level, meaning that the results of a study would be expected 900 out of 1000 times in identical settings. The confidence level affects the calculation of the margin of error. As the confidence level decreases, the margin of error narrows, and thus the likelihood of obtaining statistically significant results increases (albeit with greater tolerance for false positives).
Why aren't my results statistically significant?
Many factors could contribute to lift results not being statistically significant:
Sufficient sample collection is needed to reduce sampling error and the margin of error, thereby obtaining trustworthy results.
If observing your campaign's lift results in the early stages or at more granular levels, such as for a particular demographic group or media variable, always keep sample size in mind.
If lift percentages have stabilized and show little difference between the exposed and control groups, the sample size will need to be greatly expanded to achieve statistical significance. This does not invalidate the study's findings.
Practical evaluation of the data in the context of your campaign's goals will help determine the appropriate next steps to ensure future campaigns achieve the desired market impact.
What does 'statistical significance' mean in the context of brand lift on my advertising campaign?
It means the measures and results displayed were not the result of chance and were likely a direct result of your advertising campaign.
What does it mean if something is significant at 80% but not 90%?
The percentage refers to the confidence level, which pertains to the repeatability of a study's results. Impact Measurement defaults to a 90% confidence level, meaning the results of a study would be expected 900 out of 1000 times in identical settings.
The confidence level affects the margin of error calculation: as the confidence level decreases, the margin of error decreases, and thus the likelihood of statistically significant results increases.
Researchers often use results at 80% as directional findings, whereas results at 90% are more actionable.