In this article
Significance testing compares two measurements, considers observations and spread. It assesses confidence that differences are real, not random.
What is Significance Testing?
Significance Testing evaluates the difference between two measurements by taking into account the number of observations and the standard deviation or spread of the data. For instance, when conducting a customer satisfaction survey, you might have sent it to 1000 people last year and 5000 people this year. Last year, 500 people responded with "Likely to recommend" while this year, 2000 people responded with the same answer. Significance testing helps determine whether this change is real and not just due to random noise. It allows us to assess how confident we are that the improvement in customer satisfaction is not just a result of normal variations in data.
What Significance Testing Doesn't Do
When conducting significance testing, it's crucial to keep in mind that it does not provide a definite conclusion on whether you or anyone in your organization is doing anything right or wrong. Instead, it simply indicates that the differences observed are likely caused by something. However, that something could be something as innocuous as cultural differences in customer satisfaction between different countries. For instance, cultural norms may lead customers in Italy to give lower ratings compared to customers in Brazil. It's important to also consider the possibility that some of your less satisfied customers may not have responded to this year's survey, which could explain why your scores are significantly better than last year.
While significance testing is a valuable tool that can guide you to where you need to focus, it's essential to understand what the numbers are indicating and what they are not.
Where Can Significance Testing Be Used?
Significance testing can be used in the following widgets:
Chart (chart)
Data Grid (table)
Historical Benchmark (table)
Key Metrics Scorecard (table)
Confidence level
When conducting Significance Testing, you have the option to choose from six confidence levels. The higher the confidence level, the greater the certainty that the difference between the two groups being tested is real. For example, at a 95% confidence level, there is a 5% chance that the difference observed is just due to random noise in the numbers. If you opt for a lower confidence level of 80%, the likelihood of obtaining significant results increases, but so does the chance that the observed increase is just due to noise, with a one-in-five probability. The confidence level you choose affects the level of certainty that the differences are worth paying attention to.
In Studio, you can select the following confidence levels:
80
90
95
98
99
99.8
Minimum Sample Size
The Minimum Sample Size refers to the minimum number of responses required before conducting significance testing. This ensures that the results of the test are more accurate.