Overview The statistical significance level is the chance that a relationship was due to the chance of random sampling. Thus if a correlation is significant at the .05 level, this means there is 5% chance or less that a correlation as strong or stronger than the given one would result from an unusual random sampling of data when in fact the correlation was zero. There are many, many specific significance tests. Common tests are listed below, but in addition each statistical procedure has associated significance tests which are discussed in the respective sections dealing with each procedure. Parametric Tests. Parametric tests make distributional assumptions, particularly that samples are normally distributed. When these assumptions are met, parametric tests are more powerful than their non-parametric counterparts and thus are preferable. Binomial one-sample test of significance of dichotomous distributions t-tests of the difference of means Normal curve z-tests of differences of means and proportions Nonparametric Tests. These tests do not assume the normal distribution. Chi-square tests of the significance of crosstabs and goodness-of-fit Fisher's exact test for 2-by-2 tables with a cell < 5 One-Sample runs test of randomness One-Sample Kolmogorov-Smirnov goodness-of-fit test Tests for two independent samples: Mann-Whitney U, Wald-Wolfowitz runs, Kolmogorov-Smirnov Z, and Moses extreme reactions tests Tests for more than two independent samples: Kruskal-Wallis, median, and Jonckheere-Terpstra tests Tests for two dependent samples: McNemar, marginal homogeneity, Wilcoxon, and sign tests Tests for more than two dependent samples: Friedman, Kendall's W, and Cochran Q tests Resampling methods for nonparametric significance estimates

Contents Key concepts and terms Assumptions Frequently asked questions Bibliography




Key Concepts and Terms

• When significance testing applies. The "p" probability level reported in significance tests is the probability that a coefficient or result as strong or stronger than the observed one would be expected to occur just by chance of sampling.
  • Random samples. A significance level at or below .05 means that were another random sample taken, one would expect a result as strong as the observed one less than 5% of the time. This is the default standard in social science for accepting that the result is not due to chance. Put another way, there is less than 5% chance of making a Type I error (see below). An implication is that the researcher may generalize to the population from which the sample was drawn when the significance level for a random sample is at or below .05.

  • Enumeration data. Enumeration data exists when the researcher has data on all the cases in the population of interest (ex., GPA data on all students at a college or budget data on all counties in a state). In practice, up to 5% missing data are often allowed and even more if it can be shown to be missing completely at random. With enumeration data there is no sampling and therefore the research issue toward which significance testing is directed is moot. Any result, even a very weak one, is a real result and in that sense is "significant". The researcher may generalize to the enumerated population.

    Effect size measures (ex., correlation) are reported for enumeration data but not significance tests. Since significance levels confound effect size and sample size, this is a misleading method of assessing what findings are important: effect size measures should be used instead. Nonetheless, some publishers insist on reporting significance levels anyway. If significance levels are reported for enumeration data, a footnote may be added stating, "Computed significance levels are reported in order to follow social science convention. However, as the data are an enumeration of all cases, the actual significance level for all findings is .000, not the computed level, which assumes the data are a random sample of size n." (where n is the researcher's population size).

  • Non-random samples. If data are non-randomly sampled, as in most "man-on-the-street" and most Internet surveys, one would like to assess significance but one cannot. Computed significance levels will be in error to an unknown degree. While bootstrap methods may be used to compute standard errors, the reported significance level says nothing about the representativeness of the non-random sample. The researcher is not entitled to generalize to the population from which the non-random sample was taken.

    The best that can be done with non-random sampled data is to establish that characteristics of the non-random sample are proportionate to the population to which the researcher wishes to generalize. For example, in a non-random sample of students at a college, college-level data may be available on gender and the researcher may establish that the proportion female in the non-random sample is proportionate to the known percentage of females in the college. Ideally the researcher would like to establish proportionality for key variables in the study. While significance tests do not make non-random data generalizable, the chi-square goodness of fit test can be used to determine if sampled data conform to a known distribution. This is discussed in the section on chi-square and a spreadsheet example is provided.

    This is daunting and usually impossible task, but even if all variables in the study could be established to be proportionate in the population, unmeasured variables may not be proportionate and will influence the findings. Confirmatory research is almost always impossible with non-random data and findings should be reported as exploratory, with limitations of the data reported and analyzed. If significance levels are reported for non-random data, a footnote may be added stating, "Computed significance levels are reported in order to follow social science convention. However, as the data are non-random, reported significance levels are in error to an unknown degree and are presented for exploratory purposes only."

  • Randomized experimental data. Randomizing subjects in an experimental setting does not, as is stated in some textbooks, displace the need for random sampling prior to randomization. For instance, if one has a non-random sample which happens to be heavily skewed toward members of a religious cult, randomization of subjects in a psychological experiment focused on homophobia will not allow generalization to a larger population (ex., Americans over 21) no matter how well-done the randomization process.

    For experimental data based on randomization of subjects, a significance level at or below .05 means that were another randomization of subjects undertaken, one would expect a result as strong as the observed one less than 5% of the time. A significant result means only that the result is not due to chance of randomization. It says nothing about whether, if one took another sample of subjects on which to base randomization, one would be likely to get results as strong as those observed. The researcher is not entitled to generalize to a larger population for the identical reasons cited above with regard to any other non-random sample. That experimental researchers routinely violate this principle does not change the fact that research based on experiments with randomized subjects can only be generalized to the intended population if the data are a random sample of that population. With a large enough number of subjects, randomization does control for unmeasured variables but if the sample is not random, significance levels will be in error to an unknown degree just as the sample is unrepresentative to an unknown degree.

• Significance and Type I Errors. Significance is the percent chance that a relationship found in the data is just due to an unlucky sample, such that if we took another sample we might find nothing. That is, significance is the chance of a Type I error: the chance of concluding we have a relationship when we do not (false positives). Social scientists often use the .05 level as a cutoff: if there is 5% or less chance that a relationship is just due to chance, we conclude the relationship is real (technically, we fail to accept the null hypothesis that the strength of the relationship is not different from zero). Significance testing is not appropriate for enumerations or non-random samples because it only deals with the the chance of Type I error based on a random sample. Any relationship, not matter how small, is a true relationship (barring measurement error) for an enumeration. We would like to make inferences for non-random samples, but that is impossible.

• Conservative tests. The more conservative the test, the less likely the researcher is to make a Type I error but the more likely to make a Type II error (see below). If a test is called "conservative," this means that Type I error (false positive findings) is actually less than the computed alpha significance level. Put another way, the p probability levels for a conservative test are computed higher than they might otherwise be. If a finding is significant for a conservative test, the finding is valid. However, with conservative tests, there is an increased likelihood of Type II errors (false negative findings). By contrast, an anti-conservative test (sometimes called a liberal test) would have greater actual chance of Type I error than indicated by the computed p value and the test would not be valid (which is why one hears about conservative tests fairly often, but not anti-conservative tests).

• Confidence limits set upper and lower bounds on the true population parameter for a given estimate for a given level of significance (ex., the .05 level). The confidence interval is the range within these bounds. For instance, for normally distributed data, the confidence limits for a true population mean are the sample mean plus or minus 1.96 times the standard error, as discussed in the section on normal curve z-tests. As another example, confidence limits are often reported in the media in relation to opinion polls, as when the percentage of Americans who support the president's policies is said to be, say, 60%, plus or minus 3 percentage points. Some researchers recommend reporting confidence limits wherever point (ex., mean) estimates and their significance are reported. This is because confidence limits provide additional information on the relative meaningfulness of the estimates. Thus significance has a different meaning when, for example, the confidence interval is the entire range of the data, as compared to the situation where the confidence interval is only ten percent of the range.

• Power and Type II Errors. Where significance deals with Type I errors, power deals with Type II errors. A Type II error is accepting a false null hypothesis (false negatives: thinking you do not have a relationship when in fact you do). Power is 1 - q, where q is the chance of making a Type II error. Social scientists often use the .80 level as a cutoff: there should be at least an 80% chance of not making a Type II error. This is more lenient than the .05 level used in signficance testing. Leniency is justified on the ground that greater care should be taken in asserting a relationship exists (as shown by significance < .05) than in failing to conclude that a relationship exists. Obviously, the .80 level of power is arbitrary and the researcher must set the level appropriate for his or her research needs. In general, there is a trade-off between significance and power. Selecting a stringent significance level such as .001 will increase the chance of Type II errors and thus will reduce the power of the test. However, if two types of significance tests show the same level of significance for given data, the test with the greater power is used. It should be noted that in practice many social scientists do not consider or report the power of the significance tests they use, though they should.

• One-tailed vs. two-tailed tests. A two-tailed test tests the probability of getting a result as strong as or stronger than the observed result, where "strong or stronger" refers to different in either direction (ex., that far above or below the mean, or that different from zero in either a positive or negative direction). In a one-tailed test, the researcher has ruled out interest in one of the directions (ex., negative values are impossible, or below-mean values are not of interest), and the test is the probability of getting a result as strong or stronger only in one direction. In t-tests of significance or normal curve z-tests, a .05 one-tailed test finding will correspond to a .10 two-tailed test finding; likewise a .05 two-tailed test finding will correspond to a .025 one-tailed test finding. That is, a one-tailed test is more "liberal" in the sense that it is easier to find a relationship to be significant.

• Asymptotic vs. exact vs. Monte Carlo significance. Most significance tests are asymptotic, meaning that they assume adequate sample size. When significance is computed for small datasets, sparse datasets, or unbalanced datasets can lead to erroneous conclusions. While cutoffs for these criteria vary according the data context, as rules of thumb, sample size should be 50 or more, count in cells formed by the factors should be > 5 for 80% of cells and no cell should have zero count, and balance between groups should never exceed 90:10. Exact tests are available in the SPSS Exact Tests add-on module. Exact tests do not face these limitations and can handle small sample size, sparse tables, and unbalanced designs, but may not be computed for large samples.

  For problematic large samples, Monte Carlo significance estimates are available, also in the Exact Tests module. Monte Carlo estimates are derived from repeated sampling of the current dataset to form empirical data distribution parameters (as opposed, for instance, to assuming a normal distribution). Monte Carlo estimates are data-driven and may overfit the data (may reflect noise in the current dataset), but have no distribution assumptions.




Assumptions

• Random sampling is assumed for inferential statistics (significance testing). "Inferential" refers to the fact that conclusions are drawn about relationships in the data based on inference from knowledge of the sampling distribution characteristics of known common forms of data distribution, notably the bell-shaped normal curve.

• Sample size is assumed not to be small. Since significance tests reflect both strength of association and sample size, making inference based on small samples may lead to excessive Tyoe I errors even for moderate or strong relationships.

• Substantive significance should not be assumed merely because statistical significance is demonstrated. For large samples, even very weak relationships may be statistically significant.

• A priori testing is assumed. That is, the significance tests undertaken should be ones selected a priori based on theory. If a posteriori tests are done, say on all possible crosstabulations in a dataset, in order to determine which are significant, then for the .05 significance level one would expect one such test in 20 to be a Type I error. Put another way, a posteriori testing a a nominal alpha significance level of .05 really is testing at an effective level which is much more lenient than that.

• Correspondence of significance levels with research purpose is assumed. Specifically, it is inappropriate to set a stringent significance level in exploratory research (a .10 level is acceptable in exploratory research). Likewise, it is inappropriate to set a lenient significance level in confirmatory research (a .10 level is not acceptable in most confirmatory research settings).

• Intervening and common anteceding variables are absent for purposes of causal inference. The observed significant relationship between A and B may be spurious if they share a common anteceding cause(ex., ice cream sales and fires appear related, but that is only because they share a mutual anteceding cause -- heat of the day). If there is an intervening variable (A causes C, which causes B), the relationship of A to B is indirect.




Frequently Asked Questions

• How is significance related to association?
  It isn't. A relationship can be strongly significant even when the association of variables is very weak (this may happen in large samples, where even weak associations may be found significant). Also, two variables may be strongly but not significantly associated (this may happen in small samples). That is, significance coefficients reflect not only the strength of association but also sample size. This is why researchers should report measures of both significance and of association.

• Should I "fail to accept" or should I "reject" the null hypothesis?
  If a result is significant, this means the relationship can be considered different from no relationship (zero). The null hypothesis would be that the relationship is zero. Some say the finding of significance means we fail to accept the null hypothesis. Others say it means we reject the null hypothesis. This second way of putting it is more accurate because a finding of non-significance is not enough to accept the null hypothesis.

  Technically there is a difference between on the one hand failing to reject the null hypothesis in the case of a finding of non-significance, and on the other hand actually accepting the null hypothesis. To accept the null hypothesis given a finding of non-significance, power should be .80 or higher. It is not enough that significance be higher in magnitude than .05.

• If cases in my sample are weighted, am I getting accurate significance values?
  Not necessarily. SPSS, for instance, computes significance levels based on the weighted total number of cases. This biases significance levels toward Type I errors (thinking you have significant results when you don't).




Bibliography

• Henkel, Ramon E. (1976). Tests of significance. Thousand Oaks, CA: Sage Publications.
• Sheskin, David (2003). Handbook of parametric and nonparametric statistical procedures: Third edition. Boca Raton, FL: Chapman and Hall/CRC.

Copyright 1998, 2008, 2009, 2011 by G. David Garson.
Last update, 12/16/2011.