Overview The statistical significance level is the chance that a relationship was due to the chance of random sampling. Thus if a correlation is significant at the .05 level, this means there is 5% chance or less that a correlation as strong or stronger than the given one would result from an unusual random sampling of data when in fact the correlation was zero. There are many, many specific significance tests. Common tests are listed below, but in addition each statistical procedure has associated significance tests which are discussed in the respective sections dealing with each procedure. Parametric Tests. Parametric tests make distributional assumptions, particularly that samples are normally distributed. When these assumptions are met, parametric tests are more powerful than their non-parametric counterparts and thus are preferable. Binomial one-sample test of significance of dichotomous distributions t-tests of the difference of means Normal curve z-tests of differences of means and proportions Nonparametric Tests. These tests do not assume the normal distribution. Chi-square tests of the significance of crosstabs and goodness-of-fit Fisher's exact test for 2-by-2 tables with a cell < 5 One-Sample runs test of randomness One-Sample Kolmogorov-Smirnov goodness-of-fit test Tests for two independent samples: Mann-Whitney U, Wald-Wolfowitz runs, Kolmogorov-Smirnov Z, and Moses extreme reactions tests Tests for more than two independent samples: Kruskal-Wallis, median, and Jonckheere-Terpstra tests Tests for two dependent samples: McNemar, marginal homogeneity, Wilcoxon, and sign tests Tests for more than two dependent samples: Friedman, Kendall's W, and Cochran Q tests Resampling methods for nonparametric significance estimates

Contents Key concepts and terms Assumptions Frequently asked questions Bibliography




Key Concepts and Terms

• Significance and Type I Errors. Significance is the percent chance that a relationship found in the data is just due to an unlucky sample, such that if we took another sample we might find nothing. That is, significance is the chance of a Type I error: the chance of concluding we have a relationship when we do not (false positives). Social scientists often use the .05 level as a cutoff: if there is 5% or less chance that a relationship is just due to chance, we conclude the relationship is real (technically, we fail to accept the null hypothesis that the strength of the relationship is not different from zero). Significance testing is not appropriate for enumerations or non-random samples because it only deals with the the chance of Type I error based on a random sample. Any relationship, not matter how small, is a true relationship (barring measurement error) for an enumeration. We would like to make inferences for non-random samples, but that is impossible.

• Conservative tests. The more conservative the test, the less likely the researcher is to make a Type I error but the more likely to make a Type II error (see below). If a test is called "conservative," this means that Type I error (false positive findings) is actually less than the computed alpha significance level. Put another way, the p probability levels for a conservative test are computed higher than they might otherwise be. If a finding is significant for a conservative test, the finding is valid. However, with conservative tests, there is an increased likelihood of Type II errors (false negative findings). By contrast, an anti-conservative test (sometimes called a liberal test) would have greater actual chance of Type I error than indicated by the computed p value and the test would not be valid (which is why one hears about conservative tests fairly often, but not anti-conservative tests).

• Confidence limits set upper and lower bounds on the true population parameter for a given estimate for a given level of significance (ex., the .05 level). The confidence interval is the range within these bounds. For instance, for normally distributed data, the confidence limits for a true population mean are the sample mean plus or minus 1.96 times the standard error, as discussed in the section on normal curve z-tests. As another example, confidence limits are often reported in the media in relation to opinion polls, as when the percentage of Americans who support the president's policies is said to be, say, 60%, plus or minus 3 percentage points. Some researchers recommend reporting confidence limits wherever point (ex., mean) estimates and their significance are reported. This is because confidence limits provide additional information on the relative meaningfulness of the estimates. Thus significance has a different meaning when, for example, the confidence interval is the entire range of the data, as compared to the situation where the confidence interval is only ten percent of the range.

• Power and Type II Errors. Where significance deals with Type I errors, power deals with Type II errors. A Type II error is accepting a false null hypothesis (false negatives: thinking you do not have a relationship when in fact you do). Power is 1 - q, where q is the chance of making a Type II error. Social scientists often use the .80 level as a cutoff: there should be at least an 80% chance of not making a Type II error. This is more lenient than the .05 level used in signficance testing. Leniency is justified on the ground that greater care should be taken in asserting a relationship exists (as shown by significance < .05) than in failing to conclude that a relationship exists. Obviously, the .80 level of power is arbitrary and the researcher must set the level appropriate for his or her research needs. In general, there is a trade-off between significance and power. Selecting a stringent significance level such as .001 will increase the chance of Type II errors and thus will reduce the power of the test. However, if two types of significance tests show the same level of significance for given data, the test with the greater power is used. It should be noted that in practice many social scientists do not consider or report the power of the significance tests they use, though they should.

• One-tailed vs. two-tailed tests. A two-tailed test tests the probability of getting a result as strong as or stronger than the observed result, where "strong or stronger" refers to different in either direction (ex., that far above or below the mean, or that different from zero in either a positive or negative direction). In a one-tailed test, the researcher has ruled out interest in one of the directions (ex., negative values are impossible, or below-mean values are not of interest), and the test is the probability of getting a result as strong or stronger only in one direction. In t-tests of significance or normal curve z-tests, a .05 one-tailed test finding will correspond to a .10 two-tailed test finding; likewise a .05 two-tailed test finding will correspond to a .025 one-tailed test finding. That is, a one-tailed test is more "liberal" in the sense that it is easier to find a relationship to be significant.

• Asymptotic vs. exact vs. Monte Carlo significance. Most significance tests are asymptotic, meaning that they assume adequate sample size. When significance is computed for small datasets, sparse datasets, or unbalanced datasets can lead to erroneous conclusions. While cutoffs for these criteria vary according the data context, as rules of thumb, sample size should be 50 or more, count in cells formed by the factors should be 5 or more for 80% of cells and no cell should have zero count, and balance between groups should never exceed 90:10. Exact tests are available in the SPSS Exact Tests add-on module. Exact tests do not face these limitations and can handle small sample size, sparse tables, and unbalanced designs, but may not be computed for large samples. For problematic large samples, Monte Carlo significance estimates are available, also in the Exact Tests module. Monte Carlo estimates are derived from repeated sampling of the current dataset to form empirical data distribution parameters (as opposed, for instance, to assuming a normal distribution). Monte Carlo estimates are data-driven and may overfit the data (may reflect noise in the current dataset).




Assumptions

• Random sampling is assumed for inferential statistics (significance testing). "Inferential" refers to the fact that conclusions are drawn about relationships in the data based on inference from knowledge of the sampling distribution characteristics of known common forms of data distribution, notably the bell-shaped normal curve.

• Sample size is assumed not to be small. Since significance tests reflect both strength of association and sample size, making inference based on small samples may lead to excessive Tyoe I errors even for moderate or strong relationships.

• Substantive significance should not be assumed merely because statistical significance is demonstrated. For large samples, even very weak relationships may be statistically significant.

• A priori testing is assumed. That is, the significance tests undertaken should be ones selected a priori based on theory. If a posteriori tests are done, say on all possible crosstabulations in a dataset, in order to determine which are significant, then for the .05 significance level one would expect one such test in 20 to be a Type I error. Put another way, a posteriori testing a a nominal alpha significance level of .05 really is testing at an effective level which is much more lenient than that.

• Correspondence of significance levels with research purpose is assumed. Specifically, it is inappropriate to set a stringent significance level in exploratory research (a .10 level is acceptable in exploratory research). Likewise, it is inappropriate to set a lenient significance level in confirmatory research (a .10 level is not acceptable in most confirmatory research settings).

• Intervening and common anteceding variables are absent for purposes of causal inference. The observed significant relationship between A and B may be spurious if they share a common anteceding cause(ex., ice cream sales and fires appear related, but that is only because they share a mutual anteceding cause -- heat of the day). If there is an intervening variable (A causes C, which causes B), the relationship of A to B is indirect.




Frequently Asked Questions

• How is significance related to association?
  It isn't. A relationship can be strongly significant even when the association of variables is very weak (this may happen in large samples, where even weak associations may be found significant). Also, two variables may be strongly but not significantly associated (this may happen in small samples). That is, significance coefficients reflect not only the strength of association but also sample size. This is why researchers should report measures of both significance and of association.

• Should I "fail to accept" or should I "reject" the null hypothesis?
  If a result is significant, this means the relationship can be considered different from no relationship (zero). The null hypothesis would be that the relationship is zero. Some say the finding of significance means we fail to accept the null hypothesis. Others say it means we reject the null hypothesis. This second way of putting it is more accurate because a finding of non-significance is not enough to accept the null hypothesis.

  Technically there is a difference between on the one hand failing to reject the null hypothesis in the case of a finding of non-significance, and on the other hand actually accepting the null hypothesis. To accept the null hypothesis given a finding of non-significance, power should be .80 or higher. It is not enough that significance be higher in magnitude than .05.

• How come significance is reported so often for enumeration or non-random data?
  Such uses should be accompanied by a footnote such as the following: "Because the present study does not use randomly sampled data, significance tests are not appropriate for inferential analysis. However, significance is reported here as an arbitrary criterion in deference to its widespread use in social science for exploratory analysis of non-random data."

• If cases in my sample are weighted, am I getting accurate significance values?
  Not necessarily. SPSS, for instance, computes significance levels based on the weighted total number of cases. This biases significance levels toward Type I errors (thinking you have significant results when you don't).




Bibliography

• Henkel, Ramon E. Tests of Significance. Thousand Oaks, CA: Sage Publications, 1976. ISBN 0803906528.

Copyright 1998, 2008, 2009 by G. David Garson.
Last update, 3/16/2009.