Significance Tests for More Than Two Dependent Samples:
Friedman Test, Kendall's W, Cochran's Q

Overview This section treats significance tests when there are more than two samples and the samples are dependent (correlated). The tests are nonparametric, meaning that, unlike repeated measures ANOVA, they do not require that the variables be normally distributed. Panel studies in which the same individuals are interviewed at three or more time points are of this type. In SPSS, these tests assume that the Exact Tests add-on module be installed.

Contents Key concepts and terms Friedman test Kendall's W Cochran's Q Assumptions Frequently asked questions Bibliography

Key Concepts and Terms

• Dependent samples, also called related samples or correlated samples, are ones in which the response of the nth subject in one sample is partly a function of the response of the nth subject in an earlier sample. Examples of dependent samples include before-during-after samples of the same people or matched triplets of similar people.

• Type of significance estimate. The Exact button in the SPSS dialog above allows the researcher to select among asymptotic, exact, or Monte-Carlo estimates of the significance of the particular test value. These three types of estimates are discussed separately in the section on significance testing. This requires that the SPSS Exact Tests add-on module be installed.

  
  

• The Friedman Test

  • Purpose: The Friedman test, also known as Friedman two-way analysis of variance, tests the null hypothesis that measures from k dependent samples come from the same population. It is based on the rationale that if the groups do not differ on the criterion variable, then the rankings of each subject will be random and there will be no difference in mean ranks between groups on the criterion variable. That is, the Friedman test tests if multiple ordinal responses may be assumed to come from the same population.

  • In SPSS, select Analyze, Nonparametric Tests, K Related Samples; select Test Variables; under Test Type, select Friedman. OK.

  • Calculation. The Friedman test statistic is distributed approximately as chi-square, with (k - 1) degrees of freedom, where k is the number of groups in the criterion variable, from i = 1 to k. Let n be the number of subjects and let T_i be the sum of ranks for each group. Friedman chi-square is then computed by this formula:
    Chi-square_Friedman = ([12/nk(k + 1)]*[SUM(T_i²] - 3n(k + 1))

    SPSS prints out Friedman chi-square, degrees of freedom, n, and the corresponding significance level.

  • Interpretation: The Friedman test can be seen as a two-way analysis of variance with one observation per cell. It can also be seen as a repeated measures analysis of variance for one group. A typical use of the Friedman test is to test inter-rater reliability: cases (rows) are judges, variables (columns) are items being judged, and the Friedman statistic tests the hypothesis that there is no systematic difference in the ratings. If the significance of Friedman chi-square is less than .05, the researcher concludes that the groups do differ on the criterion variable.

  • Example. In the example illustrated here, 10 subjects rated a candidate over time, coded 0 = "would vote for someone else" and 1 = "would vote for candidate", with measurements taken eight times. . The asymptotic significance approximates the probability of obtaining a chi-square value as higher or higher than 15.27 in repeated trials (times, samples) if the ratings were not in fact different. Since there is less than 5% chance of that, the researcher concludes that the subjects' candidate ratings do differ across times (across dependent samples).

    Because there are only two values, 0 and 1, the Friedman test is identical to the Cochran's Q test discussed below. However, the Friedman test would have handled data where there were three or more ordinally arranged values (ex., 0 = "not vote", 1 = "may vote", 2 = "will vote"), whereas Cochran's Q is only for dichotomies.

    

  
  

• Kendall's W Test

  • Purpose; Kendall's W, referred to as a coefficient of concordance, is a normalization of the Friedman test to vary from 0 to 1.

  • In SPSS, select Analyze, Nonparametric Tests, K Related Samples; select Test Variables; under Test Type, select Kendall's W. OK.

  • Interpretation: Kendall's W can be interpreted as a coefficient of agreement among raters. Each case (row) is a rater and each variable (column) is an item being rated. The coefficient W ranges from 0 to 1, with 1 indicating complete inter-rater agreement, and 0 indicating complete disagreement among raters.

  • Example. As can be seen from the SPSS output for the same data below, the Kendall's W test and significance are the identical to Friedman's test. What is new is the Kendall's W value of .218, which is interpreted as a measure of inter-subject agreement on rating the candidate over time. Here, inter-rater agreement is relatively low.
    

  
  

  Cochran's Q Test

  • Purpose. Cochran's Q tests whether the percentages (proportions) of a given variable are the same across multiple dependent samples. As such, it extends the McNemar test beyond two related samples. Cochran's Q is used to test similar distributions among several dependent samples when variables are dichotomous. It tests the null hypothesis that the dependent samples have the same mean on a dichotomous variable. For dichotomous items coded 0, 1, the mean is the percentage of cases with the value "1". Cochran's ! a test if these proportions are equal.

  • In SPSS, select Analyze, Nonparametric Tests, K Related Samples; select Test Variables; under Test Type, select Cochran's Q. OK.

  • Interpretation: Cochran's Q is a chi-square statistic which is an extension of the McNemar test to more than two dependent samples. A large Q value indicates that the variation across trials (ex., across measurement times) is large compared to the variation within subjects (that is, for the set of all subjects for any given trial/time).

    SPSS reports the asymptotic probability level, which is the large-sample estimate of the probability of obtaining a Q value as large or larger than the observed one if the frequency of trial success (ex., getting "1" on a 0/1 item) is random. If sign(Q) < .05, then the researcher rejects the null hypothesis that the proportion of subjects having low (or high) values on a set of dichotomous items is the same across items. Cochran's Q is discussed further in the section on reliability.

  • Example. In the example illustrated here, 10 subjects rated a candidate over time, coded 0 = "would vote for someone else" and 1 = "would vote for candidate", with measurements taken eight times. The Q value of 15.72 corresponds to a significance level of .033, which leads to the inference that the proportion of subjects who would vote for the candidate is not stable across times.
    

    In fact, if descriptive statistics are requested using the Statistics button in the SPSS dialog, one can see that the proportion of subjects who would vote for the candidate is generally increasing over time:

    

  
  

Assumptions

• Random sampling is assumed, as in all significance tests.

• Dependent samples. The three tests discussed in this section all permit multiple dependent samples. Data may represent repeated measures from a single sample or may represent the same measure for each of multiple matched samples.

• Data distribution. The three tests discussed in this section are nonparametric tests which do not assume the normal distribution.

  
  

Frequently Asked Questions

• Where are these tests found in SPSS?
  From the SPSS menu, select Statistics, Nonparametric Tests, K Related Samples. In the "Tests for Several Related Samples" dialog box, check Friedman, Kendall's W, and/or Cochran's Q under "Test Type."

  
  

Bibliography

• Conover, W. J. (1980). Practical nonparametric statistics, 2nd ed. NY: John Wiley & Sons.
• Daniel, W. W. (1995). Biostatistics, 6th ed. NY: John Wiley & Sons.
• Siegel, Sidney (1956). Nonparametric statistics for the behavioral sciences. NY: McGraw-Hill. A standard reference work.
• Siegel, S. & Castellan, N. J. (1988). Nonparametric statistics for the behavioral sciences. New York: McGraw-Hill, Inc.. A newer version.