Significance Tests for Two Dependent Samples:
McNemar, Marginal Homogeneity, Sign, and Wilcoxon Tests

Overview The tests in this section apply to correlated samples, such as in before-after or matched-pairs studies. The McNemar test assesses the significance of the difference between two dependent samples when the variable of interest is a dichotomy. It is used primarily in before-after studies to test for an experimental effect. Also for two dependent samples, the marginal homogeneity test is an extension of the McNemar test for multinomial variables (more than two categorical values). The sign test and the more powerful Wilcoxon signed-ranks test are for two dependent samples when the variable of interest is continuous.

Contents Key concepts and terms McNemar test Marginal homogeneity test Sign test Wilcoxon test Assumptions Frequently asked questions Bibliography

Key Concepts and Terms

• Dependent samples, also called related samples or correlated samples, occur when the response of the nth person in the second sample is partly a function of the response of the nth person in the first sample. There are two common forms of sample dependency: (1) before-after and other studies in which the same people are surveyed at different points in time, including panel studies; and (2) matched-pairs studies in which similar people are surveyed at different points in time.

• Type of significance estimate. The Exact button in the SPSS dialog above allows the researcher to select among asymptotic, exact, or Monte-Carlo estimates of the significance of the particular test value. These three types of estimates are discussed separately in the section on significance testing. This requires that the SPSS Exact Tests add-on module be installed.

• SPSS interface. This set of tests is available in the SPSS menu by selecting Analyze, Nonparametric Tests, 2 Related Samples; then check which tests are desired in the dialog box illustrated below. Note that in SPSS, these tests require that the Exact Tests add-on module be installed.
  

McNemar test

• Symmetry. McNemar's test is sometimes called McNemar's test of symmetry or McNemar symmetry chi-square because it, and the marginal homogeneity test which extends it beyond dichotomous data, apply to square tables in which the diagonal reflects subjects who did not change between the before and after samples (or matched pair samples). The test variable may be nominal or ordinal. The test of symmetry tests whether the counts in cells above the diagonal differ from counts below the diagonal. If the two counts differ significantly, this reflects change between the samples, such as change due to an experimental effect between the before and after samples.

• Computation of the McNemar Test. For the McNemar test, data are arranged as below (for convenience, cell letters have been added in red):
  

  Note that you cannot have a simple table where the columns are "Time 1" and "Time 2", and the rows are "Exposed" and "Not Exposed", because then observations would appear twice in the table!

  The McNemar test uses the chi-square distribution, based on this formula:

  Chi-square = (|b - c| - 1)²)/(b + c) degrees-of-freedom = (rows - 1)(columns - 1) = 1

  The formula above is the usual continuity-corrected version of the McNemar test. One occasionally encounters the uncorrected version, sometimes used for large samples, in which case the "-1" term in the numerator is omitted.

  Example:

  Chi-square = (|20 - 10| - 1)²)/(20 + 10) = 81/30 = 2.7 d.f. = (2 - 1)(2 - 1) = 1

  
  

• McNemar test in SPSS. Select Analyze, Descriptive Statistics, Crosstabs; select Exposure 1 (coded 0=No; 1=Yes) as the column variable and select Exposure 2 as the row variable; click Statistics; select McNemar
  

• Interpreting McNemar test chi-square. Using a table of the distribution of chi-square, with 1 degree of freedom, if the computed chi-square is less than the critical value found in the table for the desired significance level (usually .05), then the difference between samples (ex., the difference between the before and after samples) is not significant. Thus, at the .05 level of significance, the critical value of chi-square is 3.841; since 2.700 computed for the example above is less than this, the difference between samples is not significant. Specifically, it is significant only at the .100 level.

The marginal homogeneity test

• The marginal homogeneity test is similar to the McNemar test, extending it to the case where the variable of interest may assume more than two nominal values. While it may be used with ordinal data, the sign test is preferred. In the example below, the marginal homogeneity test is output for the same two-valued data as for the McNemar test above. The result is the same (non-significance, meaning the two samples cannot be assumed to differ significantly), though with a smaller finding of non-significance (that is, the McNemar test is the more conservative test, making a finding of non-significance more likely for the same data).
  

The sign test

• The sign test is for two dependent samples, where the variable of interest is ordinal or higher. It examines the sign (plus, minus, tied) of the difference between data pairs (ex., between the 1st subject in the 1st sample and the 1st subject in the 2nd sample, etc., on up to the nth pair). When there is no significant difference between the two samples, the number of positive and negative differences will be the same or similar in the two samples. While the sign test may be used with interval data, the Wilcoxon test is preferred.

  The sign test is considered a weak test in that it simply assesses how often the two measures in a pair differ in being above or below the median. In contrast to the Wilcoxon test, the sign test does not measure how much the pair differs. When the sign test returns a finding of non-significance ( p > .05), the researcher concludes that one cannot assume the two samples differ.

• Computation of the sign test. Ties are be assigned to the plus or minus category by use of a random number table. The computation of the sign test is similar to the binomial test, once data have been recoded to plus/minus (1, 0) format, with p = q = .5, n = the number of paired observations, and r = the number of pluses or minuses, whichever are fewer.

• SPSS output.
  

  For the data above, a low-medium-high response was obtained for a weekday sample (Time1) and a weekend sample (Time2), interviewing the same people each time, with distributions shown in the crosstabulation above. The chi-square test, which (wrongly) assumes the samples are independent, shows a high significance level (p = .001), implying that responses significantly correlate between weekdays and weekends. The sign test, which (rightly) assumes the two responses are related samples (not independent) returns a finding of non-significance (p = .490), indicating that one cannot assume that the weekday sample (Time1) differs from the weekend (Time2) sample.

The Wilcoxon signed-rank test, T

• The Wilcoxon signed-rank test, sometimes called the Wilcoxon matched-pairs test, is also for two dependent samples where the variable of interest is interval. However, it is more powerful than the sign test because it takes more information into account. Specifically, the Wilcoxon test factors in the size as well as the sign of the paired differences. It assesses the null hypothesis that the medians of two samples do not differ, or that the median of one sample does not differ from a known value.
  • Computation of Wilcoxon's T
    1. List the paired scores in two columns corresponding to the two dependent samples.
    2. Subtract each score in the second column from its pair in the first column, creating a third column of differences.
    3. Delete pairs with zero differences (ties).
    4. Rank the differences from smallest (including negative) to largest absolute difference, from 1 to n, the sample size. Assign the average of ranks which would otherwise have been assigned in the cases of tied pairs.
    5. Let T be the smaller of the sum of ranks for positive differences, or the sum of ranks for negative differences.
    6. If computed T is equal to or less than the critical value of T in a table of the distribution of Wilcoxon's T, then the two samples can be said to differ significantly. In actual practice, computer programs like SPSS compute the probability of the standardized signed-ranks difference, Z, for the researcher, without need for recourse to a table of critical values.

  • SPSS output. For the SPSS-supplied sample data file, mutualfunds.sav, one may test if the average Standard and Poor's median stock price differs from the median mutual fund price for a fund centered on the subset of technology stocks. In this example, two Wilcoxon tests are run, one for 2000 and one for 2001.
    

    For each year, the Wilcoxon signed-ranks test is non-significant, meaning that the two variables (overall S & P median and technology mutual fund median) cannot be concluded to differ in median value. That is, the chance of getting a standardized signed-ranks difference (the "Z" row above) that high or higher in absolute value is greater than .05, so we fail to reject the null hypothesis that the medians differ. In the "Ranks" table we see that ties are zero. Had ties been numerous, tied cases would have been ignored, introducing the possibility of bias.

Assumptions

• Value distribution. The McNemar test is used when variables are assumed to be dichotomies which are mutually exclusive and exhaustive. For two dependent samples and multinomial categorical data, the marginal homogeneity test is used. The sign test and the Wilcoxon signed-ranks test assume a continuous value distribution, with the latter requiring interval or near-interval data.

• Data distribution. The tests in this section are all nonparametric (do not assume the normal distribution, or any other particular distribution other )

• Two samples. Data are from two samples, not necessarily from the same population. There is another set of tests for more than two dependent samples.

• Dependent samples. The two samples may be before-after samples or panel studies of the same subjects, or matched-pairs samples of similar subjects, or may be otherwise dependent (correlated). It is not required that they be dependent. For independent samples, for instance, the Wilcoxon test is directly related to the Mann-Whitney U test for two independent samples: U plus W sum to the constant m(m + 2n + 1)/2, where m is the number of observations in the smaller group and n is the number in the larger group. For this reason one would not use both tests for independent samples, and U is much more prevalent for this purpose.

• Adequate sample size. For the McNemar test, the number of cases on the a-d diagonal (see the illustration above) should be at least 10. If it is not, the binomial test is used with n = a + d and with r = a or d, whichever is smaller, as defined in the section on the binomial test.

Frequently Asked Questions

• Where is these tests (McNemar, marginal homogeneity, sign, Wilcoxon) found in SPSS?
  From the SPSS menu, select Statistics, Nonparametric Tests, 2 Related Samples. In the "Two Related Samples Tests" dialog box, check the desired test under "Test Type."

  
  

Bibliography

• Conover, W. J. (1980). Practical nonparametric statistics, 2nd ed. New York: John Wiley and Sons.
• Siegel, Sidney (1956). Nonparametric statistics for the behavioral sciences. NY: McGraw-Hill. A standard reference work.
• Siegel, Sidney & N. J. Castellan (1988). Nonparametric statistics for the behavioral sciences. New York: McGraw-Hill. A more recent version.