Overview Researchers must demonstrate instruments are reliable since without reliability, research results using the instrument are not replicable, and replicability is fundamental to the scientific method. Reliability is the correlation of an item, scale, or instrument with a hypothetical one which truly measures what it is supposed to. Since the true instrument is not available, reliability is estimated in one of four ways: Internal consistency: Estimation based on the correlation among the variables comprising the set (typically, Cronbach's alpha) Split-half reliability: Estimation based on the correlation of two equivalent forms of the scale (typically, the Spearman-Brown coefficient) Test-retest reliability: Estimation based on the correlation between two (or more) administrations of the same item, scale, or instrument for different times, locations, or populations, when the two administrations do not differ on other relevant variables (typically, the Spearman Brown coefficient) Inter-rater reliability: Estimation based on the correlation of scores between/among two or more raters who rate the same item, scale, or instrument (typically, intraclass correlation, of which there are six types discussed below). These four reliability estimation methods are not necessarily mutually exclusive, nor need they lead to the same results. All reliability coefficients are forms of correlation coefficients, but there are multiple types discussed below, representing different meanings of reliability and more than one might be used in single research setting.

Contents Key concepts and terms Internal consistency reliability Split-half reliability Odd-even reliability Test-retest reliability Inter-rater reliability Assumptions Frequently asked questions Bibliography




Key Concepts and Terms

• General

  • Scores are the subject's responses to items on an instrument (ex., a mail questionnaire). Observed scores may be broken down into two components: the true score (commonly labeled tau) plus the error score. The error score, in turn, can be broken down into systematic error (non-random error reflecting some systematic bias, as due, for instance, to the methodology used -- hence also called method error) and random error (due to random traits of the subjects -- hence also called trait error). The smaller the error component in relation to the true score component, the higher the reliability of an item, which is the ratio of the true score to the total (true + error) score.
    • Number of scale items. Note that the larger the number of items added together in a scale, the less random error matters as it will be self-cancelling (think of weighing a subject on 100 different scales and averaging rather than on using just one scale), and therefore some reliability coefficients (such as Cronbach's alpha) also compute higher reliability when the number of scale items is higher.

  • Models supported by SPSS under the Models button of the Reliability dialog are:
    1. Alpha (Cronbach). This models internal consistency based on average correlation among items.

    2. Split-half. This model is based on the correlation between the parts of a scale which is split into two forms.

    3. Guttman. This is an alternative split-half model which computes Guttman's lower bounds for true reliability, discussed below.

    4. Parallel. This method uses maximum likelihood to test if all items have equal variances and error variances. Cronbach's alpha is the maximum likelihood estimate of the reliability coefficient when the parallel model is assumed to be true (SPSS, 1988: 873). If the chi-square goodness of fit significance for the parallel model is <=.05, the researcher rejects the null hypothesis that the items have equal variances and error variances in the population.

    5. Strict parallel. This method also uses maximum likelihood to test for equal variances, equal error variances, and equal population means across items.

  • Triangulation is the attempt to increase reliability by reducing systematic (method) error, through a strategy in which the researcher employs multiple methods of measurement (ex., survey, observation, archival data). If the alternative methods do not share the same source of systematic error, examination of data from the alternative methods gives insight into how individual scores may be adjusted to come closer to reflecting true scores, thereby increasing reliability.

  • Calibration is the attempt to increase reliability by increasing homogeneity of ratings through feedback to the raters, when multiple raters are used. Raters meet in calibration meetings to discuss items on which they have disagreed, typically during pretesting of the instrument. The raters seek to reach consensus on rules for rating items (ex., defining the meaning of a "3" for an item dealing with job satisfaction). Calibration meetings should not involve discussion of expected outcomes of the study, as this would introduce bias and undermine validity.

  
  

• Internal consistency reliability

  • Cronbach's alpha is the most common form of internal consistency reliability coefficient. Alpha equals zero when the true score is not measured at all and there is only an error component. Alpha equals 1.0 when all items measure only the true score and there is no error component.
    • Interpretation: Cronbach's alpha can be interpreted as the percent of variance the observed scale would explain in the hypothetical true scale composed of all possible items in the universe. Alternatively, it can be interpreted as the correlation of the observed scale with all possible other scales measuring the same thing and using the same number of items.

    • Cut-off criteria. By convention, a lenient cut-off of .60 is common in exploratory research; alpha should be at least .70 or higher to retain an item in an "adequate" scale; and many researchers require a cut-off of .80 for a "good scale." Cronbach's alpha is discussed further in the section on standard measures and scales, along with other coefficients such as Cohen's kappa.

    • Formula. Standardized alpha is a ratio. The numerator is the number of items times the average of the covariances of all pairs of items. The denominator is the average variance of the items plus the quantity N-1 times the average covariance, where N is the number of items.

    • Other characteristics:
      1. Number of items. Note that Cronbach's alpha increases as the number of items in the scale increases, even controlling for the same level of average intercorrelation of items. This assumes, of course, that the added items are not bad items compared to the existing set. Increasing the number of items can be a way to push alpha to an acceptable level. This reflects the assumption that scales and instruments with a greater number of items are more reliable. It also means that comparison of alpha levels between scales with differing numbers of items is not appropriate.

      2. Alpha if deleted. SPSS will compute "Cronbach's Alpha if Item Deleted," which is the estimated value of alpha if the given item were removed from the model. The researcher may wish to drop items where the alpha if deleted is higher than the overall alpha as another way to improve the alpha level. Note, however, that when an item has high random error it is possible that it would be removed on this basis when, in fact, it does measure the same construct.

         In the example below, dropping any item will lower alpha. This is especially true of "RS Highest Degree", which in the table further below is shown correlated at the .872 level with "Highest Year of School Completed." Correlations over .80 may signal multicollinearity, which in turn might wrongly lead the researcher to drop "RS Highest Degree" from the scale, in turn leading to the conclusion that the remaining three items do not constitute a scale suitable even for exploratory purposes (since .539 is below the .60 cutoff level). However, high correlation of the constituent items of a scale is not considered multicollinearity because the scale score, not the separate items scores, will appear in the regression (or other) analysis.

         

      3. The item-total correlation is also part of SPSS output above. This is the Pearsonian correlation of the item with the total of scores on all other items. A low item-total correlation means the item is little correlated with the overall scale (ex., < .3 for large samples or not significant for small samples) and the researcher should consider dropping it. A negative correlation indicates the need to recode the item in the opposite direction. The reliability analysis should be re-run if an item is dropped or recoded. Note a scale with an acceptable Cronbach's alpha may still have one or more items with low item-total correlations.

      4. The squared multiple correlation, R² is the R² for an item when it is predicted from all other items in the scale. The larger the R², the more the item is contributing to internal consistency. The lower the R², the more the researcher should consider dropping it. Note the R² of some items may be low even on a scale which has an acceptable Cronbach's alpha overall.

      5. Negative alphas. Note also that a negative Cronbach's alpha indicates inconsistent coding (see assumptions) or a mixture of items measuring different dimensions, leading to negative inter-item correlations. This is discussed in the FAQ section below.

      6. The Kuder-Richardson (KR20) coefficient is the same as Cronbach's alpha when items are dichotomous.

    • In SPSS, Cronbach's alpha is found under Analyze, Scale, Reliability Analysis. Then in the Statistics button, check Scale to get alpha. You can also check Scale if deleted, in which case alpha will be computed both for all variables entered, and also for all remaining variables if any one is dropped (the alpha if deleted is listed in a table, one for each variable). That is, the 'scale if deleted' option lets the researcher assess the reliability of each item.
      

  • Standardized item alpha, also shown in SPSS output above, is the average inter-item correlation when item variances are equal. It is also called the Spearman-Brown stepped-up reliability coefficient or simply the "Spearman-Brown Coefficient," but these terms should not be confused with the Spearman-Brown split-half reliability coefficient discussed below. The difference between Cronbach's alpha and standardized item alpha is a measure of the dissimilarity of variances among items in the set. In a second use, standardized item alpha can be used to estimate the change in reliability as the number of items in an instrument or scale varies. In SPSS, the Spearman-Brown stepped-up reliability coefficient is labeled "Cronbach's alpha based on standardized items" and is part of the default output in the "Reliability Statistics" table, next to Cronbach's alpha..
    r_SB2 = (N* r_ave)/[1 + (N-1)* r_ave)]
    where
    
    r_SB2 = the Spearman-Brown stepped-up reliability = standardized item alpha
    r_ave = the average of inter-item correlations
    N = total number of items
    

  • Ordinal reliability alpha. Zumbo, Gadermann, & Zeisser (2007) use a polychoric correlation matrix input to calculate alpha parallel to Cronbach. Their simulation studies lead them to conclude that ordinal reliability alpha provides "consistently suitable estimates of the theoretical reliability, regardless of the magnitude of the theoretical reliability, the number of scale points, and the skewness of the scale point distributions. In contrast, coefficient alpha is in general a negatively biased estimate of reliability" for ordinal data (p. 21). Ordinal reliability alpha will normally be higher than the corresponding Cronbach's alpha.

  • Raykov's reliability rho (ρ), also called reliability rho or composite reliability, tests if it may be assumed that a single common factor underlies a set of variables. Raykov (1998) has demonstrated that Cronbach's alpha may over- or under-estimate scale reliability. Underestimation is common. For this reason, rho is now preferred and may lead to higher estimates of true reliability. Raykov's reliability rho is not to be confused with Spearman's median rho, an ordinal alternative to Cronbach's alpha, discussed below. The acceptable cutoff for rho would be the same as the researcher sets for Cronbach's alpha since both attempt to measure true reliability. Raykov's reliability rho is ouput by EQS. See Raykov (1997), which lists EQS and LISREL code for computing composite reliability.Graham (2006) discusses Amos computation of reliability rho.

  • Armor's reliability theta is a similar measure developed by Armor (1974). Theta =θ = [p/(p-1)]*[1-(1/λ₁)], where p = the number of items in the scale and where λ₁ denotes the first and therefore largest eigenvalue from the principal component analysis of the correlation of items comprising the scale. See Zumbo, Gadermann, & Zeisser, 2007: 22. Reliability theta is interpreted similar to other reliability coefficients. While not directly computed by SPSS or SAS, it is easily calculated from principal components factor results using the formula above.
    • Ordinal reliability theta. Zumbo, Gadermann, & Zeisser (2007) use a polychoric correlation matrix input to principal components analysis to calculate an ordinal version of reliability theta, using simulation studies to demonstrate ordinal reliability theta "consistently suitable estimates of the theoretical reliability, regardless of the magnitude of the theoretical reliability, the number of scale points, and the skewness of the scale point distributions." (p. 21). Ordinal reliability theta will normally be higher than the corresponding Cronbach's alpha.

  • Spearman's reliability rho.. Spearman's rho is a form of rank order calculation. It is calculated with the same formula as for Pearson's r correlation, but using rank rather than interval data. The median rho between all pairs of items in a scale is a classic measure of reliability, in the sense of internal consistency, and as such is an rdinal alternative to Cronbach's alpha. Rho > .60 is considered the minimum for adequate scale reliability. This is not to be confused with Raykov's reliability rho.

  
  

• Split-half reliability

  • Split-half reliability, which measures equivalence, is also called parallel forms reliability or internal consistency reliability. It is administering two equivalent batteries of items measuring the same thing in the same instrument to the same people. If split halves is requested in SPSS, four coefficients will be generated: Cronbach's alpha for each form, the Spearman-Brown coefficient, the Guttman split-half coeffcient, and the Pearsonian correlation between the two forms (aka, "half-test reliability"). (Note: Some authors label split-half reliability as a subtype of internal consistency reliability.)

    In SPSS, select Analyze, Scale, Reliability Analysis; list your variables; click Statistics; select Item Scale, Scale if Item Deleted; select Split-Half from the Model drop-down list. OK. SPSS will take the first half of the items as the first split form, and the second half as listed in the dialog box as the second split form. If there are an odd number of items, the first form will be one item longer than the second form. You can also use the Paste button to call up the Syntax window and alter the /MODEL=SPLIT parameter to be /MODEL=SPLIT n, where n is the number of items in the second form.

    

    • Spearman-Brown split-half reliability coefficient, also called the Spearman-Brown prophecy coefficient and not to be confused with the Spearman-Brown stepped-up reliability coefficient (standardized item alpha) above, is a form of split-halves reliability measure. The Spearman-Brown prophecy coefficient is used to estimate full test reliability based on split-half reliability measures. A common rule of thumb is .80 or high for adequate reliability and .90 or higher for good reliability. However, for exploratory research, a cutoff as low as .60 is not uncommon.
      

      In the example above, SPSS has divided the four-item education scale into two subscales. As shown in the table footnotes, the first subscale is Highest Year of School completed plus Father's Highest Degree. The second subscale is Mother's highest Degree and Respondent's Highest Degree. Comparing scores on these two subscales yields a Spearman-Brown reliability coefficient of .915. On a split-half basis, the researcher concludes the 4-item education scale is reliable.

      The Pearson correlation of split forms estimates the half-test reliability of an instrument or scale. The Spearman-Brown "prophecy formula" predicts what the full-test reliability would be, based on half-test correlations. This coefficient will be higher than the half-test reliability coefficient. This coefficient is usually equal to and easily hand-calculated as twice the half-test correlation divided by the quantity 1 plus the half-test reliability. In SPSS, two Spearman-Brown split-half reliability coefficients will appear in the "Reliability Statistics" portion of the output when split-half is selected under the Model button: (1) "Equal length" gives the estimate of the reliability if both halves had equal numbers of items, and (2) "Unequal length" gives the reliability estimate assuming unequal numbers.

      r_SB1 = (k* r_ij)/[1 + (k-1)* r_ij)]
      where
      
      r_SB1 = the Spearman-Brown split-half reliability
      r_ij = the Pearson correlation between forms i and j
      k = total sample size divided by sample size per form (k is usually 2)
      
      
      

      As with other split-halves measures, the Spearman-Brown reliability coefficient is highly influenced by alternative methods of sorting items into the two forms, which is preferably done randomly. Random assignment of items to the two forms should assure equality of variances between the forms, but this is not guaranteed and should be checked by the researcher.

    • Guttman split-half reliability coefficient is an adaptation of the Spearman-Brown coefficient, but one which does not require equal variances between the two split forms. Select Scale in the Descriptives area of the Statistics button to generate the Scale Statistics table, footnotes to which will list which items are in each of the two subscales created automatically by the SPSS split half algorithm, similar to that in the Reliability Statistics table illustrated above for Spearman-Brown split half analysis.
      • Guttman's lower bounds (lambda 1-6) are a set of six coefficients, L1 to L6, generated when in SPSS one selects "Guttman" under the Model button. Below is output for the four-item education scale discussed previously.
        
        1. L1: An intermediate coefficient used in computing the other lambdas.

        2. L2: More complex than Cronbach's alpha and preferred by some researchers, though less common.

        3. L3: Equivalent to Cronbach's alpha.

        4. L4: Guttman split-half reliability.

        5. L5: Recommended when a single item highly covaries with other items, which themselves lack high covariances with each other.

        6. L6: Recommended when inter-item correlations are low in relation to squared multiple correlations

      Guttman recommends experimenting to find the split of items which maximizes Guttman split-half reliability (L4), then using the highest of the lower bound lambdas as the reliability estimate for the set of items. The best split will be that in which each half contains highly intercorrelated items.

  
  

• Odd-Even Reliability

  • Odd-even reliability is a variant of split half reliability where one creates the two subscales from the odd-numbered and even-numbered items respectively.

  • SPSS supports odd-even reliablity in syntax but not in the menu system. For the four-item education scale, the split-half syntax code (accessed by clicking the Paste button) was:

    DATASET ACTIVATE DataSet1.
    RELIABILITY
      /VARIABLES=educ padeg madeg degree
      /SCALE('ALL VARIABLES') ALL
      /MODEL=SPLIT
      /STATISTICS=SCALE ANOVA.
    

    By default, the SPSS split half algorithm makes the first two items (educ, padeg) into subscale 1 and makes the last two items (madeg, degree) into subscale 2. To convert to odd-even split half format, the syntax of the /VARIABLES statement must be changed as illustrated below:

      /VARIABLES=educ madeg padeg degree
    

    Note that the order of the variables has been rearranged to list the 1st and 3rd variables first, followed by the 2nd and 4th. This will make subscale 1 be the odd items and subscale 2 be the even items.

    As a result, the Spearman Brown split half reliability coefficient will differ, but the ANOVA table for the model as a whole will not change.

    

  
  

• Test-retest reliability

  • Test-retest reliability, which measures stability over time, is administering the same test to the same subjects at two points in time. The appropriate length of the interval depends on the stability of the variables which causally determine that which is measured. A year might be too long for an opinion item but appropriate for a physiological measure. A typical interval is several weeks. Statistically, test-retest reliability is treated as a variant of split-half reliability and also uses the Spearman-Brown coefficient.

    Test-retest methods are disparaged by many researchers as a way of gauging reliability. Among the problems are that short intervals between administrations of the instrument will tend to yield estimates of reliability which are too high. There may be invalidity due to a learning/practice effect (subjects learn from the first administration and adjust their answers on the second). There may be invalidity due to a maturation effect when the interval between administrations is long (the subjects change over time). The bother of having to take a second administration may cause some subjects to drop out of the pool, leading to nonresponse biases. Note, however, that test-retest designs are still widely used and published and there is support for this. McKelvie (1992), for instance, reports that reliability estimates under test-retest designs are not inflated due to memory effects. Researchers using test-retest reliability must address the special validity concerns, but may decide to go ahead if warranted.

  
  

• Inter-rater reliability

  • Inter-rater reliability, which measures homogeneity, is administering the same form to the same people by two or more raters/interviewers so as to establish the extent of consensus on use of the instrument by those who administer it. In the data setup, judges are the columns and judgees are the rows. For categorical data, consensus is measured as number of agreements divided by total number of observations. For continuous data, consensus is measured by intraclass correlation, discussed below. Note raters should be as blind as possible to expected outcomes of the study and should be randomly assigned.

  • Cohen's Kappa for inter-rater reliability can be used to assess inter-rater reliability if there are just two raters. Cohen developed a multi-rater version of Kappa, but it is not implemented in SPSS.
    Let there be two raters who each independently rate the same n objects, each of which might fall into one of k categories. For instance, two raters could rate 100 letters to the editor as "liberal," "conservative," or "bipartisan." The raters' choices would be organized into a square table, with the k available choices for Rater #1 being the columns and the k choices for Rater #2 being the rows. A cell count of 10 in the Rater #1-liberal column and the Rater #2-bipartisan row would mean that 10 of the 100 cases were rated liberal by Rater #1 and bipartisan by Rater #2, and so on.

    Counts in diagonal cells will reflect inter-rater agreement and cells off the diagonal will represent disagreements. Kappa is a function of the ratio of agreements to disagreements in relation to expected frequencies. In SPSS it is not available in the Reliability module. Rather one must obtain it from the Crosstabs procedure (Kappa is a choice under the Statistics button in Crosstabs; it is not a default option). In SAS, weighted and unweighted kappa is computed by the FREQ procedure.

    Interpretation. By convention, a Kappa > .70 is considered acceptable inter-rater reliability, but this depends highly on the researcher's purpose. Another rule of thumb is that K = 0.40 to 0.59 is moderate inter-rater reliability, 0.60 to 0.79 substantial, and 0.80 outstanding (Landis & Koch, 1977). For inter-rater reliability of a set of items, such as a scale, one would report mean Kappa.

    Manual computation: let a = the sum of counts on the diagonal, reflecting agreements. Let e = the sum of expected counts on the diagonal, where expected is calculated as [(row total * column total)/n], summed for each cell on the diagonal. Let n = the total number of ratings (observations). Kappa then equals the ratio of the surplus of agreements over expected agreements, divided by the number of expected disagreements. This is equivalent to K = (a - e)/(n - e). Fleiss and Cohen (1973) have shown ICC, discussed below, is mathematically equivalent to weighted Kappa.

    Weighted Kappa: For ordinal rankings or better, one can weight each cell in the agreement/disagreement table by a weight between 0 and 1, where 1 corresponds to the row and column categories being the same and 0 corresponds to the categories being maximally dissimilar.

  • Intraclass correlation (ICC) is used to measure inter-rater reliability for two or more raters when data may be considered interval level. It may also be used to assess test-retest reliability. ICC may be conceptualized as the ratio of between-groups variance to total variance, as elaborated below. A classic citation for intraclass correlation is Shrout and Fleiss (1979), though ICC is based on work going back before WWI.
    Example. The example below uses the data file tv-survey.sav, supplied with SPSS as a sample. Some 906 respondents were asked if they would watch a particular show for....
    1. Any reason
    2. No other popular shows on at that time
    3. Critics still give the show good reviews
    4. Other people still watch the show
    5. The original screenwriters stay
    6. The original directors stay
    7. The original cast stays
    where answers were coded 0=No or 1=Yes.

    Sample size: ICC vs. Pearson r: When there are just two ratings, ICC is preferred over Pearson's r only when sample size is small (< 15). Since Pearson's r makes no assumptions about rater means, a t-test of the significance of r reveals if inter-rater means differ. For small samples (< 15), Pearson's r overestimates test-retest correlation and in this situation, intraclass correlation is used instead of Pearson's r. Walter, Eliasziw, & Donner (1998) set optimal sample size for ICC based on desired power level, magnitude of the predicted ICC, and the lower confidence limit, concluding that if the researcher used the customary .95 confidence level and the .80 power level, and had two ratings per subject, then the needed sample size (needed to prove the estimated ICC was different from 0) would range from 5 when the estimated ICC was .9 to 616 when it was only .1; for three ratings, the corresponding range was 3 to 225; for four ratings, 3 to 123; for five ratings, 3 to 81; for 10 ratings, 2 to 26; for 20 ratings, 2 to 11 (pp. 106-107). Bonnett (2002: 1334) investigated the sample size issue for ICC, concluding that optimum sample size is a function of the size of the intraclass correlation coefficient and the number of ratings per subject, as well as the desired significance level (alpha) and desired width (w) of the confidence interval. For alpha = .95 and w=.2, Bonnett concluded that the optimal sample size for two ratings varied from 15 for ICC=.9 to 378 for ICC = .1; for three ratings, it varied from 13 to 159; five ratings, 10 to 64; and 10 ratings, 8 to 29. That is, the fewer ratings and the smaller the ICC level, the larger the needed sample size. For this example, with 906 people rating 7 items, described above, sample size is more than adequate.

    Data setup: In using intraclass correlation for inter-rater reliability, one constructs a table in which optionally column 1 is the target id (1, 2, ..., n) and subsequent columns are the raters (A, B, C, ...). It may be necessary to transpose the data (Data, Transpose in the SPSS menus) to make the raters be the columns, as was done below for the example data on 906 respondents rating TV shows on 7 items.

    

    The row variable is the target of the ratings. The target might be attributes (in this example the target is TV show attributes), or it might be persons who are rated (Subject1, Subject2, etc.) or neighborhoods which are rated (E, W, N, S), for instance. The cell entries after the first id column are the raters' ratings of the target on some interval variable or interval-like variable, such as some Likert scale, or, in this example, a binary 0/1 scale. The purpose of ICC is to assess the inter-rater (column) effect in relation to the grouping (row) effect, using two-way ANOVA.

    Interpretation: ICC is interpreted similar to Kappa, discussed above. ICC will approach 1.0 when there is no variance within targets (for any target, all raters give the same ratings), indicating total variation in measurements is due solely to the target (ex., TV attribute) variable. That is, ICC will be high when any given row tends to have the same score across the columns (which are the raters). For instance, one may find all raters rate an item the same way for a given target, indicating total variation in the measure of a variable depends solely on the values of the variable being measured -- that is, there is perfect inter-rater reliability. Put another way, ICC may be thought of as the ratio of variance explained by the independent variable divided by total variance, where total variance is the explained variance plus variance due to the raters plus residual variance. ICC is 1.0 only when there is no variance due to the raters and no residual variance to explain.

    In SPSS, select Analyze, Scale/Reliability Analysis; select your variables; click Statistics; in the Descriptives group, select Item and select Intraclass correlation coefficient.; select a model from the Model drop-down list (ex., One-way random); select a type from the Type drop-down list (ex., consistency). Continue. OK. Models and Types are discussed below.

    

    Single versus average measures: Each model has two versions of the intraclass correlation coefficient:

    1. Single measure reliability: individual ratings constitute the unit of analysis. That is, single measure reliability gives the reliability for a single judge's rating. In the example above, single measure ICC is .143, indicating a relatively low level of inter-rater consistency on the seven TV attribute items taken one at a time. Single measure reliability is the one of usual interest, used to assess if the ratings of one judge are apt to be the same as for another judge. For this example, given the low single measure ICC, we must assume different people rate TV attributes differently.

    2. Average measure reliability: the mean of all ratings is the unit of analysis. That is, average measure reliability gives the reliability of the mean of the ratings of all raters. Use this if the research design involves averaging multiple ratings for each item across all raters, perhaps because the researcher judges that using an individual rating would involve too much uncertainty. In the example above, average measure ICC is .993, indicating a high level of inter-rater consistency on the average of all 7 ratings. That average measure ICC is high means that when TV attributes are averaged across all raters, the mean ratings are very stable. It does not mean that raters agree in their ratings (that is single measure ICC).

       Average measure reliability is close to Cronbach's alpha. Average measure reliability for either two-way random effects or two-way mixed models will be the same as Cronbach's alpha. In this example, for the one-way random model, the ICC and Cronbach's alpha differ, but not greatly.

       Average measure reliability requires a reasonable number of judges to form a stable average. The number of judges required is estimated beforehand as nj = ICC*(1 - rl)/rl( 1 - ICC*), where nj is the number of judges needed, rl is the lower bound from the (1-a)*100% confidence interval around the ICC, discovered in a pilot study; and ICC* is the minimum level of ICC acceptable to the researcher (ex., .80).

    Models: ICC varies depending on whether the judges are all judges of interest or are conceived as a random sample of possible judges, and whether all targets are rated or only a random sample, and whether reliability is to be measured based on individual ratings or mean ratings of all judges. These considerations give rise to six forms of intraclass correlation, described in the classic article by Shrout and Fleiss (1979). In SPSS, these types are selected under the Model button of the Reliability dialog and under the Type drop-down list (3 models times 2 types = the six forms of ICC). .

    1. One-way random effects model. Judges/raters are conceived as being a random selection of possible raters/judges, who rate all targets of interest. This assumption is used in the example output above. That is, in this model judges are treated as a random sample and the focus of interest is a one-way anova testing if there is a subject/target effect. This model applies even when the researcher cannot associate a particular subject with a particular rater because information is lacking about which judge assigned which score to a subject. This would happen if the columns were first rating of a subject, second rating, third rating, etc., but a given rating (ex., the first rating) for one subject might be by a different judge than the first rating for another subject, etc. This in turn means there is no way to separate out a judge/rater effect. There would also be no way to separate out a judge/rater effect if each judge rates only one subject, even if it is known which judge assigned which score. In either of these situations the researcher uses a one-way random effects model. This model conceptualizes that there is a target/subject factor, with each observed actual subject representing a level of that target/subject factor. The rater/judge factor cannot be measured and is absorbed into error variance. The ICC is interpreted as the proportion of target/subject variance associated with differences among the scores of the subjects.

    2. Two-way random effects model. Judges are conceived as being a random selection from among all possible judges, and targets/subjects are conceived as being a random factor too. Raters rate all n subjects/targets chosen at random from a pool of targets/subjects and it is known how each judge rated each subject. The ICC is interpreted as the proportion of Subject plus Rater variance that is associated with differences among the scores of the subjects. The ICC is interpreted as being generalizable to all possible judges.

    3. Two-way mixed model. All judges of interest rate all targets, which are a random sample. This is a mixed model because the judges are seen as a fixed effect (not as a random sample of all possible raters/judges) and the targets are a random effect. The ICC coefficients will be identical to the two-way random effects model, but the ICC is interpretated as not being generalizable beyond the given judges.

    Types: Under the Model button of the SPSS Reliability dialog, the Type drop-down list allows the researcher to specify one of two types of ICC computation:

    1. Absolute agreement: Measures if raters assign the same absolute score. Absolute agreement is often used when systematic variability due to raters is relevant.

    2. Consistency: Measures if raters' scores are highly correlated even if they are not identical in absolute terms That is, raters are consistent as long as their relative ratings are similar. Consistency agreement is often used when systematic variability due to raters is irrelevant.

    Use in other contexts. ICC is sometimes used outside the context of inter-rater reliability. In general, ICC is a coefficient which approaches 1.0 as the between-groups effect (the row effect) is very large relative to the within-groups effect (the column effect), whatever the rows and columns represent. In this way ICC is a measure of homogeneity: it approaches 1.0 when any given row tends to have the same values for all columns. For instance, let columns be survey respondents and let rows be Census block numbers, and let the attribute measured be white=0/nonwhite=1. If blocks are homogenous by race, any given row will tend to have mostly 0's or mostly 1's, and ICC will be high and positive. As a rule of thumb, when the row variable is some grouping or clustering variable, such as Census areas, ICC will more and more approach 1.0 as the size of the clusters decreases and becomes more compact (ex., as one goes from metropolitan statistical areas to Census tracts to Census blocks). ICC is 0 when within-groups variance equals between-groups variance, indicative of the grouping variable having no effect. Though less common, note that ICC can become negative when the within-groups variance exceeds the between-groups variance.




Assumptions

• Additivity. Each item should be linearly related to the total score. Tukey's test of non-additivity, a choice under the Statistics button of the Reliability dialog in SPSS, tests the null hypothesis that there is no multiplicative interaction between the cases and the items. If this test is significant (&le .05) then there is multiplicative interaction. The Tukey significance is found in the "Nonadditivity" row of the "ANOVA with Tukey's Test for Nonadditivity" table in SPSS output.

  If Tukey's test shows multiplicative interaction, any model computing scores for cases based on the scale must include the case main effect, the item main effect, and the case-by-item interaction effect. In a footnote to the Tukey test output, SPSS prints an estimates of the power to which items in a set would need to be raised in order to be additive. (Warning: while transforms may eliminate non-additivity, raising item scores to too high a power will generate large values for all subjects, obscuring differences among subjects).

  In SPSS, select Analyze, Scale, Reliability Analysis; click Statistics; check Tukey's test of additivity. The output below is General Social Survey data for the four education items in illustrations above. Since Tukey's test is significant, multiplicative interaction is indicated for these data.

  

• Independence. Observations for one subject/case should be independent of observations for any other subject/case in any administration of the instrument. However, the fact that test-retest designs involve correlated data between administrations does not pose a statistical problem in assessing reliability and does not in itself violate assumptions of reliability analysis.

• Uncorrelated error. Errors should be uncorrelated .

• Consistent coding. High values must have the same meaning across items.

• Random assignment of subjects. In split-half tests, random assignment of items to forms is assumed. Typically, odd-numbered items become one form and even-numbered items become the second form. Sequential assignment may involve a subject fatigue factor with regard to the second form.

• Equivalency of forms. In split-half tests (recall split half analysis is selected in the "Model" pull-down of the initial SPSS Reliability dialog), the two forms should be equivalent. A test of this is to see if the mean response is the same in the two groups. In split half models, Hotelling’s T² is a multivariate test for equality of means between groups. A significant T² means that the null hypothesis that means were equal can be rejected by the researcher. Apart from split-half test analyses, T² tests if the means of a set of items differ and again, a significant T² means one rejects the null assumption of equal means, as in the example below for data on the four-item education scale in previous examples. This test assumes multivariate normality of items. Hotelling’s T² is a choice under the Statistics button of the Reliability dialog in SPSS.
  

• Equal variances. In split-half tests, the Spearman-Brown split-half reliability coefficient assumes the split halves have equal variances. The chi-square test of parallel models tests the null hypothesis that the variances are equal. If the chi-square significance is <= .05, then the researcher concludes the models are not parallel and that the variances differ significantly. In SPSS, select "Parallel" under the model button.

• Similar difficulty of items. In internal consistency analysis using Cronbach's alpha, it assumes the scale items all measure the same dimension equally (ex., and assortment of math problems of equal difficulty). However, if the scale is of the Guttman scale type where higher items (ex., solving division problems) imply responses to lower items (ex., solving addition problems), but not vice-versa, internal consistency in the Cronbach's alpha sense is not expected and Cronbach's alpha gives an inappropriate estimate of reliability.

• Same assumptions as for correlation.




Frequently Asked Questions

• How is reliability related to validity?
  A measure may be reliable but not valid, but it cannot be valid without being reliable. That is, reliability is a necessary but not sufficient condition for validity.

• How is reliability related to attenuation in correlation?
  Reliability is a form of correlation. Correlation coefficients can be attenuated (misleadingly low) for a variety of reasons, including truncation of the range of variables (as by dichotomization of continuous data; reducing a 7-point scale to a 3-point scale). Measurement error also attenuates correlation. Reliability may be thought of as the correlation of a variable with itself. Attenuation-corrected correlation ("disattenuated correlation") is higher than the raw correlation on the assumption that the lower the reliability, the greater the measurement error, and the higher the "true" correlation is in relation to the measured correlation.

  The Spearman correction for attenuation of a correlation: let r_xy* be corrected r for the correlation of x and y; let r_xy be the uncorrected correlation; then r_xy* is a function of the reliabilities of the two variables, r_xx and r_yy:
  

  r_xy* = r_xy / [SQRT{r_xxr_yy}]

  This formula will result in an estimated true correlation ( r_xy*) which is higher than the observed correlation (r_xy), and all the more so the lower the reliabilities. Corrected r may be greater than 1.0, in which case it is customarily rounded down to 1.0.

  Note that use of attenuation-corrected correlation is the subject of controversy (see, for ex., Winne & Belfry, 1982). Moreover, because corrected r will no longer have the same sampling distribution as r, a conservative approach is to take the upper and lower confidence limits of r and compute corrected r for both, giving a range of attenuation-corrected values for r. However, Muchinsky (1996) has noted that attenuation-corrected reliabilities, being not directly comparable with uncorrected correlation, are therefore not appropriate for use with inferential statistics in hypothesis testing and this would include taking confidence limits. Still, Muchinsky and others acknowledge that the difference between a correlation and attenuation-corrected correlation may be useful, at least for exploratory purposes, in assessing whether a low correlation is low because of unreliability of the measures or because the measures are actually uncorrelated.

  
  

• How should a negative reliability coefficient be interpreted?
  It should be interpreted as a data entry error, a data measurement problem, as a problem based on small sample size, or as indicative of multidimensionality. As negative reliability is rare, the researcher should first check to see if there are coding or data entry errors.

  One situation in which negative reliability might occur is when the scale items represent more than one dimension of meaning, and these dimensions are negatively correlated, and one split half test is more representative of one dimension while the other split half is more representative of another dimension. As Krus & Helmstadter point out, factor analyzing the entire set of items first would reveal if the set of items is plausibly conceptualized as unidimensional.

  A second scenario for negative reliability is discussed by Magnusson (1966: 67), who notes that when true reliability approaches zero and sample size is small, random disturbance in the data may yield a small negative reliability coefficient.

  In the case of Cronbach's alpha, Nichols (1999) notes that values less than 0 or greater than 1.0 may occur, especially when the number of cases and/or items is small. Negative alpha indicates negative average covariance among items, and when sample size is small, misleading samples and/or measurement error may generate a negative rather than positive average covariance. The more the items measure different rather than the same dimension, the greater the possibility of negative average covariance among items and hence negative alpha.

  
  

• What is Cochran's Q test of equality of proportions for dichotomous items?
  Cochran's Q is used to test whether a set of dichotomous items split similarly. This is the same as testing whether the items have the same mean. If they test the same, then items within the set might be substituted for one another. In the ANOVA output table for a set of dichotomous items, the "Between Items" row, "Sig" for Cochran's Q column, if Sig (Q) <= .05, then the researcher rejects the null hypothesis that all items display an equal split or have the same mean.

  In SPSS, select Analyze, Scale/Reliability; select your items; click Statistics; in the Descriptives area, select Item, Scale, Scale if Deleted; in Summarize, select summary statistics (Means, Variances, Covariances, Correlations); and in the ANOVA table group, select Cochran chi-square. Continue. OK.

  Cochran's Q is discussed further in the section on significance tests for more than two dependent samples.

• What is the derivation of intraclass correlation coefficients?

  Derivation of the ICC formula, following Ebel (1951: 409-411): Let A be the true variance in subjects' ratings due to the normal expectation that different subjects will have true different scores on the rating variable. Let B be the error variance in subjects' ratings attributable to inter-rater unreliability. The intent of ICC is to form the ratio, ICC = A/(A + B). That is, intraclass correlation is to be true inter-subject variance as a percent of total variance, where total variance is true variance plus variance attributable to inter-rater error in classification. B is simply the mean-square estimate of within-subjects variance (variance in the ratings for a given subject by a group of raters), computed in ANOVA. The mean-square estimate of between-subjects variance equals k times A (the true component) plus B (the inter-rater error component), since each mean contains a true component and an error component.

  Given B = ms_within, and given ms_between = kA + B, substituting these equalities into the intended equation (ICC = A/[A+B]), the equation for ICC reduces to the formula for the most-used version of intraclass correlation (Haggard, 1958: 60):

  ICC = r_I = (ms_between - ms_within)/(ms_between + [k - 1]ms_within)
  where
  • ms_between is the mean-square estimate of between-subjects variance, reflecting the normal expectation that different subjects will have true different scores on the rating variable

  • ms_within is the mean-square estimate of within-subjects variance, or error attributed to inter-rater unreliability in rating the same person or target (row).

  • k is the number of raters/ratings per target (person, neighborhood, etc.) = number of columns. If the number of raters differs per target, an average k is used based on the harmonic mean: k' = (1/[n-1])(SUMk-[SUMk²]/SUMk).

• What are Method 1 and Method 2 in the SPSS RELIABILITY module?
  These are two methods of computing the reliability coefficient (Cronbach's alpha) for a set of items thought to comprise a scale. Method 1 allows constant terms to remain in the scale, while Method 2 deletes constant terms. Method 2 also produces a standardized item alpha, as if data had been input in standardized form. Method 2 can be forced when using the syntax window by adding the clause METHOD=COV. Method 2 is the default in SPSS.




Bibliography

• Armor, D. J. (1974). Theta reliability and factor scaling. Pp. 17-50 in H. Costner, ed., Sociological methodology. San Francisco: Jossey-Bass.
• Bonett, Douglas G. (2002). Sample size requirements for estimating intraclass correlations with desired precision. Statistics in Medicine 21: 1331-1335.
• Ebel, Robert L. (1951). Estimation of the reliability of ratings. Psychometrika 16: 407-424.
• Fleiss, J. L., Cohen, J. (1973). The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational and Psychological Measurement, 33: 613-619.
• Graham, James M. (2006). Congeneric and (essentially) tau-equivalent.estimates of score reliability: What they are and how to use them: Educational and Psychological Measurement 66; 930-944.
• Haggard, E. A. (1958). Intraclass correlation and the analysis of variance. NY: Dryden.
• Krus, D.J., & Helmstadter, G.C. (1993) The problem of negative reliabilities. Educational and Psychological Measurement, 53, 643-650.
• Landis, J. R., Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics 33:159-174. This article sets cut-offs for Cohen's Kappa.
• Litwin, Mark S.(2002). How to assess and interpret survey psychometrics. The Survey Kit series, Vol. 8. Thousand Oaks, CA: Sage Publications). Covers test-retest, alternate-form, internal consistency, interobserver, and intraobserver reliability.
• Magnusson, D. (1966). Test theory. Reading, MA: Addison-Wesley.
• McGraw, K.O. and S.P. Wong (1996). "Forming inferences about some intraclass correlation coefficients," Psychological Methods 1(1): 30-46.
• McKelvie, S. J. (1992). Does memory contaminate test-retest reliability? Journal of Gen Psychology 119(1):59-72. This article reports that reliability estimates under test-retest designs are not inflated due to memory effects.
• McNemar, Q. (1969). Psychological Statistics. Fourth edition. New York: Wiley Covers F tests for intraclass correlation (p. 322).
• Muchinsky P.M. (1996) The correction for attenuation. Educational & Psychological Measurement 56(1), 63-75.
• Nichols, David P. (1999). My coefficient a is negative! SPSS Keywords, Number 68. Retrieved 1/28/2009 from http://www.ats.ucla.edu/stat/Spss/library/negalpha.htm.
• Nunnelly, J. C. (1970) Psychometric Theory. Second ed., 1978. New York: McGraw Hill. Cited as a reference in support of the .70 cut-off for Cronbach's alpha. Classic on reliability in psychological and educational testing.
• Raykov, Tenko (1997). Estimation of composite reliability for congeneric measures. Applied Psychological Measurement, 21, 173-184
• Raykov, Tenko (1998). Coefficient alpha and composite reliability with interrelated nonhomogeneous items Applied Psychological Measurement, 22(4), 375-385.
• Shrout, P.E., and J. L. Fleiss (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin (86): 420-428. Classic article on intraclass correlation.
• SPSS (1988). SPSS-X User's Guide, Third ed.. Chicago, IL: SPSS Inc.
• Walter, S. D.; Eliasziw, M. ; and Donner, A. (1998). Sample size and optimal designs for reliability studies. Statistics in Medicine 17: 101-110.
• Winne, Philip H. & Belfry, M. Joan (1982). Interpretive problems when correcting for attenuation. Journal of Educational Measurement, 19(2), 125-134.
• Zumbo, B. D.; Gadermann, A. M.; & Zeisser, C.. (2007). Ordinal versions of coefficients alpha and theta for likert rating scales. Journal of Modern Applied Statistical Methods, 6, 21-29.

Copyright 1998, 2008, 2009, 2010 by G. David Garson.
Last updated 1/30/2010.