Overview The progress of science, including social science, depends heavily on building on the past. This is particularly so when in comes to the measurement of constructs. Coming to a disciplinary consensus on how to operationalize a construct (ex., "trust in goverment") is itself an exercise which illuminates surrounding issues. Conversely, if each social scientist operationalizes the same construct differently, confusion and "talking past one another" abounds and the state of the art progresses more slowly. While often there is no one perfect way to operationalize a construct, by using the same operationalization, social scientists achieve a far greater level of comparability across varied research than would otherwise be the case. Searching out existing standard scales and measures for constructs is therefore often an important early step in the research process, even if in the end the researcher opts for an original approach. See also multidimensional scaling, discussed separately.

Contents Key concepts and terms Methods of combining items Likert scales Guttman scales Proximity scales Mokken scales Measures of internal consistency Measures of agreement Assumptions Frequently asked questions Bibliography




Key Concepts and Terms

• Constructs vs. categories.Constructs are latent variables. They are latent in the sense that they cannot be measured directly, but only though measureable indicator variables. Scales assume that what is being measured is a single dimension, usually an construct like "efficiency" or "happiness." Constructs may be contrasted with indexes of measured variables. The category "environmental factors" might include many measures combined in an index, but the constituent items might represent many dimensions and these might not inter-correlate. Indexes and scales are discussed below. See also other statistical methods of measuring constructs discussed separately, such as factor analysis, structural equation modeling, partial least squares regression, canonical correlation, latent class analysis.

• Benchmarks are target values on quantitative measures composed of one or more items which have been accepted formally by a professional association, or informally by a profession. Observed values are compared with benchmark target values to assess performance. Benchmarks may be constituent elements in scales or indexes.

• Data-gathering techniques
  • Stand-alone data gathering methods are those where each survey item or other measure may be given a value without reference to other items or bases of comparison.
    • Likert survey items. Probably the most common data-gathering method for scale construction is the survey containing Likert items proposed to be indicators to be combined in some way in an index, score, or scale. Likert scales are discussed below.

    • Ranking scales. For each of a set of items, the respondent gives a value from 1 to some number, often 10. The respondent is given meanings for the anchor points (ex., 1 = very strongly dislike, 10 = very strongly like). "Semantic differential scales" are a type of ranking scale where the respondent circles a ranking (often 1 to 7) on a series of attributes of some target (ex., a political candidate) such as in the format below. Interesting       1    2    3    4    5    6    7      Uninteresting Finally, a "continuous ranking scale" is a graphical version of a ranking scale, where the respondent marks a point on a line between two defined limits.

    • Social distance scales. An ordered set of items is presented to the respondent regarding social relationships. For instance, the items might range from finding it acceptable for Arabs to visit the country, to immigrate to the country, to immigrate and live in this community, and so on down to live next door and marry into my family. While the respondent may check as many items as desired, the respondent's score is the most tolerant attitude checked (ex., 3 - live in this community). Such scales are often called Bogardus social distance scales, after the originator, Emory S. Bogardus(1933). This is a type of ordered scale or Guttman scale.

  • Comparative data gathering methods require the respondent or rater to assign a value with reference to one or more other items or bases of comparison.
    • Paired comparisons. In this method respondents must rank one of two items as higher on some dimension than the other. For instance, in an electoral study, the respondent may be asked to rate which of two candidates is preferred. See the separate module on multidimensional scaling for further discussion of paired comparisons and other comparative data gathering methods.

    • Q-Sorts. A usually large number of items (over a hundred is not exceptional) are sorted into a finite number of piles (typically 9) acccording to some criterion. The criterion may be nominal (ex., "Sort these names into nine piles such that names in a pile are most like other names in the same pile.") or ordinal ("Sort these adjectives into nine piles, ranging from 1 =very characteristic of you to 9 = very uncharacteristic of you."). See Block, 1961, 2008.

    • Fixed resources allocations. The respondent must allocate a fixed sum of resources (ex., $100 in imaginary money) to a set of choices. For instance, "Enter dollar amounts totalling $100 to indicate donations you would wish to make to the following five charities." This method works best, of course, in a computer environment where the fixed total is enforced automatically.

  • Other methods. While opinion data are the most common, almost any type of data (ex., inter-city mileage data) may become a constituent item in an index or scale.

• Combining methods are alternative approaches for representing multiple survey items or other measures as a single value. While some researchers refer to summary values derived by any combining method as a "scale", the researcher nonetheless should be aware of the distinctions below.
  • Indexes are sets of items which are simple functions of their constituent items, where the function is usually additive but (rarely) may be multiplicative or another function. Items in indexes may be weighted, and indexes may be normed (ex., to vary from 0 to 100). Items in an index normally will be more intercorrelated with each other than with other items but the index cannot be presumed to be unidimensional as the consituent items have not been established to be an ordinal scale or an ordered scale, as discussed below. (Of course, some authors use index and scale interchangeably).

  • Ordinal scales are combinations, usually formed additively, from a set of items each of which is thought to measure the same single dimension. Unidimensionality is established by demonstrating that the scale items are closely related. The most common test of whether items are closely related enough is Cronbach's alpha, discussed below. Though technically ordinal, in some cases ordinal scales may display interval or ratio data level properties.

  • Ordered scales. In contrast to ordinal scales, ordered scales are formed from items which can be ordered by difficulty (increasing probability that a subject will respond in a certain way or have a certain value) such that the observed value on one item predicts all lower difficulty items. The classic example is arithmetic problems, where the order by increasing difficulty is addition, subtraction, multiplication, and division. A student able to solve a multiplication problem will be able to solve addition and subtraction problems but not necessarily division problems. Guttman and Mokken scales meet this test of ordered relation among items but Likert scales, though ordinal, are usually are not ordered. Usually items in an ordered scale are not aggregated as with ordinal scales, but rather the subject is assigned the value of the most difficult scale item passed (ex., 3 = multiplication difficulty). Sometimes, however, such items are additively or otherwise aggregated in the same manner as other ordinal scales, but having met the criterion of ordered relationship, they can said to be presumptively unidimensional (on the difficulty dimension).

    • Ordered vs. ordinal scales. Whereas in ordinal scales items intercorrelate at a high level, this is not true of items in an ordered scale. For instance, in an ordinal scale measuring marital happiness, two items might be "I like to spend time with my spouse" and "I argue with my spouse less than once a week", and a person who agrees with one will be apt to agree with the other. In an ordered scale, in contrast, two items might be addition ability and division ability. These two items will correlate only to a moderate extent because, while division ability students will also display addition ability, the reverse will not be true.

    • Guttman vs. Mochen ordered scales. For the Guttman model, prediction is determinate whereas for Mokken scales prediction is probabilistic. The coefficients of scalability and reproducibility are common tests of whether items are sufficiently in an ordered relationship to justify their combination in an ordered scale.

    • Rasch scales, discussed below, are a type of ordered scale. and Rasch scaling is a well-developed statistical approach for such data.

  • Composite scores As defined here, composite scores reflect absolute totals on a finite set of constituent elements that may or may not be correlated. Such scores may or may not be rescaled (for example, rescaling the highest value as 100). For instance, in measuring the construct "philanthropy," there may be items on dollar amount of giving to environmental, poverty, religious, and other causes. Adding these dollar values together gives an overall measure of philanthropy. However, environmental giving, poverty giving, and religious giving may not be highly intercorrelated and may not stand in an ordered relationship to one another.

  • Factors or latent variables. Indicators (measured items) may also be combined as usually linear functions of other variables in the data model, as is done for factors and components in exploratory factor analysis, for latent variables in confirmatory factor analysis, or for latent factors in partial least squares path modeling, all discussed separately. The constructs based on any of these combining methods are ordinal scales, not ordered scales.

    • Ordered and composite score items in factor or latent variable procedures. Factor and latent variable procedures assume that indicator items are unidimensional in the ordinal scale sense. If ordered scale items or composite score items are entered into such techniques, the lower intercorrelation among items will be interpreted misleadingly as lower-difficulty items being weaker or even non-significant because they are associated with lower correlation, as discussed above.

  • Multidimensional scales. Multidimensional scaling (MDS), discussed separately, is another multivariate method of combining items into ordinal scales. MDS is based on a variety of alternative distance measures, one of which is correlation, and well-fitting multidimensional scales are composed of items with close distance with one another. As such MDS assumes that indicator items are unidimensional in the ordinal scale sense. As with factor methods discussed above, entering ordered or composite score items into MDS will lead to misleading inferences due to lower correlation of items in ordered scales and composite scores.

• Likert scales are considered a misnomer by some, as their ordinality refers only to an ordinal relationship of values within a single item and not to any particular combination of items, though Likert items are often combined in ordinal scales discussed above. Likert items are by far the most common type of survey item, in which the usual response categories are "strongly agree," "agree," "don't know,", "disagree," and "strongly disagree."
  • "Likert scales" as Likert items in ordinal scales. Values are ordinal within any given Likert item but sets of Likert items are not necessarily ordinal with respect to each other. Sets of Likert items typically 4 to 40 items) may be created on the same subject.When such sets pass the Cronbach's alpha or some other test of intercorrelation, they may be considered "scales." When no test of intercorrelation is employed to assure a common meaning of items comprising a latent variable, they may be considered "indexes".

  • Data level of Likert scales. Likert scales, as opposed to single Likert items, are commonly treated as interval in data level. Carifio & Perla (2008: 1150; see also Carifio & Perla, 2007) write, "a variety of studies on the nature of Likert scales (as opposed to single Likert items) has shown that Likert response format produces empirically interval data and in fact can approximate ratio data, in theory and actuality, if a hundred millimeter response line is used for marking regions, which has "always" and "never" as anchors. The weight of the empirical evidence, therefore, clearly supports the view and position that Likert scales (collections of Likert items) produce interval data, particularly if the scale meets the standard psychometric rule-of-thumb criterion of comprising at least eight items." Pell (2005) takes a similar view, though some scholars note that even Likert scales may fail to be interval (ex., Jamieson, 2004, 2005). Given widespread support for using Likert scales with interval procedures, particularly since reversion to nonparametric ordinal procedures involves loss of power, most researchers opt for assuming interval procedures are robust against small deviations from strict metricity.

  • Index of discriminative power (DP). DP is sometimes used as a statistical criterion for selecting more discriminating Likert (or other) items over less discriminating ones. Items with a high DP coefficient are ones where the mean score of the top 25% of respondents' scores on the item is very different from the mean score of the bottom 25% of respondents' scores on the item. That is, for a set of judges a set of Likert items, all meant to measure the same variable, is administered. For each judge, the mean score is computed for all items in the set (recoding where necessary, of course, so a "5", for instance, is always "high" on the variable). Judges are then ranked by mean score on the set of items, For the top 25% of judges and for the bottom 25% of judges, a mean value of judges is computed and the difference of the two mean values is the DP coefficient. In composing a scale of the variable, the items with the highest DP value are selected for inclusion in the final survey.

• Guttman Scales are ones in which the items constitute a unidimensional series such that an answer to a given item predicts the answers to all previous items in the series (ex., in an arithmetic scale, correctly answering a subtraction item predicts correctly answer a prior item on addition but not necessarily a later item on multiplication). That is, a respondent who answers an item in a positive way must answer less difficult items also in a positive way. Note that Guttman scales have been criticized as too stringent and deterministic, leading to the rise in popularity of Mokken scales, discussed below. Guttman scaling is not supported by SPSS or SAS, but PEPI freeware is available.

  • Counting scale errors in Guttman scaling.. Taking original dichotomous data shown in the upper half of the figure below, items are sorted to attempt to form as perfect a Guttman scale as possible. See the FAQ section for SPSS syntax to get data into Guttman scale format. However, even when resorted as shown in the bottom portion of the figure below, various counting issues arise.

    
    • Scale error (SE). Scale error is simply the number of scale deviations from one of the perfect scale patterns. For instance, in the figure above, all cases but 4, 7, and 8 may be considered to have one of the perfect scale patterns, where perfect is defined as having no interior "0's" and no exterior "1's" violating the triangular pattern shown in color. In the example above, one of the perfect scale patterns is missing (no subject responded "1" to items 1 and 7 and "0" to item 3 and all those to the left. If even with a larger sample one found this scale pattern absent or nearly so, this would mean that items 1 and 3 were redundant in the sense of not differentiating the subjects on the dimension being measured.
      • Guttman method. The Guttman method of counting scale error is to count so as to minimize error. In this method subject 4 is assigned to the perfect scale type illustrated by subject 5. This means subject 4 has 1 scale error, namely the response of "0" to item 3. Alternatively, subject 4 may be assigned to the perfect scale type where only items 1 and 7 are scored "1", in which case there would still be only one error (the "1" for item 2). Other cells labeled "G" are Guttman-method scale errors also. For this exampe, SE = 3 by the Guttman method.

      • Goodenough-Edwards method. The Goodenough-Edwards method, sometimes called the Likert method, although always having more errors than the Guttman method, is the method used by most software programs. In this method, the subject is assigned to the perfect scale pattern which, as coded here, has the same number of 1's as the subject. For the example above, subject 4 has three 1's and is therefore assigned to the same pattern as perfectly illustrated by subject 3. This gives subject 4 two errors, the cells labeled "GE" in the figure above - subject 4 "should have" responded "0" to item 2 and "1" to item 3. Other cells labeled "GE" are Goodenough-Edwards method scale errors also. For this example, SE = 5 by the Goodenough-Edwards method.

      • Minimum marginal reproducibility (MMR). If the researcher were guessing subject responses and the researcher knew the modal category of each item and always guessed the subject responded in the modal way, the researcher would always have 50% or more correct guesses. In fact, as the total in the modal categories, shown by the T_n row in the figure above, become a large proportion of all possible responses (subjects times items), the researcher's guessing success would become better and better. For the example above, the total responses (TR) = 9 subject * 8 items = 72. The total in the modal categories (T_n) sum to 53. Therefore MMR = 53/72 = .74, or 74%. T_n relates to how many errors are expected by chance because, for instance, if there are 15 subjects and 13 of them responded a given way to item 1, then there cannot possibly be more than (15 - 13) = 2 errors for that item, and similarly for all the other items.

      • Marginal error (ME) is the sum of non-modal categories = TR - MMR. For this example, ME = 72 - 53 = 19.

  • The coefficient of scalability, C_s is the standard method for assessing whether a set of items form a Guttman scale. C_s = 1 - (SE/ME) = (ME - SE)/ ME. Thus for the example above, C_s = (19 - 5)/19 = .74 Goodenough-Edwards method) or C_s = (19-3)/19 = .84 (Guttman method). C_s has a proportionate-reduction-of-error (PRE) interpretation: given that ME is maximum errors and given C_s =.74, we can say that scalar arrangement of the items reduces errors by (1 - .74) = 26%.

    By arbitrary convention, C_s should be .60 or higher to consider a set of items to be adequately scalable as an ordered Guttman-type scale. However, C_s should be understood in the context of MMR. MMR should not excessively high. Also, C_s should be appreciably higher than MMR (not the case for this example). See McIver & Carmines (1980: 50).

  • The coefficient of reproducibility, C_r is an alternative measure for assessing whether a set of items form a Guttman scale. C_r = 1 - (SE/TR), where SE = the number of scale errors and TR equals the total number of scale responses (which is the number of items times the number of subjects). The coefficient of reproducibility should be .90 or higher for the set of items to be considered scalable by Guttman criteria. For this example, C_r = 1 - (5/72) = .93 by the usual Goodenough-Edwards method of counting scale errors (it is even higher for the Guttman method).

  • Stouffer's H technique is a variant which gives greater stability to Guttman scales by basing each Guttman scale position on three or more items, rather than just one. If three items are used per position, then passing either two or three of the items will be entered as passing that position in the Guttman scale. This is not to be confused with Loevinger's H, which is used in conjunction with Mokken scaling.

  • Chi-square significance of Guttman scales. The researcher constructs a Guttman scale in the usual way and then constructs a chance, expected scale using a table of random numbers. Two lists are then made, one for the observed and one for the expected data, listing the number of respondents with no scale errors, the number with one error, and so on, These observed and expected frequencies can then be substituted into the usual chi-square formula, using the result with (n - 1) degrees of freedom to read a probability value from a table of the chi-square distribution. This allows the researcher to state not only, say, that the scale is Guttman scalable at .60 on the coefficient of scalability, but also that the scale is significant, say, at the .05 level.

• Proximity scales. With Guttman scales the assumption is that a person answering a given item will also answer less extreme or less difficult items but not necessarily those which are more extreme or more difficult. With proximity scales the assumption is that a person answering a given item will also answer nearby items but not necessarily those more extreme in either direction. Thus in a Guttman scale, items range from the least extreme to the most extreme. For a Guttman scale of conservatism, for instance, if a person answered a given item conservatively, they would answer all less extreme statements in a conservative direction also though not necessarily more extreme items. However, it might be assumed instead that political moderates might agree with less extreme conservative positions and with less extreme liberal positions, but not with the more expreme positions of either the right or left. When subjects are assumed to agree with nearby statements, proximity scaling should be adopted rather than Guttman scaling.

  Proximity scaling uses many of the same statistics (ex., coefficient of reproducibility) as Guttman scaling, but the definition of scale errors is different. In Guttman scaling, a perfect set of scores forms a triangle of responses, in which interior blanks are errors. In proximity scaling, a perfect set of scores forms a parallelogram of responses, in which interior blanks are errors. Note that proximity scales are not ordinal because a score on one item does not predict scores on all less extreme items.

  

  Herbert Weisberg (1972) compared a Guttman scale with a proximity scale of House voting on the compromise of 1850, which included five votes: fugitive slaves, Utah territory, Texas/New Mexico, California statehood, and D. C. slavery. The best Guttman scale displayed 15% of votes not conforming to Guttman criteria. The best proximity scale displayed only 2% of votes not conforming to proximity scale criteria, with error patterns as in the table above.

• Mokken scales are similar to Guttman scales but they are probabilistic whereas Guttman scales are deterministic. That is, in Mokken scales a respondent answering an item positively will have a significantly greater probability than null to answer a less difficult item in a positive way as well, whereas in perfect Guttman scales answering an item positively means the respondent will answer all less difficult items positively also.

  All items in a Mokken scale have different difficulties, as reflected in different proportions of positive responses. The graphic representation (called a trace line) of the probability of a positive response to an item should increase monotonically as the latent trait increases along the x axis (and where the y axis, of course, is the probability). Double monotony must not exist (that is, trace lines of items in a scale should not intersect). Also, trace lines must be steep enough to produce only a limited number of Guttman errors (exceptions to the rule that a positive answer to an item implies a positive answer to all easier items). Loevinger's H measures the conformity of a set of items to Mokken's criteria and validates their use together as a scale of a unidimensional latent variable.

  Loevinger's H is based on the ratio of observed Guttman errors to total errors expected under the null assumption that items are totally unrelated. Let E = the probability of a Guttman error and let Eo equal the same probability under the null model of totally unrelated items. H = 1 - E/Eo, as discussed below.

  Item	i	j
  	0	0
  	0	1
  	1	1
  Error pattern below
  	1	0

  Let item j be easier than item i, which in formulaic expression means that P(Xj = 1) > P(Xi = 1) -- the probability j is 1 is greater than the probability i is 1. Then Hij = l-E/Eo, where E = P(Xi = 1,Xj = 0) and Eo =P(Xi= 1)*P(Xj= 0) for a random subject. When there are no Guttman errors, Hij = 1. When the response is random (the null model), Hij = 0. (Of course, when computing these values one must recode where necessary so that the 1's and 0's have a consistent meaning across items).

  Then one can sum across all j to get Hi = 1 - E/Eo, which gives a measure of fit of item i to the Mokken scale. That is, Hi is the mean Hij for all pairs involving item i.

  Then one can sum across all item pairs to get H = 1 - E/Eo to get a measure of the quality of the total Mokken scale. H (Loevinger's H) is the mean Hij for all pairs. The arbitrary but customary criterion for validating a set of items as a Mokken scale is that H and all Hi must be .30. A rule of thumb is to speak of a "strong scale" for values exceeding 0.50, a "moderate scale" for values from .40 to .50, and a "weak scale" for values from .30 to .40. See further discussion below.

  
  
• Measures of Internal Consistency Internal consistency measures estimate how consistently individuals respond to the items within a scale. Note that measures of internal consistence are not tests of the unidimensionality of items in a scale. For example, if the first half of an instrument is educational items which correlate highly among themselves and the second is political items which correlate highly among themselves, the instrument would have a high Cronbach's alpha anyway, even though two distinct dimensions were present. Note that measures of internal consistency are often called measures of "internal consistency reliability" or even "reliability," but this merges the distinct concepts of internal consistency and reliability, which do not necessarily go together.

  1. Cronbach's alpha (a.k.a., "the reliability coefficient"), popularized in a 1951 article by Cronbach based on work in the 1940s by Guttman and others, is the most common estimate of internal consistency of items in a scale. Alpha measures the extent to which item responses obtained at the same time correlate highly with each other. Though widely interpreted as such, strictly speaking alpha is not a measure of unidimensionality. Rather, alpha is a measure of level of mean intercorrelation weighted by variances (in contrast to standardized item alpha, discussed below, which equalizes variances), or a measure of mean intercorrelation for standardized data, stepped up for number of items. A set of items can have a high alpha and still be multidimensional. This happens when there are separate clusters of items (separate dimensions) which intercorrelate highly, even though the clusters themselves do not intercorrelate hightly. Also, a set of items can have a low alpha even when unidimensional if there is high random error.

     • Alpha tied to number of items. In addition to estimating internal consistency (a.k.a. "reliability") from the average correlation, the formula for alpha also takes into account the number of items on the theory that the more items, the more reliable a scale will be. That is, when the number of items in a scale is higher, alpha will be higher even when the estimated average correlations are equal. As the number of items rises, alpha rises.

     • Alpha tied to variation. Also, the more consistent within-subject responses are, and the greater the variability between subjects in the sample, the higher Cronbach's alpha will be. Finally, alpha will be higher when there is homogeneity of variances among items than when there is not.

     • Alpha cut-off criteria. The widely-accepted social science cut-off is that alpha should be .80 or higher for a set of items to be considered a scale, but some use .70 as acceptable even for confirmatory research. Using alpha equal to or higher than .60 but less than .70 is sometimes considered acceptable for exploratory research, but this practice is controversial. That .70 is as low as one may wish to go is reflected in the fact that when alpha is .70, the standard error of measurement will be over half (0.55) a standard deviation.

     • Negative alphas. Under some circumstances, alpha may be negative. This reflects a serious coding error in the data: data should be recoded if necessary to assure that all items are coded in the same conceptual direction. If there are n negatively coded items and m positively coded items, there will be n*m negative correlations. The number of positive correlations is based on the number of combinations within each set, namely n!/(n-2)!*2 + m!/(m-2)!*2. Two variables have 1 combination, three have 3, four have 6, five have 10, six have 15, etc. For some circumstances there will be more negative correlations than positive. For instance, if the negatively and positively coded sets have 5 and 4 items respectively, there will be 5*4=20 negative correlations and only 6+10 positive correlations and alpha will be negative. The researcher should check the covariance matrix of scale items to make sure there are no negative covariances reflecting coding error.

     • SPSS. In SPSS, Cronbach's alpha is found under Analyze, Scale, Reliability Analysis. Then in the Statistics button, select Scale to get alpha. You can also check Scale if deleted, in which case alpha will be computed both for all variables entered, and also for all remaining variables if any one is dropped (the alpha if deleted is listed in a table, one for each variable). See discussion in reliability section.

     • Alpha vs. reliability. Alpha makes no assumptions about what one would obtain at a different time (the latter is "reliability," discussed below). Alpha = (# of items/(# of items - 1)) * (1 - (sum of the variances of the items/variance of the total score)). See Miller, M.B. (1995). Coefficient alpha: A basic introduction from the perspectives of classical test theory and structural equation modeling. Structural Equation Modeling, Vol. 2, No. 3: 255-273.

     • Congeneric v. tau-equivalent v. parallel indicators. The psychometric literature sorts indicator sets for latent variables into three types. Congeneric sets are those which are presumed to have passed some test of convergent validity, such as Cronbach's alpha, discussed above. Tau-equivalent sets also have equal variances for the indicators. Parallel sets are tau-equivalent sets where the items have equal reliability coefficients. Note that when items are less than tau-equivalent, alpha is only a lower bound on reliability.
       • Tau-equivalence in SEM. In structural equation modeling (SEM), tau-equivalence is tested by comparing an unconstrained model with one in which the factor loadings of the indicator variables on the factor are all set to 1.0, then seeing is the chi-square difference is insignificant. If the indicator model for the factor is found to be tau-equivalent, then the original model is compared to one in which measurement error variances are all constrained to be equal, and another chi-square difference test is conducted.

  2. Standardized item alpha. Standardized item alpha is a variant of alpha designed to be used to test k one-item parallel tests and is meant to be interpreted as the average Spearman-Brown-corrected reliability estimate from all possible splits in a series of applications of split half reliability on a single test. It is also the value of computed alpha when all scale items are standardized to have equal means and variances. Regular Cronbach's alpha will usually but not always be less than or equal to standardized item alpha. Both versions usually yield the same substantive conclusions and both versions are widely used. Some argue that the more restrictive standardized item alpha is appropriate when assuming classic parallel tests, otherwise conventional Cronbach's alpha should be used.

  3. Factor analysis, discussed in a separate section, is often used to establish internal consistency. Intercorrelated items will sort themselves out on separate factors if the data exhibit a simple factor structure, and this is taken as evidence that the indicator items "belong together" in a scale, and also as evidence of discriminant validity, that scales are different from one another.

  4. Rasch scales. Rasch logistic models, discussed in the section on validity, are often preferred to Cronbach's alpha or factor analytic methods because Rasch models take into account not only additive relations among indicator items, as alpha and factor analysis does, but also relations where items exhibit an ordered relationship (ex., order of difficulty) even when correlations may not be high. See Pesudovs, Garamendi, Keeves, & Elliott (2003) for an example of using Rasch models to reevaluate and refine a scale originally developed using additive methods such as Cronbach's alpha and factor analysis.

  5. Tarkkonen's measure of scale reliability was introduced by Lauri Tarkkonen (1987) and further studied by Kimmo Vehkalahti (2000). As yet much less widely used than Cronbach's alpha, Tarkonnen's measure has some statistical properties superior to it.

  6. Loevinger's H, discussed above with reference to Mokken scales, is another alternative to Cronbach's alpha. H is a better approximation to the classical reliability coefficient rho than Cronbach's alpha, which has been widely used to measure the internal consistency of a scale. Specifically, alpha strongly underestimates rho when there is large variation in item difficulties.

  7. Kuder-Richardson Formula 20, KR20 is a special case of Cronbach's alpha, for ordinal dichotomies. That is, Alpha = KR20 when the pair of items are both dichotomous. KR20 = (# of items/(# of items - 1)) * (1 - ((sum of p*q)/variance of the total score)), where p = proportion correct and q = proportion incorrect.

  8. Spearman's rho is a form of rank order calculation. It is calculated with the same formula as for Pearson's r correlation, but using rank rather than interval data. Since scales are rank-order data, rho is often used in place of r when correlation is called for, as in inter-item correlations, below. The median rho between all pairs of items in a scale is a classic measure of reliability, in the sense of internal consistency. Rho values above .60 are considered necessary for an adequate scale.

  9. Inter-item correlation is used to spot reverse-coded items. Scale items should correlate positively with one another. A negative correlation may well indicate, for instance, that an item was coded such that "1" meant the opposite direction (ex., low) from the meaning of "1" (ex., high) for the other items in the scale.

  
  

• Measures of Agreement. Agreement is closely related to internal consistency. In this context a measure of agreement assesses the extent to which two raters give the same ratings to the same objects. The set of possible values for one rater forms the columns and the same set of possible values for a second rater forms the rows. Cohen's kappa is the leading measure of agreement and is available in SPSS (Analyze, Descriptive Statistics, Crosstabs, Statistics, Kappa).
  • Cohen's kappa is used when the row and column variable is the same, as in assessing agreement between two raters. There are three versions: (1) as originally proposed, kappa measures agreement between two raters; (2) later Fleiss and Light developed a generalized version of kappa for more than two raters; and (3) Cohen also developed a "weighted kappa" version to allow for degrees of agreement rather than simple agree/disagree classification. SPSS supports the original kappa by the menu choices Analyze, Descriptive Statistics, Crosstabs, Statistics, Kappa. SPSS also computes the standard error of kappa (the sampling distribution of kappa is discussed in Liebetrau, 1983: 33-34, who also discusses weighted kappa, 34-36).

    Kappa = (observed concordance - concordance by chance)/(1-concordance by chance), where "by chance" is calculated as in chi-square (multiple row marginal times column marginal and divide by n).

    Example: Two raters rate a target as "A," "B," or "C," as in the table below:

    Rating	A	B	C	Total
    A	11	2	4	17
    B	1	14	2	17
    C	4	2	10	16
    Total	16	18	16	50

    Observed concordance (agreement) = diagonal sum divided by n = (11+14+10)/50=0.70
    Expected concordance by chance = diagonal row*column totals divided by n, summed, divided by n = (16*17/50 + 18*17/50 + 18*16/50)/50 = (5.44 + 6.12 + 5.76)/50 = .35
    Kappa = (Observed Concordance - Expected Concordance)/(1-Expected Concordance) = (.70 - .35)/(1 - .35) = .54 (.55 without rounding).

• Measures of Reliability Reliability is a necessary but not sufficient condition for validity of scale items. Reliability is discussed in a separate section.

• Measures of Item Discrimination

  • Biserial correlation is the correlation between a subject's response to an item (right or wrong) and his or her total score on the test. It assumes that the distribution of test scores is normal and that there is a normal distribution underlying the right/wrong dichotomy. The biserial correlation has maximum values greater than unity. There is no exact test for the statistical significance of the biserial correlation coefficient.

  • Point-biserial correlation is the correlation between a subject's response to an item (right or wrong) and his or her total test score. It differs from biserial correlation in assuming that the right/wrong dichotomy is a natural dichotomy not underlaid by a normally discributed variable. Point-biserial correlation assumes that the test score distribution is normal. It ranges from +1 to -1. Its significance is tested with the Student's t test with N - 2 degrees of freedom at the desired level of confidence, where N is the total number of subjects taking the instrument.




Assumptions

• Unidimensionality. Benchmarks, indexes, and scales are usually assumed to measure a single dimension of meaning. Ambiguous results and low correlations may result using a benchmark, index, or scale when the researcher thinks he/she is measuring a single latent variable but in fact there is multidimensionality. For example, an index of "job satisfaction" may yield ambiguous results if, in fact, "task satisfaction" and "workgroup satisfaction" are separate dimensions which behave differently, yet have been forced together in the "job satisfaction" index. Guttman scaling is one test of unidimensionality.

• Equality of Means and Standard Deviations. When combining items in a scale, if items have different means then one is weighting the items with higher means more. Unequal standard deviations also lead to unequal weighting in the sense that the item with the greater standard deviation will contribute more to the tails of the scale variable distribution. Put another way, Cronbach's alpha correctly estimates reliability only under tau equivalence of the items.




Frequently Asked Questions

• Should I use existing items or create my own?
  Wherever possible, the researcher should use standard items which have been validated in other settings. In addition to heightened validity, use of standards enables the researcher to make better comparisons of his/her results with those of other researchers using the same standard instruments. Even if unique items are needed for the local research setting, it is helpful use standard instruments to determine the correlation of local items with standard items.

• How can I find standard scales and instruments?
  The central method is to explore instruments utilized in research uncovered in one's review of the literature on the substantive topic, including the bibliography in this section. Computer searches using DIALOG are another way -- the POLL database (file 468 on DIALOG), for instance, contains nearly all questionnaire items asked by major polling organizations. A third method is to use the software-based expert system, "Measurement and Scaling Strategist," which is a module of the software package Methodologist's Toolchest, published by Sage Publications, Thousand Oaks, CA.

• When may ordinal scale data be used in regression and other interval techniques?
  Technically, never.

  As an independent: Methodologists use a rule-of-thumb that there must be a certain minimum number of classes in the ordinal independent (Achen, 1991, argues for at least 5; Berry (1993: 47) states five or fewer is "clearly inappropriate"; others have insisted on 7 or more). Use of 7-point scales or higher would seem best, but it must be noted that use of 5-point Likert scales with interval procedures is extremely common in the literature.

  As a dependent: One method is to test to see if there are significant differences in the regression equation when computed separately for each value class of the ordinal dependent. If the independents seem to operate equally across each of the ordinal levels of the dependent, then use of an ordinal dependent is considered acceptable.

• Should I standardize my variables before testing for reliability of scale items with Cronbach's alpha?>
  Probably not, though this depends on what you want. You will probably get different reliability results for standardized as compared to raw data. Raw scores allow some variables to have greater variance than others, and the variables with greater variance will be more likely to correlate with other scale items and thereby pass the Cronbach's alpha filtering test for inclusion of scale items. Note SPSS's RELIABILITY procedure allows two computational methods, and only Method 2 generates standarized alpha coefficients even if the input data is raw data.

• I want to add item scores together to form a scale. How can I test if the items are additive?
  This can be done with Tukey's test of nonadditivity of items. If sig(nonadditivity) < .05 for a set of items, then the researcher concludes the items are not additive. Transformations of the variable(s) may result in additivity. A footnote to the SPSS output table may suggest a power or root needed to achieve additivity. Be warned, however, that large powers or roots should not be undertaken automatically as this may merely mask inter-item differences.

  In SPSS, select Analyze, Scale/Reliabiliy Analysis; select your variables; click Statistics; select Tukey's test of additivity. Continue. OK. Look for the "Additivity" row in the resulting ANOVA table.

• How can I get my SPSS worksheet into Guttman scaling format?
  While SPSS does not support Guttman scaling directly, the following SPSS syntax may be adapted to create a new worksheet in Guttman scaling format. The syntax does not compute Guttman scaling nor computer scaling errors.

  *GUTTMAN SCALING SPSS SYNTAX
  * by david_garson@ncsu.edu, 2008
  * Creates a new Guttman-sorted worksheet, with cases and variables sorted
  * The original worksheet is left intact.
  * This assumes cell entries are 0, 1
  * Also assumes variables are named as in the first SORT CASES command below; change as needed.
  * Also assumes 9 cases; change count in COUNT statement below by changing var009; also change N OF CASES to actual n.
  SORT CASES BY item1 item2 item3 item4 item5 item6 item7 item8.
  EXECUTE.
  FLIP VARIABLES=all.
  COUNT isum = var001 to var009 (1).  
  SORT CASES BY isum.
  FLIP VARIABLES=all.
  DELETE VARIABLES case_lbl.
  N OF CASES 9.
  EXECUTE.
  

• What software programs compute Guttman scaling coefficients of scalability and reliability?
  While early versions of SPSS computed Guttman coefficients, recent versions do not (note: while SPSS computes Guttman's lower bounds, this is something different, discussed in the section on reliability analysis.) To this author's knowledge, Guttman scaling is not supported by SAS or other major statistical packages. However, PEPI, a statistical suite of programs for epidemiologists, is available as freeware at http://www.sagebrushpress.com/pepibook.html and the "Scale" component does support Guttman scaling or up to 20 dichotomous items.

• What is the 'cluster bloc technique'?
  The cluster bloc technique was used commonly before modern scaling methods were developed. It is still used is some areas, such as legislative roll call analysis. Cluster bloc analysis allows researchers to rank individuals according to some criterion, such as ranking U. S. senators according to their agreements on a set of environmental votes. There are four steps:
  1. Selecting the criteria. A quantitative criterion is selected, such as Senators' votes on a set of votes deemed by an environmental interest group to be key environmental votes.

  2. Computing indices of agreement. An "agreement" is the match between any two Senators' votes. For n people, there will be n(n-1)/2 pairs, so for 100 Senators there would be 4,950 pairs. A simple index of agreement would be the percent of times the aye votes of Senator A matches the aye votes of Senator B. This, however, would not take into account absences. Absences could be handles by ignoring votes where one or both Senators were absent, or by substituting some estimate for the actual vote, as through knowledge of announced positions.

  3. Establishing non-arbitrary cutoff points. The researcher must then select an index value above which the pair of Senators would be considered to be in the same cluster. Typically, the cutoff is set at the lowest level significantly unlikely to occur by chance. This calculation requires a bit of probability theory. The probability of r agreements out of n votes equals
     _nC_r * p^r * q^(n-r)
     where _nC_rr is the probability of agreement times itself r times, and q^(n-r) is the probability of non-agreement times itself (n-r) times. Recall from probability theory that _nC_r = n!/(n-r)!r!, where ! is the factorial sign.Using the formula above, and defining p and 1 in terms of chance agreement being 50% (.5), there is a .004 chance of agreeement on 8 of 8 votes; a .031 chance of agreement on 7 of 8 votes; and a .109 chance of agreement on 6 of 8 votes. Taking the usual .05 significance cutoff, the lowest level significantly unlikely to occur by chance is 7 of 8 votes, or an agreement index of .875, which would be the non-arbitrary cutoff.

  4. Forming the cluster block matrix. A matrix is formed in which both rows and columns are the 100 Senators, and cell entries are the agreement scores. The Senators are ordered as follows: (1) the pair with the highest agreement score is listed in the first and second columns; (2) the next individual to be listed is the Senator with the highest agreement score with one of the two Senators listed in the first step; (3) step 2 is repeated until all Senators are listed. Whenever a Senator is listed, the corresponding cell entries are filled in for agreement indexes that meet the cutoff level, here .875. Other cells are left blank.

  The cluster bloc technique results in an ordering of Senators, displayed in a matrix, such that blocs of agreement are easily identified as clusters on the matrix. The blocs that are identified for one set of votes can be compared to blocs for another set of votes. Note that agreement is not equivalent to similar thinking: agreement may also be born of party discipline, constituent pressures, or mutual bargaining ("log-rolling").

• What are Thurstone equal-appearing interval scales?
  Practically unknown today and rare in their own day, these were attempts to create ordinal scales which approximated interval measurement. They are primarily encountered today in methodology texts simply for pedagogical reasons. The general procedure of the Thurstone technique is to have 'judges' rank opinion statements into a set of ordered piles. Judges must sort slips containing the statements according to how favorable or unfavorable the statement is to the variable being studied, not how favorable the judge is to the statement. (For example, the statement, "I would not want an African-American to purchase a home on the street where I live." would be ranked low on the variable 'tolerance,' whether or not the judge agreed with the statement.) When the judges are finished, for each judge the statement slips will be ordered into numbered piles. Each statement slip is given the number of its pile. The slips are sorted by statement and then the median (not mean) pile value is determined for each statement. Statements are sorted into piles according to their median value. The reseacher then selects some statements from each pile to form the scale, giving preference to those statements the judges agreed on in ranking.

  The Q-dispersion statistic was created by Thurstone as a measure of item ranking agreement. Q values were an ordinal analog to standard deviations. For each item a table was constructed with three columns: (1) the pile numbers into which judges sorted the item (ex., 1 to 11); (2) the number of judges who sorted the item into the corresponding pile number; and (3) the cumulative percentage of the second column. To compute the Q value, one interpolated the pile number of the 25% and 75% cumulative percentage points (called "ogive values") and subtracted. Thus if 25% of judges ranked an item 3.7 or less and 75% ranked it 5.5 or less, then Q would be 1.8. Items with the lowest Q values in each pile were selected for the scale, making sure to select items from all piles.

  Studies found Thurstone scales were not significantly better than Likert scales in establishing intervalness, and it was also found many statistical procedures were robust in the face of moderate departures from the assumption of intervalness. With these findings, the complicated and costly Thurstone procedure fell into disuse.




Bibliography

Note: While earlier social scientists relied on collections of scales such as described below, this has largely been superceded in the Internet age by web searches. For instance, in a study focusing on collaboration, the researcher may enter "collaboration scale" and variants thereof to find articles which have developed collaboration scales in the past, uncovering resources such as Anderson et al., listed below.

Methodology
• Achen, C. H. (1991). A polychotomous linear probability model. Political Methodology Society. Berkeley, CA. Achen argues that using in regression an ordinal variable with fewer than 5 categories introduces overly biased results.
• Aiken, Lewis R. and Lewis A. (1996). Rating Scales and Checklists : Evaluating Behavior, Personality, and Attitudes. NY: John Wiley & Sons, ISBN 0471127876. Covers theory, construction and evaluation of rating and attitude scales, checklists and related questionnaires for research and practice in industrial, clinical and educational contexts. A set of computer programs to assist in designing, administering, scoring and analyzing various types of psychometric instruments is included with the book.
• Aiken, Lewis R. and Lewis A. (1997). Questionnaires and Inventories : Surveying Opinions and Assessing Personality. NY: John Wiley & Sons, ISBN 0471168718. Covers the construction, administration, scoring, and interpretation of questionnaires, psychological inventories, and population surveys. A companion volume to Rating Scales and Checklists. Comes with a computer disk which assists in instrument construction.
• Berry, William D. (1993). Understanding Regression Assumptions. Series: Quantitative Applications in the Social Sciences, No. 92. Thousand Oaks, CA: Sage Publications. Berry argues that using in regression an ordinal variable with 5 or fewer categories is "clearly inappropriate" (p. 47).
• Block, Jack (1961). The Q-sort method in personality assessment and psychiatric research.. Springfield, IL: Charles C. Thomas. (Monograph).
• Block, Jack. (2008). The Q-Sort in character appraisal: Encoding subjective impressions of persons quantitatively. Washington, DC: American Psychological Association.
• Bogardus, Emory S. (1933). A social distance scale. Sociology and Social Research, 17: 265-271.
• Carifio James & Perla, Rocco (2007). Ten common misunderstandings, misconceptions, persistent myths and urban legends about Likert scales and Likert response formats and their antidotes. Journal of Social Science 2:106–16.
• Carifio, James & Perla, Rocco (2008). Resolving the 50-yer debate around using and misusing Likert scales. Medical Education 42: 1150-1152.
• DeVellis, Robert F. (2003). Scale development: Theory and applications. Second edition.. Thousand Oaks, CA: Sage Publications. Covers factor analysis and item response theory.
• Gillespie, Michael, Elizabeth M. Tenvergert, and Johannes Kingaman (1988). Using Mokken methods to develop robust cross-national scales: American and West German attitudes toward abortion. Social Indicators Research, Vol. 20: 181-203 ;
• Halachmi, Arie (1996). Organizational Performance and Measurement in the Public Sector : Toward Service, Effort and Accomplishment Reporting. Greenwood Publishing Group, ISBN: 0899309585.
• Jamieson, S. (2004). Likert scales: How to (ab)use them. Medical Education 38: 1212-1218.
• Jamieson, S. (2005). Author reply. Medical Education 9:970-971.
• Liebetrau, Albert M. (1983). Measures of  association. Newbury Park, CA: Sage Publications. Quantitative Applications in the Social Sciences Series No. 32. Discusses kappa.
• Lodge, Milton (1981), Magnitude scaling: Quantitative measurement of opinions. Thousand Oaks, CA: SAGE Publications.
• McConaghy, Maureen J. (1975). Maximum possible error in Guttman scales. Public Opinion Quarterly 39(3), 343-357.
• McIver, John P. & Carmines, Edward G. (1982). Unidimensional Scaling. Thousand Oaks, CA: Sage Publications. Quantitative Applications in the Social Sciences Series, No. 24.
• Mokken, Robert J. (1971). A theory and procedure of scale analysis. The Hague: Mouton.
• Mokken, R. J. and C. Lewis (1982). A non parametric approach to the analysis of dichotomous item responses. Applied Psychological Measurement, Vol. 6: 417-430.
• Netemeyer, Richard G., William O. Bearden, and Subhash Sharma (2003). Scaling procedures: Issues and applications. Thousand Oaks, CA: Sage Publications. Focus on reliability and validity models, use of confirmatory factor analysis for finalizing the scale.
• Pell, G. (2005). Uses and misuses of Likert scales. Medical Education 39: 97.
• Pesudovs, K., Garamendi, E., Keeves, J. P., & Elliott, D. B. (2003). The activities of daily vision scale for cataract surgery outcomes: Re-evaluating validity with Rasch analysis. Investigative Ophthalmology and Visual Science 44:2892-2899.
• Schneider, Saundra K., William G. Jacoby, and Jarell D. Coggburn (1997). The structure of bureaucratic decisions in the American states. Public Administration Review, Vol. 57, No. 3 (May.June): 240-249. Illustrates Mokken scaling in a scale of optional Medicaid program services.
• Sijtsma, K. and A. Verwey (1992). Mokken scale analysis; theoretical considerations and an application to transitivity tasks. Applied Measurement in Education, Vol. 5, No. 4: 355-373.
• Smith, Peter, ed. (1996). Measuring Outcomes in the Public Sector. Taylor & Francis, ISBN: 0748404031.
• Spector, Paul (1992). Summated Rating Scale Construction. Sage Quantitative Applications in the Social Sciences monograph #82.
• Walters, Jonathan (1998). Measuring Up : Governing's Guide to Performance Measurement for Geniuses (And Other Public Managers). Governing Magazine, ISBN 1568024584.
• Weisberg, Herbert (1972). Scaling models for legislative roll-call analysis. American Political Science Review, 66(4): 1301-1316. Cited with regard to proximity scaling.

  
  General

• Davis, James Allan and Tom W. Smith (1994). General Social Surveys, 1972-1994: Cumulative Codebook Chicago National Opinion Research Center. Distributed by The Roper Center for Public Opinion Research, Storrs, Connecticut. The GSS codebook is available online at http://www.icpsr.umich.edu/gss/codebook.htm.
• Gallup, George (1997). The Gallup Poll. Wilmington, DE: Scholarly Resources, Inc. Vols. 1 - 3 covered 1935-1971 and were published by Random House (NY). Subsequent volumes have been published every two or three years. At this writing, the most recent was the volume covering 1995-1996, published in 1997.
• Hastings, Philip K. and Jessie C. Southwick, eds. (1998). Index to International Public Opinion. NY: Greenwood Publishing Group. The first volume in this series, originally published by the Roper Public Opinion Research Center, appeared in 1974. At this writing, the most recent annual volume covered 1996-1997, published in 1998.
• Horn, Robert V. (1993). Statistical Indicators: For the Economic and Social Sciences. Cambridge, UK: Cambridge University Press. Discusses a wide range of social indicators, their politics and issues.
• * Maddox, Taddy (2002). Tests: A comprehensive reference for assessment in psychology, education, and business. Fifth Edition. Austin, TX: Pro-Ed Publishers. A leading resource on scales and instruments. Goes with Keyser, Daniel J. (2004). Test Critiques. Test Corporation of America. Published periodically, this covers over 700 tests in psychology, education, and business, giving a detailed description of each measure: applications, and discussion of validity and reliability. A leading standard source.
• POLL. Note that Dialog Information Services and other vendors carry the POLL database, containing virtually all items asked in Gallup, Roper, Harris, and many other polls, with frequencies. Data are not available from POLL, but rather may be obtained from organizations such as the Inter-University Consortium for Political and Social Research (ICPSR) and the polling organizations themselves.
• Sirgy, M. J., D. Rahtz, and Dong-Jin Lee. (2004) Community Quality-of-Life Indicators: Best Cases (Social Indicators Research Series). Berlin, Germany: Springer.

  
  Business

• American Consumer Satisfaction Index (ACSI) is a commercial scale often used in consumer research, including satisfaction with government services.
• Radaev, V. (2002). Entrepreneurial Strategies and the Structure of Transaction Costs in Russian Business. Problems of Economic Transition 44(12): 57-84. Developed a scale of administrative control.

  
  Crime

• Brodsky, Stanley L. and Smitherman, H. O'Neal (1983). Handbook of Scales for Research in Crime. New York: Plenum. Volume 5 of the series "Perspectives in Law & Psychology".
  

  Economic/Financial/Work

• William O. Bearden, William O. and Netemeyer, Richard G. (1999). Handbook of Marketing Scales: Multi-Item Measures for Marketing and Consumer Behavior Research (Association for Consumer Research). Thousand Oaks, CA: Sage Publications.
• Fabozzi, Frank J. and Harry I. Greenfield (1984). The Handbook of Economic and Financial Measures. Homewood, IL: Dow Jones-Irwin.
• Institute for Social Research, University of Michigan (1976). Measures of Occupational Attitudes and Occupational Characteristics.

  
  Education

• Bishop, Lloyd K. and Paula E. Lester (1993). Instrumentation in Education : An Anthology . Garland Publishing, ISBN 0815306385.
• ETS test directory. A description of tests available from the Educational Testing Service, including the SAT, Academic Profile, TOEFL, and many others.
• PRO-ED Inc. is a leading publisher of standardized tests, books, curricular and therapy materials, and professional journals covering Speech, Language, and Hearing; Psychology and Counseling; Special Education (including Developmental Disabilities, Rehabilitation, and Gifted Education); Early Childhood Intervention; and Occupational and Physical Therapy.

  
  Politics

• Miller, Warren E. and Santa A. Traugott (1989). American National Election Studies Data Sourcebook, 1951-1986. Camridge, MA: Harvard University Press.
• Robinson, John P., Phillip R. Shaver, and Lawrence S. Wrightsman, eds. (1998). Measures of Political Attitudes. Academic Press, ISBN 012590245X.
• Robinson, John P., Jerrold G. Rusk and Kendra B. Head (1968). Measures of Political Attitudes. Survey Research Center, Institute for Social Research.
• Russett,Bruce M. & Hayward R. Alder (1977). World Handbook of Political and Social Indicators. Westport, CT; Greenwood Press.
• Taylor, Charles Lewis, Michael C. Hudson, and D.A. Jodice (1983). World Handbook of Political and Social Indicators. New Haven, CT: Yale University Press.

  
  Psychology

• American Psychological Association (1986). Standards for Educational and Psychological Testing, Revised Edition Amer Psychological Assn; ISBN 0912704950.
• Buro, O. (1987). Tests in Print. Highland Park, NJ: Gryphon Press. Precursor of the series most recently edited by Impara and Plake, below.
• Exner, John E. (1993). The Rorschach : A Comprehensive System - Basic Foundations, Third Edition. NY: John Wiley & Sons, ISBN 0471559024. This is the basic text/reference used to administer, score, and interpret the Rorschach projective test according to the "Comprehensive System" --the most widely used approach to the test.
• Graham, John R. (1993). MMPI-2 : Assessing Personality and Psychopathology, Second Edition. NY: Oxford Univ Press; ISBN: 0195079221. The standard reference for the Minnesota Multiphasic Personality Inventory.
• Groth-Marnat, Gary (1996). Handbook of Psychological Assessment, Third Edition. NY: John Wiley & Sons, ISBN 0471052205. Reviews commonly used psychological assessment tests such as the Wechsler Intelligence Scales, Bender-Gestalt, Rorschach, Thematic Apperception Test (TAT), Minnesota Multiphasic Personality Inventory (MMPI/MMPI-2), projective drawings, and California Psychological Inventory. Discusses validity and reliability of each.
• Impara, James C. and Barbara S. Plake, eds., The Thirteenth Mental Measurements Yearbook (1998). Buros Inst., ISBN 091067454X (13th Ed). Published annually, this is the leading source for psychological scales. Has articles, bibliography, and critical reviews of psychological tests. Earlier volumes should be consulted also.
• Newmark, Charles S. (1996). Major Psychological Assessment Instruments, Second Edition. Boston: Allyn and Bacon, ISBN 0205168698.
• Perrin, Eward B & Koshel, Jeffery J. (1997). Assessment of Performance Measures for Public Health, Substance Abuse, and Mental Health.. Washington, DC: National Academy Press.
• Russell, Charles H. and Inger Megaard (1988). General Social Survey, 1972-1986 : The State of the American People . Recent Research in Psychology Series. NY: Springer Verlag; ISBN 038796746.
• * Spies, Robert A., Plake, Barbara S., and Murphy, Linda L. (2005). The sixteenth mental measurements yearbook (Mental Measurements Yearbook). This is the leading directory of scales in psychology. It goes with this index: Murphy, Linda L. (2005). Tests in print VI: An index to tests, test reviews, and the literature on specific tests (Tests in Print). Omaha, NB: University of Nebraska Press.
• Stamm, B. Hudnall, ed. (1996). Measurement of Stress, Trauma, and Adaptation. Sidran Press, ISBN 1886968020.
• Weiner, Elliot A. and Barbara J. Stewart (1984). Assessing Individuals: Psychological and Educational Tests and Measurements. Boston: Little, Brown.

  
  Public Administration/Organizations

• Ammons, David N. (1996). Municipal Benchmarks: Assessing Local Performance and Establishing Community Standards. Thousand Oaks, CA: Sage Publications.
  
  Presents relevant national standards developed by professional associations, with actual performance targets and performance results from a sample of governments. Areas covered include animal control, city attorney, city clerk, courts, data processing, development administration, emergency communications, emergency medical, finance & budgeting, fire service, library, parks & recreation, personnel, police, property appraisal, public health, public transit, public utilities, public works/engineering, purchasing/warehousing, solid waste collection, and traffic control.
• Anderson, C. M., Martin, M. M., & Infante, D. A. (1998). Decision-making collaboration scale: Tests of validity. Communication Research Reports, 15, 245-255.
• Boyne, George A., Kenneth J. Meier, Laurence J. O'Toole, & Richard M. Walker, eds. (2007). Public Service Performance: Perspectives on Measurement and Management. Cambridge, UK: Cambridge University Press. Leading scholars discuss performance measures.
• Holliday, Andrew J. (1997). The Definition and Measurement of Antitrust Enforcement. J.A.I. Press, ISBN: 0762303239.
• Huls, Mary Ellen (1985). Productivity/Performance Measures for Public Services at the Local Level: A Bibliography (Public Administration Series--Bibliography, P-1742) . Monticello, IL: Vance Bibliographies.
• Jacobs, Rowena, Smith, Peter C. and Street, Andrew (2006). Measuring efficiency in health care: Analytic techniques and health policy. Cambridge, UK, and NYC: Cambridge University Press.
• Perrin, Edward B. and Jeffrey J. Koshel, eds., (1997). Assessment of Performance Measures for Public Health, Substance Abuse, and Mental Health. National Academy Press, ISBN: 0309057965.
• Perry, James L (1996). Measuring public service motivation: An assessment of construct reliability and validity. Journal of Public Administration Research and Theory 6(1): 5-22.
• Price, James L. and Charles W. Mueller (1986). Handbook of Organizational Measurement. Marshfield, MA: Pitman.
• Richardson, Valerie J. (2002) Annual Performance Planning: A Manual for Public Agencies. Vienna,VA: Management Concepts. An OMB-oriented manual with case illustrations.
• Schneider, S. K., Coggburn, J. D., & Jacoby, W. G. (1997). The structure of bureaucratic decisions in the American states. Public Administration Review, 573): 240-249 . Developed a scale of administrative decisions in state Medicaid programs.
• Volkwein, J. F., Malik, S. M., and Napierski-Prancl, M. (2004). Administrative satisfaction and the regulatory climate at public universities. Journal Research in Higher Education 39(1): 43-63. Describes a scale of administrative flexibility.

  
  Social

• Beere, Carole A. (1979). Women and women's issues: A handbook of tests and measures. San Francisco: Jossey-Bass.
• Beere, Carole A. (1990). Sex and Gender Issues: A Handbook of Tests and Measures. Westport, CT: Greenwood Press.
• Bowling, Ann (1997). Measuring Health: A Review of Quality of Life Measurement Scales (Second Edition). Buckingham, UK and Philadelphia, PA: Open University Press.
• Bonjean, Charles M., Richard J. Hill, and S. Dale McLemore (1967). Sociological Measurement: An Inventory of Scales and Indices. San Francisco:Chandler.
• Converse, Philip E., Jean D. Dotson, Wendy J. Hoag, and William H. McGee, eds. (1980). American social attitudes data sourcebook, 1947-1978. Cambridge, MA: Harvard University Press.
• Glatzer, Wolfgang (2002), Rich and Poor: Disparities, Perceptions, Concomitants (Social Indicators Research Series). Discusses measurement of inequality. Berlin: Springer.
• Martin, Elizabeth, Diana McDuffee, and Stanley Presser (1981). Sourcebook of Harris National Surveys: Repeated Questions, 1963-1976. Chapel Hill, NC: Institute for Research in the Social Sciences, UNC Press.
• Miller, Delbert C. and Neil J, Salkind (2002). Handbook of Research Design and Social Measurement, 6th Edition. Thousand Oaks, CA: Sage Publications.
• Mueller, Daniel J. (1986). Measuring Social Attitudes: A Handbook for Researchers and Practitioners. New York: Teachers College Press.
• Reed, David (1991). Density Measures and Their Relation to Urban Forms. Milwaukee, WI: University of Wisconsin Press, ISBN 0938744607.
• Reeder, L. G., Ramacher, L.. & Gorelnik, S. (1976). Handbook of scales and indices of health behavior. Pacific Palisades, CA: Goodyear.
• Robinson, John P., Robert Athanasiou, and Kendra B. Head (1969). Measures of Occupational Attitudes and Occupational Characteristics. Ann Arbor, MI: Institute for Social Research.
• Robinson, John P. and Phillip R. Shaver (1991). Measures of Personality and Social Psychological Attitudes . Academic Press, ISBN 0125902441.
• Robinson, John P. and Philip R. Shaver (1976). Measures of Social Psychological Attitudes. Institute for Social Research, University of Michigan.
• Ruggles, Patricia (1990). Drawing the Line : Alternative Poverty Measures and Their Implications for Public Policy. Urban Institute Press, ISBN: 0877664463.
• Tarkkonen, L. (1987). On reliability of composite scales. Statistical Research Reports 7. Finnish Statistical Society.
• Touliatos, John, Barry F. Perlmutter, Murray A. strauss, and George W. Holden (2001). Handbook of family measurement techniques, Vols. 1-3. Thousand Oaks, CA: Sage Publications. Covers over 1,300 instruments related to family studies.
• Tzeng, Oliver C. S. (1993). Measurement of love and intimate relations: Theories, scales, and applications for love development, maintenance, and dissolution. Westport, CT: Praeger Publishers.
• Vehkalahti, K. (2000). Reliability of measurement scales. Statistical Research Reports 17. Finnish Statistical Society. http://ethesis.helsinki.fi/julkaisut/val/tilas/vk/vehkalahti/.

  
  

Copyright 1998, 2008, 2011 by G. David Garson.
Last updated 12/20/2011.