Overview A study is valid if its measures actually measure what they claim to, and if there are no logical errors in drawing conclusions from the data. There are a great many labels for different types of validity, but they all have to do with threats and biases which would undermine the meaningfulness of research. Be less concerned about defining and differentiating the types of validity (researchers disagree on the definitions and types, and yes, they do overlap) and be more concerned about all the types of questions one should ask about the validity of research (researchers agree on the importance of the questions).

Contents Key concepts and terms Construct validity: convergent and discriminant Content validity Internal validity Statistical validity Bibliography

Key Concepts

1. Historical background: Some early writers simply equated validity with establishing that a construct's scale correlated with a dependent variable in the intended manner and, indeed, a scale might be considered valid as a measure of anything with which it correlated (Guilford 1946). Types of validity were codified in 1954 by the American Psychological Association, which identified four categories: content validity, construct validity, concurrent validity, and predictive validity (APA, 1954). Each type corresponded to a different research purpose: content validity had to do with subject-matter content testing, construct validity with measuring abstract concepts like IQ, concurrent validity with devising new scales or tests to replace existing ones, and predictive validity with devising indicators of future performance. A 1966 update to the APA typology combined the last two types under the label criterion-related validity (APA, 1966). Later, Sheperd (1993) was among those who argued that both criterion and content validity were subtypes of construct validity, leaving only one type of validity. This unified view of validity supported the notion that only rarely could a researcher establish validity with reference to a single earlier type. Moreover, Cronbach's (1971: 447) earlier argument that validity could not be established for a test or scale, only for interpretations researchers might make from a test or scale, also became widely accepted in the current era. Some, such as Messick (1989), accept construct validity as the only type, but argue for multiple standards for assessing it: relevant content, based on sound theory or rationale, internally consistent items, external correlation with related measures, generalizability across populations and time, and explicit in its social consequences (ex., racial bias). In a nutshell, over the last half century the concept of validation has evolved from establishing correlation with a dependent variable to the idea that researchers must validate each interpretation of each scale, test, or instrument measuring a construct and do so in multiple ways which only taken together form the whole of what validity is.

   The outline below largely accepts the unified view of validity, centering on construct validity, but adds to it separate coverage in three areas: (1) content validity, focusing on the labeling of constructs; (2) internal validity, focusing on research design bias; and (3) statistical validity, focusing on meeting assumptions of empirical procedures. While all three might be (and by some are) considered subtypes of construct validity, they do not fall neatly in its two major subdomains, convergent and discriminant validity, and so in the discussion below have been treated separately.

2. Construct validity, sometimes also called factorial validity, has to do with the logic of items which comprise measures of social concepts. A good construct has a theoretical basis which is translated through clear operational definitions involving measurable indicators. A poor construct may be characterized by lack of theoretical agreement on its content, or by flawed operationalization such that its indicators may be construed as measuring one thing by one researcher and another thing by another researcher. A construct is a way of defining something, and to the extent that a researcher's proposed construct is at odds with the existing literature on related hypothesized relationships using other measures, its construct validity is suspect. For this reason, the more a construct is used by researchers in more settings with outcomes consistent with theory, the more its construct validity. Researchers should establish both of the two main types of construct validity, convergent and discriminant, for their constructs.

   • Convergent validity is assessed by the correlation among items which make up the scale or instrument measuring a construct (internal consistency validity), by the correlation of the given scale with measures of the same construct using scales and instruments proposed by other researchers and, preferably, already accepted in the field (criterion validity), and by correlation of relationships involving the given scale across samples (ex., racial tolerance with subject data and with spousal data) or across methods (ex., survey data and archival data). One expects these correlations to be at least moderate to demonstrate external validity.

     • Internal consistency is a type of convergent validity which seeks to assure there is at least moderate correlation among the indicators for a concept. Poor convergent validity among the indicators for a construct may mean the model needs to have more factors.
       • Example. In their study of direct-to-consumer (DTC) advertising, Huh, Delorme, and Reid (2006) developed consumer attitude constructs, which they validated by reporting Cronbach's alpha levels of 0.87, 0.88, and 0.89, respectively.
                See Huh, J, Delorme, D. E., & Reid, L. N. (2006). Perceived third-person effects and consumer attitudes on prevetting and banning DTC advertising. Journal of Consumer Affairs 40(1): 90

       • Cronbach's alpha is commonly used to establish internal consistency construct validity, with .60 considered acceptable for exploratory purposes, .70 considered adequate for confirmatory purposes, and .80 considered good for confirmatory purposes.

       • Simple factor structure is another test of internal consistency, seeking to demonstrate for a valid scale that indicator items for a given construct load unambiguously on their own factor. This tests both convergent and discriminant validity, as discussed below.

       • Rasch models, also called one-parameter logistic models, are an internal consistency test used in item response theory for binary items (though polytomous Rasch models are available also). Rasch models, like Guttman scales, establish that items measuring a construct form an ordered relationship (see Rasch, 1960; Wright, 1977, 1996). Note that a set of items may have ordered internal consistency even though they do not highly correlate (additive internal consistency, such as tested by Cronbach's alpha or factor structure). Ordered internal consistency reflects a difficulty factor, whereby answering a more difficult item predicts responses on less difficult items but not vice versa.

         That is, when factor analysis is used to validate the inclusion of a set of indicator variables in the scale for a construct, the researcher is assuming a linear, additive model. Linearity is assumed as part of correlation, which is the basis for clustering indicator variables into factors. Additivity is also assumed, meaning that items will be judged to be internally consistent only if they are mutually highly correlated. However, items may lack high intercorrelation but have a strong ordered relationship (ex., a scale of math ability composed of items of ascending difficulty). For this reason, many researchers prefer to use a Rasch model to guide scale construction, in preference to additive models like Cronbach's alpha or factor analysis. For Rasch modeling in SPSS, see Tenvergert, Gillespie, & Kingma (1993).

       • AVE. Alternatively, and less commonly, Fornell and Larcker (1981) consider a construct to display convergent validity if average variance extracted (AVE) is at least .50 (that is, when variance explained by the construct is greater than measurement error). AVE is discussed below in the section on discriminant validity.

       • Common method variance is a type of spurious internal consistency which occurs when the apparent correlation among indicators or even constructs is due to their common source (spurious convergence). For instance, if the data source is self-reports, the correlation may be due the propensity of the subject to answer similarly to multiple items even when there is no true correlation of constructs. Common method variance is perhaps best assessed using structural equation modeling of error terms, discussed separately. Common method variance is sometimes also assessed by (1) factoring all indicators in the study to see if a single common factor emerges, indicative of common method variance; or (2) observing correlations between different indicators of the same construct using the same and different methods, with the expectation that these correlations will be high in the same method data and low in the cross-method data if common method variance is a problem.

     • Criterion validity, also called concurrent validity, has to do with the correlation between scale or instrument measurement items and known and accepted standard measures or criteria. Ideally these criteria are direct, objective measures of what is being measured (ex., how well is self-reported voting correlated with actual voting shown in voting records?). Where direct objective measures are unavailable, the criteria may be merely closely associated. The researcher is asking if proposed measures for a given concept exhibit generally the same direction and magnitude of correlation with other variables as do measures of that concept already accepted within the social science community. For instance, does a new measure of "alienation" exhibit the same general correlative behavior as established scales of social anomie? Are people who score high on a proposed scale of "Republican identifiers" in fact more likely to vote Republican and/or be registered as Republican? Do people who score high on an employment test also rate high on current evaluations of actual job performance by their superiors? (Note: while some authors contrast construct and criterion validity, this author - and many others - considers criterion validity to be an aspect of construct validity).
       • Predictive validity may be considered a variant on criterion validity, where the criterion is in the future. For instance, comparing if people who score high on an employment test also rate high on subsequent job performance evaluations would be an effort to establish predictive validity because the criterion is not concurrent with the test.

       • Example: An instrument measuring child-reported sexual abuse was analyzed for four different age groups of 103 Dutch children. There was only a weak correlation between the instrument and the outcome of the cases of child abuse (the objective criterion). The authors conclude that the results suggest that the instrument "cannot yet be used as a scientifically validated instrument for judging the truthfulness of allegations of child sexual abuse." Logic: A good instrument would identify valid cases of child abuse and for valid cases, the outcomes would tend to be different.
                See Lamers-Winkelman, F. & Heemstede, A.L. (1998). Statement validity analysis: Its application to a sample of Dutch children who may have been sexually abused. Journal of Aggression, Maltreatment & Trauma, 2(2): 59-81.

     • External validity has to do with possible bias in the process of generalizing conclusions from a sample to a population, to other subject populations, to other settings, and/or to other time periods. The questions raised are, "Are findings using the construct scale consistent across samples?" and "To what population does the researcher wish to generalize his/her conclusions, and is there something unique about the study sample's subjects, the place where they lived/worked, the setting in which they were involved, or the times of the study, which would prevent valid generalization?" Naturally, when a sample of observations is non-random in unknown ways, the likelihood of external validity is low, as in the case of convenience samples. All other things equal, different samples should generate similar relationships. When they do not, this indicates different samples are affected by significant variables not in the model and which differ in value across groups.
       • Example: In the NELS:88 study, indicators of student disability were obtained from students, parents, teachers, and school officials. Each indicator was worded differently, but all were intended to measure "student disability." Results of comparisons of these measures showed that very little overlap (well under 50%) in the population of students identified as disabled by these separate sources. Logic: Different groups should have identified largely the same students as disabled, but they did not. This might mean all the similar instruments were invalid. But Rossi et al chose to think it meant the different wording had a big effect and the instruments measured different constructs even though that was the opposite of their intent.
                 See Rossi, Robert, Jerald Herting, and Jean Wolman. 1997. Profiles of Students With Disabilities as Identified in NELS:88 (NCES 97-254). Washington, DC: U.S. Department of Education, National Center for Education Statistics.

       • Ecological validity. Not to be confused with the "ecological fallacy" discussed below under "statikstical validity," ecological validity has to do with whether or not subjects are studied in their natural envionments or, say, in a lab setting. Removing subjects from their envionment may lead subjects to display different behavior from their "true" behavior (ex., people may express different opinions about race in a lab setting compared to the opinions they express among friends). The greater the difference in cues and influences between the natural environment and the measurement setting, the greater the potential bias. Four strategies are open to the researcher: (a) showing ecological factors are not important; (b) replicating ecological factors in the measurement environment (ex., by priming questions in surveys); (c) taking measurement in the natural environment (where it is harder to control for external variables); or (d) acknowledging that lab behavior cannot be extrapolated to behavior in the environment, but justifying this as a method of understanding behavior when removed from environmental influences.

   • Discriminant validity, the second major type of construct validity, refers to the principle that the indicators for different constructs should not be so highly correlated as to lead one to conclude that they measure the same thing. This would happen if there is definitional overlap between constructs. Discriminant validity analysis refers to testing statistically whether two constructs differ (as opposed to testing convergent validity by measuring the internal consistency within one construct, as Cronbach's alpha does).
     • Correlational Methods. In constructing scales, researchers often reject an indicator if it correlates more highly with a construct different from the one which it was intended to measure. Some researchers use r = .85 as a rule-of-thumb cutoff for this assessment, fearing that correlations above this level signal definitional overlap of concepts. Other researchers use the criterion that two constructs differ if the correlations between a given scale and a specific criterion measure are greater in magnitude than the correlations between that same scale and criterion measures used for other unrelated scales. This is considered a less stringent test of discriminant validity. In a more stringent correlational method, the same logic is applied but the comparison is to correlations between the given scale and criterion measures used for related scales or subscales.
       • Example. For a population of 211 demented and 94 mentally handicapped patients, Dijkstra, Buist, and Dassen concluded that the low correlations between the Scale for Social Functioning (SSF) full scale score and the other tested scales (the BOSIP Behavior Observation Scale for Intramural Psychogeriatrics) affirm the discriminant validity of the SSF scale.
                Dijkstra, A., Buist, G., & Dassen, T. (1998). A criterion-related validity study of the nursing care dependency. International Journal of Nursing Studies 35: 163-170.

     • Factor Methods. Some researchers conclude that constructs are different if their respective indicators load most heavily on different factors in principal components factor analysis (see Straub, 1989). In one version of this approach, all items for all constructs are factored. In a more stringent version, indicator items for each pair of constructs are factored separately..

     • AVE Method. An alternative factor-based procedure for assessing discriminant validity is that proposed by Fornell and Larcker (1981). In this method, the researcher concludes that constructs are different if the average variance extracted (AVE) for one's constructs is greater than their shared variance. That is, the square root of the average variance extracted (AVE) for a given construct should be greater than the absolute value of the standardized correlation of the given construct with any other construct in the analysis. For standardized data, squared covariances are equivalent.

       • Computation. AVE is the variance in indicator items captured by a construct as a proportion of captured plus error variance. AVE is calculated as the sum of the squared standardized indicator item loadings on the factor representing the construct, divided by this sum plus the sum of indicator item error. Thus, let S1 = the sum of squared principal components analysis factor loadings of the indicator variables on the factor representing their construct. Let S2 = the quantity (1 - the squared loading) summed for all indicators. Then AVE = (S1)/( S1 + S2). AMOS, LISREL, and PLS all provide output for calculation of AVE, with PLS doing so in the most user-accessible way.

       • Presentation. In terms of presentation, it is customary to provide a matrix of squared covariances of each construct with each other construct, replacing the diagonal elements with the AVE for the column construct. If there is discriminant validity, then the diagonal element for a given column (construct) should be larger than any of the squared covariances in the column.or row in which it is found.

     • SEM Methods. Confirmatory factor analysis within structural equation modeling, discussed elsewhere, is a common method of assessing discriminant validity. If goodness of fit measures for the measurement model in SEM are adequate, the researcher concludes that the constructs in the model different.
       • Nested models. A more rigorous (and more widely accepted) SEM-based alternative approach to discriminant validity is to run the model unconstrained and also constraining the correlation between constructs to 1.0. If the two models do not differ significantly on a chi-square difference test, the researcher fails to conclude that the constructs differ (see Bagozzi et al., 1991). In this procedure, if there are more than two constructs, one must employ a similar analysis on each pair of constructs, constraining the constructs to be perfectly correlated and then freeing the constraints. This method is considered more rigorous than either the SEM measurement model approach or the AVE method.
         • Example. In a study of industrial relations, Deery, Erwin, & Iverson (1999) wrote, "The discriminant validity was tested by calculating the difference between one model, which allowed the correlation between the constructs (with multiple indicators) to be constrained to unity (i.e., perfectly correlated), and another model, which allowed the correlations to be free. This was carried out for one pair of constructs at a time. For example, in testing organizational commitment and union loyalty, the chi-square difference test between the two models (p<.001) affirmed the discriminant validity of the constructs. "
                  See Deery, S., Erwin, P., & Iverson, R. (1999). Industrial relations climate, attendance behaviour, and the role of trade unions. British Journal of Industrial Relations 37(4): 533-558.

     • Warning about rate variables. A common violation of discriminant validity is the use of rate variables with the same denominator as both predictor and dependent (for example, police budget per 10,000 population as a predictor of property crimes per 10,000 population). See Bollen & Ward (1979), who warned that such a design confounded the predictor effect with the effect of the common denominator variable. Rates may, however, be used as a control method when the same denominator is only on one side of the equation. This is discussed further in an example in the partial correlation section.

   • Multi-method, multi-trait methodologies are considered to have higher construct validity.
     • Mono-method and/or mono-trait biases. Use of a single data-gathering method and/or a single indicator for a concept may result in bias. Various data-gathering methods have their associated biases (ex., the yea-saying bias in survey research, where people tell pollsters what they think they want to hear). In the same vein, has the researcher used randomization of items to eliminate order effects of the instrument, or established the unimportance of order effects? Likewise, basing a construct like "work satisfaction" on a single item dealing with, say, socializing with peers at work, biases the construct toward a particularistic meaning.

     • Multitrait-multimethod validation. In a multi-method, multi-trait validation strategy, the researcher not only uses multiple indicators per concept, but also gathers data for each indicator by multiple methods and/or from multiple sources. For instance, in assessing the concept of "tolerance," the researcher may have indicators for racial tolerance, religious tolerance, and sexual orientation tolerance; and each may be gathered from the subject, the subject's spouse (assessing tolerance indicators for subject, not spouse), and the subject's parent (assessing tolerance indicators for subject, not parent).
       1. A correlation matrix is created in which both rows and columns reflect the set of three tolerance indicators, grouped in three sets -- once for subject data, once for spousal data, and once for parental data.

       2. Cross-validation. Has the researcher made efforts to cross-validate subjective items with objective measures where possible? Has the researcher developed the instrument on a calibration sample and then cross-validated it on an independent validation sample?

3. Content validity, also called face validity, has to do with items seeming to measure what they claim to (studies can be internally valid and statistically valid, yet use measures lacking face validity). In content validity one is also concerned with whether the items measure the full domain implied by their label. Though derogated by some psychometricians as too subjective, failure of the researcher to establish credible content validity may easily lead to rejection of his or her findings. Use of surveys of panels of content experts or focus groups of representative subjects are ways in which content validity may be established, albeit using subjective judgments.

   Are the measures which operationalize concepts ones which seem by common sense to have to do with the concept? Or could there be a naming fallacy? Indicators may display construct validity, yet the label attached to the concept may be inappropriate.

   • Are the labels attached to constructs too broad in domain? For instance, in a small group study of corruption, "monetary incentives" may be used to induce participants to covertly break the rules of a game. The researcher may find "monetary incentives" do not lead to corruption, but what may be involved would be better labeled "small monetary incentives," whereas "large monetary incentives" may have a very different effect. Likewise, a scale may be labeled "liberalism" but the items may only deal with cultural issues like abortion and gay rights, but lack any content on economic or environmental issues, and thus might be better labeled "cultural liberalism."

   • Example: Stratford and Kennedy (2004) noted this about a measure of functional status of lower extremities in a medical setting: "Clearly, a lower extremity functional status measure that does not overtly inquire about ambulation lacks content validity. For this reason, we caution against using the DISSIMILAR-8 as an outcome measure for clinical trials and as the basis for decisions in clinical practice."
            See Stratford, P. W. & Kennedy, D. M. (2004). Does parallel item content on WOMAC's Pain and Function Subscales limit its ability to detect change in functional status? BMC Musculoskeletal Disorders. 2004; 5: 17.

4. Internal validity has to do with defending against sources of bias arising in research design, which would affect the cause-effect process being studied by introducing covert variables. When there is lack of internal validity, variables other than the independent(s) being studied may be responsible for part or all of the observed effect on the dependent variable(s). If there is no causal phenomenon under study, internal validity is not at issue.
   • Hawthorne effect (experimenter expectation). Do the expectations or actions of the investigator contaminate the outcomes? (Named after famous studies at Western Electric's Hawthorn plant, where work productivity improvements were found to reflect researcher attention, not interventions like better lighting).

   • Mortality bias. Is there an attrition bias such that subjects later in the research process are no longer representative of the larger initial group?

   • Selection bias. How closely do the subjects approach consituting a random sample, in which every person in the population of interest has an equal chance of being selected? When multiple groups are being studied, there can be differential selection of the groups which can be associated with differential biases with regard to history, maturation, testing, mortality, regression, and instrumentation (that is, selection may combine differentially with other threats to validity mentioned on this page). See section on two-stage least squares regression for a discussion of testing for selection bias.

   • Evaluation apprehension. Does the sponsorship, letter of entry, phrasing of the questions, or other steps taken by the researcher suffice to mitigate the natural apprehension people have about evaluations of their beliefs and activities, and diminish the tendency to give answers which are designed to make themselves "look good"?

   • Special problems involving control groups (social interaction threats to validity):

     • Compensatory equalization of treatments. Were those administering the setting pressured, or did they decide on their own, to compensate the control group's lack of the benefits of treatment by providing some other benefit for the control group? Parents may pressure school administrators, for instance, to provide alternative learning experiences to compensate for their children in the control group not receiving the special test curriculum being studied in the experimental group.

     • Compensatory rivalry occurs when one group, typically the comparison or control group, learns of differences with the other group, typically the treatment group. Awareness of differences may create feelings of relative deprivation and may promote competitive attitudes which may bias the study. Compensatory rivalry has unpredictable outcomes and may lead on the one hand to feelings the group should work harder to compete with the other group, or that the group should give up trying in light of the known differences.
       • Resentful demoralization is a type of compensatory rivalry where he comparison group learns of differences with the treatment group and becomes discouraged (and sometimes angry), leading to psychological or actual withdrawal from the study.

       • Treatment imitation or diffusion is also a type of control awareness invalidity, arising from the control group imitating the treatment or benefitting from information given to the treatment group and diffused to the control group.

     • Unintended treatments. The Hawthorne effect (see above) is an example, where the experimental group was also receiving the unmeasured "treatment" of researcher attention. However, either the experimental or control group may receive different experiences which constitute unmeasured variables.

   • Special problems of before-after studies and time series:
     • Instrumentation change. Variables are not measured in the same way in the before and after studies. A common way for this to occur is when the observer/raters, through experience, become more adept at measurement.

     • History (intervening events) . Events not part of the study intervene between the before and after studies and have an effect. Did some historical event occur which would affect results? For instance, outbreak of a war often solidifies public opinion behind the Commander-in-Chief and could be expected to affect a study of causes of changes in presidential support in public opinion polls, even if the items had nothing to do with foreign policy.

     • Maturation. Invalid inferences may be made when the maturation of the subjects between the before and after studies has an effect (ex., the effect of experience), but maturation has not been included as an explicit variable in the study.

     • Mortality. Subjects may drop out of the subject pool for a variety of reasons, making the "before" and "after" samples non-comparable.

     • Regression toward the mean. If subjects are chosen because they are above or below the mean, one would expect they will be closer to the mean on remeasurement, regardless of the intervention. For instance, if subjects are sorted by skill and then administered a skill test, the high and low skill groups will probabily be closer to the mean than expected.

     • Test experience. The before study impacts the after study in its own right, or multiple measurement of a concept leads to familiarity with the items and hence a history or fatigue effect.

5. Statistical validity has to do with basing conclusions on proper use of statistics. Violation of statistical assumptions is treated elsewhere, in the discussion of each specific statistical procedure. In addition, the following general questions may be asked of any study:
   • Reliability. Has the research established the statistical reliability of his/her measures? (A measure is reliable if measurement of the same phenomena at different times and places yields the same measurement.) Reliability is discussed in a separate section.

   • Type I Errors and Statistical Significance. A Type I error is when the researcher thinks there is a relationship, but there really isn't. If the researcher rejects the null hypothesis because p<=.05. leading to the conclusion there is a relationship, ask these questions:

     • If data are from a random sample, is significance established to be of an appropriate level (in social science, usually .05 for confirmatory analysis but often only .10 for exploratory analysis)?

     • Are significance tests applied to a priori hypotheses, or is a shotgun approach used in which large numbers of relationships are examined, looking a posteriori for significant ones? If the latter, note that one table or relationship in 20 will be found to be statistically significant just by chance alone, by definition of .05 significance. Multiple a posteriori tests require a higher operational alpha significance level to achieve the same nominal alpha level; this is achieved by using a Bonferroni adjustment, discussed elsewhere.

   • Type II Errors and Statistical Power. A Type II error is when the researcher thinks there is no relationship, but there really is. If the researcher has accepted the null hypothesis because p>.05, leading to the conclusion there is no relationship, ask these questions:

     • Has the researcher used statistical procedures of adequate power?

     • Does failure to reject the null hypothesis merely reflect small sample size?

   • Fallacies of aggregation. Discussed by the statistician G. Udny Yule early in the 20th century, called the "ecological falacy" by Robinson (1950), and "Simpson's Paradox" based on 1951 work by Edward Simpson, the ecological fallacy is assuming that individual-level correlations are the same as aggregate-level correlations. Robinson showed that individual level correlations may be larger, smaller, or even reverse in sign compared to aggregate level correlations. For instance, at the state level there is a strong correlation of race and illiteracy, but this largely disappears at the individual level. The reason is that many African Americans live in the South, which also has high illiteracy, for whites as well as blacks. More generally, what is true at one level of aggregation (ex., states) need not be true at another level (ex., nations). Various methods have been proposed for dealing with ecological inference, but there is no easy solution.
     • Strategies. There is no good "solution" to the problems posed by the ecological fallacy. The monograph by Langbein and Lichtman (1978) addressed this long ago. They debunked a number of purported procedures, such as "homogenous grouping" (restricting analysis to aggregate units which do not vary on variables of interest, as by looking at counties which are very high in Catholicism and counties very high in Protestantism to assess the individual-level voting behaviors or Catholics and Protestants), which they note has "disabling defects" (p. 39). Langbein and Lichtman found the best of bad alternatives to be "ecological regression" (pp. 50-60), a technique which involves using regression to model the effects of grouping. Ecological regression, however, has often been criticized as vulnerable to statistical bias. Ecological regression fails whenever factors affecting the dependent variable covary with the independent variable(s). Epstein and O"Halloran (1997) give a good critique of ecological regression, noting, "For instance," (in a study in which the dependent is votes for the minority candidate),"say that voter registration rates are not constant across districts, but rather minority registration rates increase as the percent of Black voters in a district rises. In such a case, ecological regression would attribute the rise in the number of votes for the minority candidate to polarized voting, when in fact it is partly due to increased minority registration." (p. 4). Epstein and O'Halloran propose logit and probit techniques instead of ecological regression. See Achen and Shively (1995) and King (1997).

   • Interaction and non-linearity. Has the researcher taken possible interaction effects and nonlinear effects into account? Is there interaction among multiple treatments?

   • Causal ambiguity. Has the researcher misinterpreted the causal direction of relationships, particularly in correlative studies?

   
   

Bibliography

• Adcock, R. and D. Collier (2001). Measurement validity: A shared standard for qualitative and quantitative research. American Political Science Review 95: 529-546.
• APA (1954). Technical recommendations for psychological tests and diagnostic techniques. Psychological Bulletin, 51(2, supplement): 201-238.
• APA (1966). Standards for educational and psychological tests and manuals. Washington, DC: American Psychological Association.
• Bagozzi, R. P., Y. Yi and L. W. Phillips (1991). Assessing Construct Validity in Organizational Research. Administrative Science Quarterly 36(3): 421-458.
• Bollen, Kenneth A. & Ward, Sally (1979). Ratio variables in aggregate data analysis: Their uses, problems, and alternatives. Sociological Methods & Research 7(4): 431-450.
• Campbell, Donald T. & J. C. Stanley (1963a). Experimental and quasi-experimental designs for research on teaching. In N. L. Gage, ed., Handbook of research on teaching. Chicago: Rand McNally, 1963: 171-246 . The seminal article on types of validity.
• Campbell, Donald T. & Stanley, J. C. (1963b). Experimental and quasi- experimental designs for research. Chicago: Rand-McNally.
• Carmines, E. G. & Zeller, R. A. (1979). Reliability and validity assessment. Quantitative Applications in the Social Sciences series 07-017. Newbury Park, CA: Sage Publications.
• Cronbach, L. J. (1971). Test validation. In R.L.Thorndike (ed.). Educational measurement. Second ed.. Washington, D.C.:American Council on Education.
• Cronbach, L.J. & Meehl, P.E. (1955). Construct validity in psychological tests. Psychological Bulletin 52: 281-302.
• Cook, Thomas D. & Campbell, Donald T. (1979). Quasi-experimentation: Design and analysis issues for field settings. Boston: Houghton-Mifflin. Chapter 2 is a classic statement of types of validity.
• Fornell, C., & Larcker, F. D. (1981). Evaluating structural equation models with unobservable variables and measurement error. Journal of Marketing Research, 18(1): 39-50.
• Guilford, J. P. (1946). New standards for test evaluation. Educational and Psychological Measurement, 6(5): 427-439.
• Messick, S. (1989). Validity. Pp. 13-103 in R. Linn, ed., Educational measurement. Third ed.New York: American Council on Education and Macmillan Publishing Company.
• Rasch, G. (1960). Probabilistic models for some intelligence and achievement tests. Copenhagen: Danish Institute for Educational Research (Expanded edition, 1980. Chicago: University of Chicago Press).
• Shadish, W., Cook, T., & Campbell, D. (2002). Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Boston:Houghton Mifflin.
• Shepard, L. A. (1993). Evaluating test validity. In L. Darling-Hammond, ed., Review of research in education, Vol. 19, pp. 405-450. Washington, DC: American Educational Research Association.
• Straub, D.W. (1989). "Validating Instruments in MIS Research," MIS Quarterly, 13(2): 147-166.
• Tenvergert, E., Gillespie, M., & Kingma, J. (1993). Testing the assumptions and interpreting the results of the Rasch model using log-linear procedures in SPSS. Behavior Research Methods, Instruments & Computers 25(3), 350-359.
• Wright B. D. (1977). Solving measurement problems with the Rasch model. Journal of Educational Measurement 14, 97–116.
• Wright, B. D. (1996) Comparing Rasch measurement with factor analysis Structural Equation Modeling 3, 3-24.

Copyright 1998, 2008, 2009 by G. David Garson.
Last update, 4/24/2009.