Social Judgment Theory

Overview

Social judgment theory is both a theoretical framework for analysis of decisions and also a methodology for data collection and analysis. It is useful for identifying clusters of decision-makers who make judgment on a similar basis - that is, using similar cues. Alternatively, it can be used to identify clusters of decision scenarios which are similar for a set of decision-makers. Social judgment analysis is also useful for assessing the relative importance of different cues to different groups of decision-makers or for evaluating the likelihood of acceptance of messages in a campaign (a set of messages aimed at persuasion).

Key Concepts and Terms

• Social judgments. Sherif and his colleagues (Sherif, 1936; Sherif & Hovland, 1961) proposed long ago that to understand acceptance of a message one had to look less at the content of the message than at the social judgment of the message by the receiver. That is, acceptance of a message is primarily based on subjective, not objective factors.

• Behavioral episodes. As framed by Egon Brunswik (1956), a behavioral episode is the making of a judgment in the process of adaptation to the environment.

• Proximal cues and the distal state. The distal state is the goal of the decision-maker. As such the distal state represents the decision-maker's decision-making policy (ex., maximize equity, minimize cost, or maximize political payoffs). Proximal cues are information probabilistically relating judgments to the distal state. "Probabilistically" means that some cues are more reliable than others. Some may be so low in reliability that they are unnecessary. The salience of individual cues is particular to the judgment context and also varies by decision-maker. For example, in Willoughby & Finn's (1996) study of legislative budget analysts, cues included universalistic/rational cues (ex., agency performance measures), heuristic cues (ex., incremental budget change from last year), and particularistic cues (ex., political factors related to the agency). Cues may also include any trait associated with the decision-maker (ex., level of professional training), the object of judgment (ex., governance structure of the agency, where judgments are made about agency budget requests), or the environment in which both decision-maker and object of judgment exist (ex., fiscal austerity vs. budget surplus in the fiscal climate of the state).

• Latitudes and zones. Social judgment theorists believe individual receivers of messages are mischaracterized if seen as having a single agreement position (Sherif, Sherif, & Nebergall, 1965; O'Keefe, 1990). Rather, for any message an individual can be characterized by three zones: a latitude of acceptance, a latitude of noncommitment, and a latitude of rejection. The "zone of ambiguity" corresponds to the latitude of non-commitment and highlights the fact that the relation of cues to the distal state is probabilistic. The weight given by a decision-maker to a cue reflects a judgment about the high or low probability that the cue is related to a judgment tied to the desired distal state. That is, cue weights assess the zone of ambiguity.

• Researching latitudes. One form of social judgment analysis focuses on defining lattitudes of acceptance for a message (ex., Smith et al., 2006). Subjects are presented with a series of statements, each of which they may rate as acceptable, unacceptable, or neither acceptable nor unacceptable. For instance, in a college setting, a statement may be "About 50% of students on this campus have smoked marijuana." Alternative statements may substitute different percentages. Analysis of such survey data allows researchers to conclude, for example, that a 40% to 70% marijuana use rate is in the latitude of acceptance for the modal student, and percentages greater than 90% or less than 20% are in the latitude of rejection, and percentages in between are in the latitude of noncommitment.
  • Assimilation vs. contrast effects. Social judgment theory predicts that when a set of messages is presented, as in an anti-drug campaign, the inclusion of one or more messages in the latitude of rejection of the audience will make audience members perceive themselves as more distant in objective position from the campaign position than they are (the contrast effect) Conversely, restricting campaign messages to only those in the latitude of acceptance will lead a subject to perceiving the campaign to be more similar to the subject's own position than the campaign may be (the assimilation effect).

  • Anchor effects: The classic Hovland-Sherif experiment. Hovland and Sherif asked subjects to rate their peceptions of weight of a set of objects. When subjects first lifted a heavy object (the "anchor" stimulus), most other objects were rated in a lower weight classes than objectively the case (the contrast effect). When subjects first lifted an object equal to the top of the range of other weights. most other objects were rated in higher weight classes than objectively the case (the assimilation effect). A control group had no anchor stimulus and rated the opbjects evenly across the six possible weight classes. Sherif & Hovland (1961) concluded that when an anchor stimulus is far from the range of possible judgments, there is a contrast effect, but when the anchor is close to this range, there is an assimilation effect.

• Researching cues. Social judgment analysis may present subjects, who represent decision-makers, with decision tasks and related cues for a set of objects (decision scenarios). For instance, subjects may be asked to make budget allocation decisions for a series of agencies, each with a different set of cues (ex., request amount, last year's actual budget, performance data, relation to the governor's political agenda, etc.). The relative weights subjects give to cues are measured for each decision task and consistency across multiple decision tasks is also measured, as discussed below. A variety of quantitative methods may be applied to these data:
  • Cluster analysis, factor analysis, or Q-mode factor analysis may be used to cluster subjects by cue-use patterns, to create profiles of decision-makers (ex., rationalists, incrementalists, politicos, bureaucrats in Willoughby & Finn, 1996). That is, each decision-maker profile type can be associated with use of cues which heavily load on a factor. It is possible to factor or cluster by subject (which sets of decision-makers are similar in use of cues), by cue (which sets of cues are similar for the set of subjects), or by judgment (which sets of agencies are similar, in a study where the decision-makers make budget allocation decisions for a series of hypothetical agencies, each with different sets of cues).

  • Multiple regression may be used to establish the relative weights of cues by profile or overall. Using cues to predict outcome (ex., dollar allocation) for the set of decision objects (ex., set of agencies), gives one regression per judge. The beta weights reflect the relative importance of the cues for a given judge across all objects (agencies). That is, the beta weights for the cues as predictors of outcome define cue weights for each judge. The beta weights can be more easily interpreted if each beta weight is divided by the sum of all beta weights for a given judge. This creates relative weights of cues, summing to 100, for each judge. If a relative weight is, say .3, this is interpreted as meaning that 30% of the variability of the given subject's judgments is explained by his or her consideration of the given cue.

    Note that it would also be possible to use cues to predict outcome (dollar allocation) for the set of judges, giving one regression per agency. The beta weights would then reflect the relative importance of the cues for a given object (agency decision scenario) across all judges. This would be appropriate if the object of research were to cluster types of objects (decision scenarios) rather than to cluster types of subjects (decision-makers).

    • Functional form. The regression may be linear, but it is also possible to add quadratic terms (cue values squared) to the regression model if it is assumed that the subject utilizes the cue on an exponential utility curve (cues perceived more useful are employed to an exponential degree).

  • Goodness of fit. The predictions from regression (ex., predicted agency budget allocations based on the cue weights assigned by a given decision-maker) can be correlated with the actual decision outcome (ex., the dollar amount allocated to the agency by the decision-maker). This R² is a measure (1) of model fit (cue weights predict actual outcomes when R² is high for all judges), and (2) rater consistency (a high R² for a given rater (subject) reflects consistency of that rater across all objects). Stewart (1988) calls this R² the predictability score and considers .80 or higher to be "good." See Willoughby & Finn (1996: 533).

  • Function forms, not to be confused with the functional form of the regression, are a graphical way of presenting findings about individual subjects or groups of subjects (where groups are determined by cluster analysis, for instance). The function form graph's y axis is the dependent variable (ex., dollar allocation given by the judge to the agency). The dependent variable may be rescaled from low to high so that the 0 mark and the 100 mark are the same for all judges (or one might rescale from -100 to +100, if cues assume negative values) The graph's x axis is the cue score (the level of the cue as presented to the subject). Thus there is one function form for each subject-cue combination. Each form reflects data on all the objects. For instance, the first function form would be for the first subject (Analyst 1) and the first cue (ex., salience of governor's agenda, GA). Let there be 50 objects (agency decision scenarios). The GA level was assigned by a random number process for each agency, creating 50 GA values which are arranged from low to high on the X axis. For each position on the X axis the object has a corresponding dollars allocated position on the Y axis. The X.Y points are plotted on the function form, as illustrated below.
    

    In the figure above, Analyst 1 is shown to allocate dollars to agencies exponentially as the GA cue has higher values. Note some authors, such as Willoughby & Finn (1996), omit the data points and just show the fit line on function forms, to simplify analytic comparisons.

    As mentioned, there can be one such function form for each analyst for each cue, enabling the researcher on a graphical basis to compare the decision-making profiles of different analysts. One can also group analysts by using cluster analysis, then create group-level function forms in order to be able to compare group profiles. This is done by computing the dependent mean by group (ex., mean dollars allocated for each of 6 subgroups of analysts identified by cluster analysis), then using cue levels to predict the agency means using as subjects the members of each group in turn (for illustration, see Willoughby & Finn, 1996: 537, 539).

Assumptions

• Specification bias. Though not necessarily inherent in social judgment theory, there can be an assumption that the discretion of the decision-maker in selecting what weight to give various cues is large and, moreover, is restricted to the measured cues. That is, it may be assumed that decisions are not heavily constrained by unmeasured factors not pertaining to subject traits or measured cues. Put another way, model specification is a major issue, as it is in other forms of modeling. To guard against specification bias, Willoughby & Finn (1996), for instance, are careful to survey the literature thoroughly with a view to including in their model a comprehensive set of all decision-making cues mentioned in the literature as being salient.

• Object sampling bias. The core of social judgment analysis methodology is the presentation of decision objects to judges. For instance, objects may be budget requests from hypothetical agencies, each with a set of associated cues. The number of objects must be large enough to support all combinations of all levels of cues, or at least a representative sample of all combinations. The Willoughby & Finn (1996) study, for instance, employed 40 objects (hypothetical agencies) for 7 cues, generating the cue levels for each of the 40 agencies by a random process.

• Subject sampling bias. Because social judgment analysis requires and experiment-like level of commitment, with the subject making judgments for many objects, it can be difficult to get participation of a representative sample of actual decision-makers of interest. The quasi-experiment may be undertaken using more available subjects, such as students, which raises questions of bias if results are then generalized to actual decision-makers. Indeed, even if actual decision-makers are subjects, questions can be raised as to whether judgments on hypothetical objects are the same as judgments on corresponding real objects (ex., hypothetical budget requests by hypothetical agencies versus actual requests by actual agencies). While Stewart (1988: 58) notes that "the goal of judgment analysis in the context of SJT (social judgment theory) is to derive a useful description of the judgment process and not necessarily to reproduce faithfully all the properties of the process itself," the utility of SJT findings ultimately must rest on the belief they are pertinent to real-world contexts. For this reason, findings from social judgment analysis benefit from an attempt to validate them through some process of triangulation, such as use of case studies and focus groups to corroborate social judgment analysis conclusions.

Illustrative Hypotheses

• Patterns of cue use will vary by the judge's socialization. (ex., political appointees will use political cues more, appointees from academia may use performance cues more, etc.).
• Inclusion of a message in the subject's latitude of rejection as part of a campaign of messages will lead the subject to perceive him- or herself as more distant from the campaign than comparison of objective position information would predict.
• Restriction of messages in a campaign to those in a subject's latitude of acceptance will lead the subject to perceive him- or herself as closer to he campaign than comparison of objective position information would predict.

Bibliography

• Adelman, Leonard; Stewart, Thomas R.; & Hammond, Kenneth R. (1975). A case history of the application of social judgment theory to policy formulation." Policy Sciences 6:137-59.
• Al-Tabtabai, Hashem & Diekmann, James E. (1992). Judgmental forecasting in construction projects. Construction Management and Economics. 10: 19-30.
• Brunswik, Egon (1956). Perception and the representative design of experiments. Berkeley, CA: University of California Press. The seminal work in social judgment theory.
• Dell'Omo, Gregory G. (1990). Capturing arbitrator decision policies under a public sector interest arbitration statute. Review of Public Personnel Administration. 10: 19-38.
• Dhir, Krishna S. (1987). Formulating management policies for value engineering/value analysis. IEEE Transactions on Engineering Management. 34:161-71.
• Grizzle, Gloria (1985). Performance measures for budget justifications: Developing a selection strategy. Public Productivity Review 9:328-41.
• Hammond, Kenneth R.; Hamm, Robert M.; Grassia, Janet; & Pearson, Tamra (1987). Direct comparison of the efficacy of intuititve and analytical cognition in expert judgment. IEEE Transactions on Systems, Man, and Cybernetics 17:753-70.
• O'Keefe, D. J. ( 1990). Persuasion: Theory and research. Newbury Park, CA: Sage.
• Sherif, C. W.; Sherif, M.; & Nebergall, R. E. ( 1965). Attitudes and attitude change: The social judgment-involvement approach. Philadelphia: W. B. Saunders.
• Sherif, M. ( 1936). The psychology of social norms. New York: Harper.
• Sherif, M., & Hovland, C. I. ( 1961). Social judgment: Assimilation and contrast effects in communication and attitude change. New Haven, CT: Yale University Press
• Smith, Sandi W.;Atkin, Charles K. ; Martell, Dennis ; Allen, Rebecca ; & Hembroff, Larry (2006). A social judgment theory approach to conducting formative research in a social norms campaign. Communication Theory 16(1), 141–152.
• Stewart, Thomas R. & Gelberd, Linda (1976). Analysis of judgment policy: A new approach for citizen participation in planning. American Institute of Planners Journal. (Jan.):33-41.
• Waller, William S. (1988). Brunswikian research in accounting and audting. In Berndt Brehmer & C.R.B. Joyce, eds. Human Judgment: The SJT View. Holland-Elsevier, 247-72.
• Whorton, Joseph W. Jr.; Feldt, James A.; and Dunn, Delmer D. (1989). Exploring the values underlying evaluation of research: A social judgment analysis. Knowledge in Society: International Journal of Knowledge Transfer 1:40-55.
• Willoughby, Katherine G. (1993). Decision making orientations of state government budget analysts: Rationalists or incrementalists? Public Budgeting and Financial Management 5:67-114.
• Willoughby, Katherine G. & Finn, Mary A. (1996). Decision strategies of the legislative budget analyst: Economist or politician?. Journal of Public Administration Research and Theory. 6(4), 523-546. An application of social judgment theory.