Figure 2From “What is measurement validity?”
Kinds of validity evidence, and what each cannot show alone
Each kind of evidence answers one question. None of them, alone, rules out every other reading of the score.
Figure 2Kinds of validity evidence, and what each cannot show alone
| Evidence | The question it answers | What it cannot show alone |
|---|---|---|
| Content | Do the items or inputs cover the defined domain? | That the score moves with the attribute, rather than with how the items were chosen |
| Response process | Does the attribute act on the score through a known process? | That the stated process is the one at work, until it is tested against the responses |
| Criterion: predictive or concurrent | Does the score correlate with a criterion measured later or at the same time? | That it measures the attribute, rather than something else that also predicts the criterion |
| Convergent and discriminant | Does it agree with measures of the same construct, and differ from measures of others? | That a shared method is not producing the agreement, unless traits are crossed with methods |
| Known groups or manipulation | Do groups known to differ on the attribute score differently, and does a deliberate change in the attribute move the score? | For known groups, that the attribute and not another difference between the groups moves the score; for a manipulation, that it changed the attribute and nothing else |
| Nomological | Do its relations with other constructs match the theory? | That no spurious cause is at work: method bias stays open without procedural or statistical controls, and other rival constructs stay open because the data are correlational |
| Consequences | Do the intended and unintended consequences of a use support it, and do adverse effects trace back to a source of invalidity? | Validity itself, on the view of Borsboom et al., who exclude consequences from it |
Swipe or scroll horizontally if the table is wider than your screen.
Source: Kinds and questions from Cronbach and Meehl (1955), pp. 281-282; Messick (1995), pp. 742, 744, 745, 746, 748; MacKenzie, Podsakoff and Podsakoff (2011), pp. 296, 317-320; Borsboom, Mellenbergh and van Heerden (2004), pp. 1061, 1067, 1069. The third column is the author's summary, drawing on the limits these sources name.
From the essay
From the research bench
What is measurement validity?
Measurement validity is whether a score measures what a decision treats it as measuring. The classics disagree on what is valid: the test or its reading.
Read the essayThis exhibit belongs to an essay: the numbers live here, the argument lives there.
Cite or embed
Contact
Bring the question you're working on.
Research, a guest lecture, or a commercial question.