The terms reliable and valid are common in our everyday language. For example, we may say that the school buses are reliable or that a student has a valid reason for missing their assignment. This ubiquity can sometimes lead to misinterpretation when the term is being applied to specific contexts. In this blog, we will focus on reliability and validity as they relate to assessment.
As we apply the terms reliability and validity to assessment, the meaning shifts slightly and the interdependence between the two becomes paramount. For assessments to be useful to practitioners, the instrument must be both reliable and valid. Although these two concepts seem intuitively simple, there are entire books devoted to the subject. The purpose of this blog is to introduce the concepts of reliability and validity as they pertain to assessment and what implications they present for educators and administrators that rely on the information assessments can provide.
When the two terms are discussed together, it is common to speak of them as validity then reliability (e.g., “the test is valid and reliable”—note the order). However, for the purpose of this blog, I would like to flip this script for reasons I will address later. No disrespect to validity or the second half of any other famous tandems like Lou Costello, Oliver Hardy, or Jordan Peele.
What is reliability in assessment?
Reliability as it pertains to assessment essentially refers to the consistency with which the assessment provides results. According to the Standards for Educational and Psychological Testing (2014), reliability refers to “consistency of scores across replications of a testing procedure” (p. 33).
Here we are not speaking to how well the measure captures our construct but rather how consistently it provides this information. This consistency is expected when the test is given on proximal testing dates. For example, if a test tells us that a student has mastered content expected in third grade, we would not expect a different outcome if we measured it again the next day. Even when a test is highly reliable, there are factors that can affect the consistency of the scores, such as fatigue, mood, illness, or variations in test settings like distractions. When understanding how students are performing on any assessment, it is important to consider potential impacts like those.
Types of reliability in assessment
There are variations of reliability that can fall into several principal types. All of these deal with an aspect of consistency but are measured in different ways.
- Test-retest reliability is the most thought about measure of a test’s consistency. This form of reliability measures how consistently the test measures a construct over multiple administrations, assuming they are reasonably proximal where changes in the construct are not expected.
Test-retest is typically measured by administering the same test to the same group of people over a short duration where no change in performance should be expected. The reliability between the two is measured by a correlation between the two sets of scores.
- Parallel forms reliability measures the consistency of two forms or versions of the same test and their ability to yield the same results.
Parallel forms are measured similarly to test-retest where the same group takes the test twice. However, in parallel forms, the second test is not identical to the first; rather, it is a parallel form with the assumption it should behave similarly to the initial form. The reliability between the two is once again measured by a correlation between the two sets of scores.
- Inter-rater reliability refers to the consistency of scoring the performance of a student when the test is scored by different people. If the raters are properly trained, the test and associated scoring rubrics should yield similar conclusions regarding student performance. These variations are normal and tolerable to a certain degree, but excessive variations can call into question the ambiguity of the rubric or the assessment itself.
Inter-rater reliability compares the scores from multiple raters on the performance of students on an assessment. These are particularly important in rubric scoring where there are multiple scorers. The reliability represents the level of agreement obtained between the two scorers. If there are significant differences, the issue may reside in the clarity of the rubric, training of the scorers, or integrity of the assessment itself.
- Internal consistency is a measure of the stability of the items purportedly measuring the same construct on the test.
Measurement of internal consistency verifies that items placed together in the same assessment measure the same construct. To test this quantifiably, a researcher may also perform a split half test where half the items are compared to the other half within the same test. The reliability between the two is measured by a correlation between the two sets of items.
What is validity in assessment?
Validity is often mentioned in the same conversation with reliability because the two are essential for the utility of assessing students.
Validity essentially describes how accurately a test measures what it is intended to measure. According to the Standards for Educational and Psychological Testing (2014), validity as it pertains to assessment is defined as “the degree to which evidence and theory support the interpretations of test scores for proposed uses of tests” (p. 11). Inherent in this definition is the idea that tests themselves are not valid or invalid but rather validity lies in the interpretation of test scores and how they are utilized. For example, you may have a valid test for fractions, but the same test may be invalid if used for geometry. Same test, different stated purpose.
Therefore, validity starts with an understanding of what the test provides in terms of reporting and how those results are interpreted. Before we can ascertain whether an assessment is valid, we must first understand the intended use of the assessment. Any violations from that intended use represent threats to the assessment’s validity. Validity is also typically not a one and done treatment but rather a collection of evidence that supports an assessment’s intended use.
Types of validity in assessment
Similar to reliability, within this global definition of validity, there are several variations that are important to understand. Each of these variations are also measured in different ways.
- Construct validity indicates the degree to which the test measures the construct of interest. For example, if we are measuring motivation, does the score provide an accurate characterization of that construct? This aspect of validity is what most people think of and seems so intuitive that it can be taken for granted. If it’s a measure of student motivation, then it must be measuring motivation (and not some related but spurious) concept . . . right?
Construct validity is measured in a variety of ways where each successive attempt adds evidence to the validity argument. The students’ performance on the assessment may be compared to other similar measures where similar performance is expected (convergent). Their performance may also be compared to dissimilar constructs where the expected relation between the two is low (discriminant). A statistical procedure such as a factor analysis can be performed to see if the items on the test are loading on a common construct or predicted set of constructs.
- Content validity indicates the degree to which the test covers all relevant aspects of the construct or content coverage. For example, in a sixth-grade math achievement test, does the assessment cover all the standards adequately? Or does a test of anxiety adequately cover all relevant aspects of that construct?
In assessing content validity we want to see if the items on the test adequately cover the domain or standards of interest. This is typically done through panels of experts who are able to recognize what test takers must tap into in order to answer the items appropriately and how that aligns to desired standards or theory surrounding the construct.
- Criterion validity provides insights into how the test’s interpretations align with other known measures of the construct. Within this there are two subcomponents (concurrent and predictive). Concurrent verifies that test results correspond with the interpretations of a similar test taken proximal to one another. Predictive indicates the test’s ability to make predictions of future outcomes believed to be related to the construct being measured.
To measure criterion validity, correlations between the test outcome and some external criterion or future outcome are examined. For example, scores on an interim test measuring sixth grade math achievement may be used to predict the test takers’ subsequent end-of-year state assessment performance.
- Face validity is a weaker, though no less important, form of validity in that it addresses whether the assessment appears to measure the construct on the surface. This is important for obtaining buy-in for the use of the test in making important decisions.
Verification of face validity can be accomplished formally from expert judgment or through simple user inspection of the assessment.
Relationship between validity and reliability in assessment
Reliability and validity are both essential but not the same. Think of validity as the measure of accuracy and reliability as the measure consistency of that accuracy. I chose to discuss reliability first in this blog because it all starts with reliability. A test can still be reliable even if it has low validity. It will simply measure the wrong thing consistently. But a test cannot be valid if its reliability is low. If you obtain different results due to low reliability, how can you ever make valid claims about a student’s performance on a construct you are measuring?
The figure below shows how the relationship between reliability and validity can be seen visually. In this example, the goal is not only to hit our target but to do so consistently.
Reliability and validity are inextricably tied together in assessment yet offer unique contributions to some key dimensions underlying assessment.
| Dimension | Validity | Reliability |
| Central Concern | Measures the correct thing: accuracy and relevance to the intended construct. | Measures consistently: stability and repeatability of results. |
| Interpretation implication | Determines whether conclusions drawn from scores are meaningful and defensible. | Determines whether observed scores are dependable enough to interpret. |
| Core question | “Are we measuring what we think we’re measuring?” | “Would we get the same result if we measured again?” |
| Practical implication | Invalid results lead to wrong conclusions, even if measurements appear consistent. | Unreliable results cannot be trusted or replicated, undermining any further analysis. |
| Evidence criteria | No single numeric threshold; evaluated as a body of evidence. | Often quantified as a coefficient (e.g., α ≥ 0.70 acceptable, ≥ 0.80 good, depending on purpose). |
| Dependence | Requires reliability as a prerequisite (a measure cannot be valid if it is unreliable). | Does not require validity (a measure can be reliable but invalid). |
| Type of error addressed | Systematic error: bias that skews results in one direction. | Random error: noise that causes results to fluctuate unpredictably. |
| How assessed | Expert judgment, factor analysis, correlation with established criterion measures. | Statistical coefficients: Cronbach’s α, test-retest r, Cohen’s kappa. |
| Example in education | A math assessment that truly reflects students’ mathematical reasoning ability (not reading skill or test-taking ability). | A math assessment that yields similar scores when repeated under similar conditions. |
Fairness in assessment: The role of validity and reliability
Fairness in testing means that all test-takers have an equal opportunity to demonstrate their knowledge, skills, or abilities, and that scores have the same meaning and lead to equivalent outcomes across different groups of interest such as language background, disability status, or socioeconomic status. The Standards for Educational and Psychological Testing (2014) explicitly integrate fairness with validity and reliability. Fairness is not treated as a separate add-on; it is embedded across reliability and validity evidence.
If a test has suspect validity, it can negatively impact the fairness of the test. For example, if a test is designed to measure math achievement but has an excessively high reading load, it may unnecessarily disadvantage multilingual learners and lead to an overrepresentation of this group in math intervention. If a test has suspect reliability, it may lead to false classification into intervention or omit students who could benefit from intervention. These issues are especially acute when students fall near the cut points. For decisions in these cases, even minor issues with validity or reliability can lead to false classifications.
Fairness can be measured in a variety of different ways:
- Differential item functioning (DIF): The purpose of DIF is to examine how each item in an assessment performs for students in different at-risk groups in relation to a reference group. Item performance is compared for students in the at-risk and reference groups with similar overall performance. If both groups have similar total scores and the at-risk group performance is relatively lower on certain items, those items are flagged for potential bias and usually modified or removed from the assessment.
- Subgroup performance comparisons: The final score is evaluated instead of individual items. Student performance is disaggregated in subgroups of interest to verify if a particular group is underperforming in unexpected ways. A subgroup difference does not necessarily invalidate a test, as there may be valid reasons for the discrepancy, but it is cause for a legitimate inquiry into what the underlying causes may be.
- Consequences of test use: Fairness is evaluated at the level of not only the test itself but also how test results are used and what they do in the real world. Consequences of test use represent real-world outcomes (intended and unintended) that follow from score interpretation and decisions. An essential element of fairness includes the examination of whether test use and interpretation have equivalent meanings and what the consequences are across subgroups. A test is considered unfair when the same score leads to inequitable consequences across populations.
The consequences of test fairness have significant and impactful consequences for students. Differential impact across groups can be manifested through disproportionate placement into remediation, unequal access to advanced coursework, and systematic under-identification or over-identification of subgroups.
Putting validity into practice
Validity and reliability are not merely technical concepts reserved for psychometricians and researchers—they are foundational to the responsible and equitable use of assessment in education. As this blog has outlined, reliability ensures that assessment results are consistent and dependable, while validity ensures that those results are accurately interpreted and meaningfully applied. Neither concept stands alone. A test that produces inconsistent results cannot support valid interpretations, and a test that measures the wrong construct offers little value no matter how consistently it does so.
For teachers and administrators, understanding these concepts has direct practical implications. Every time an assessment is used to place a student in intervention, identify a learning need, or evaluate a program’s effectiveness, both the reliability and validity of that assessment are implicitly invoked. When those properties are weak or untested, the consequences extend beyond measurement error—they can translate into real inequities for students, particularly those in historically underserved subgroups.
The integration of fairness into this framework reminds us that assessment is never a purely technical exercise. The questions we must ask go beyond “Is this test consistent?” and "Does it measure what we intend?" We must also ask, “For whom does it work well, and for whom might it fall short?” Attending to differential item functioning, subgroup performance, and the downstream consequences of test use ensures that assessment serves its intended purpose: to inform instruction and support all learners equitably.
As you evaluate the assessments used in your school or district, consider these concepts not as a checklist but as an ongoing commitment to evidence-based, fair, and purposeful practice.
***
Build on dependable insights with an integrated approach to assessment. Discover how MAP Growth integrates with other tools to support clearer, more actionable next steps.
Turn data into timely, targeted instruction for every student with our free guide.
