MATH 180: Early Outcome Effects of a Blended Learning Model for Math Intervention Instruction With Special Population Students

The four districts participating in the research study  represented several regions throughout the United States, including the South, Southwest, Midwest, and West. The districts varied considerably in size from 12,957 to 197,041 students served. Student populations were racially diverse, ranging from 33% to 70% non-white, with significant percentages of students with at-risk classifications.

During the 2013–2014 school year, 20 middle schools from four school districts participated in the MATH 180 early outcomes research study. The participating districts selected 10 schools to use the MATH 180 program in their classrooms and an equal number of comparable schools to serve as controls (non-MATH 180). The non-MATH 180 schools were selected by the districts using factors such as prior achievement and their demographic composition. One MATH 180 program school dropped out of the study due to technology issues that prevented it from implementing the program. The issues were eventually resolved but not in time to participate in the MATH 180 program.

Teachers identified to deliver the MATH 180 instruction were provided with an initial two-day training session at the beginning of the school year to familiarize themselves with the program and reinforce the importance of adhering to the implementation model. The teachers were also provided with an additional day of training in the middle of the school year. This session focused on the interpretation and effective use of student data that were currently being generated by the software and Math Inventory^® test scores. In addition to the three training days, teachers had periodic coaching visits throughout the year which averaged approximately one visit per month. The purpose of the periodic coaching visits was to check in on the teachers to address questions or concerns and observe them in the classroom for the purposes of program implementation fidelity as they delivered their MATH 180 lessons.

Only students in Grades 6–8 who were identified as  needing supplemental math instruction were selected for inclusion in the study. The criteria for inclusion varied across schools but typically focused on state assessment, Math Inventory, math course grade, and teacher recommendation data. The control schools continued providing math intervention as they had previously. Students in the MATH 180 program received the full complement of  MATH 180 services outlined below.

In the treatment condition schools, the MATH 180 program replaced the typical supplemental math course for targeted students. The MATH 180 math intervention program was designed to assist middle school students who are struggling with multiplication, division, fractions, decimal operations, and integers. MATH 180 is a blended learning model of instruction that incorporates teacher-facilitated instruction and adaptive computer-assisted instruction. The software allows the students to work at their own pace and return to previously visited material. The course consists of nine instructional blocks with three topics each for a total of 27. The MATH 180 program is designed to be implemented in a 45- to 55-minute block each day: whole-class instruction occurs during the first 5 minutes and is followed by two 20- to 25-minute rotations. During one rotation, students work on a computer using adaptive instruction and practice; during the other rotation, students participate in teacher-facilitated small-group instruction designed to foster mathematical reasoning and student self-efficacy in math. Student progress is captured and assessed through ongoing student monitoring in the software, mSkills (a curriculum-embedded assessment administered at the end of each block), and an independent measure of mathematical readiness (Math Inventory) administered three times per year (fall, winter, spring).

19 schools across four districts participated in the study; 9 schools served in the treatment condition, and 10 in the non-MATH 180 condition. From these schools a total of 926 students participated, 351 in the non-MATH 180 condition and 575 in the MATH 180 treatment condition. Of the 926 students in the study, 802 were classified as special education (SPED), English learners (EL), and/or eligible for the Free or Reduced-Price Lunch program (FRL). Table 1 provides the demographic characteristics for the students in the non-MATH 180 and MATH 180 groups. The table indicates that non-MATH 180 and MATH 180 students were reasonably comparable across all demographic indicators including race/ethnicity, gender, grade level, and status. Both groups were racially diverse with almost 85% of the samples indicated as non-white. The non-MATH 180 and MATH 180 groups also had a significant number of students classified as special education (SPED) with 17.5% and 14.4%, respectively. Students who were classified as English learners (EL) also represented a large percentage of the non-MATH 180 and MATH 180 student samples with 29.2% and 30.6%, respectively. The largest group represented in the study was students who were eligible for free or reduced-price lunch. Although this Socioeconomic Status (SES) proxy is not ideal, it serves as a useful metric for understanding performance for economically disadvantaged students. Students characterized as FRL in the non-MATH 180 and MATH 180 groups represented 81.1% and 79.5% of the students in the study respectively.

Although analyses were carried out for the SPED, EL, and FRL groups separately, these three groups were not mutually exclusive in that students could fall into multiple categories. Students were defined as special education using information provided by their schools or districts. Although the term special education refers to a wide range of student disabilities, there was no further differentiation as to the nature of the disability that warranted their inclusion in this category.English learner designation was also based on dataprovided by the student’s school or district.

The final special population identified in this research was based on Socio-Economic Status (SES). Although many people agree that a student’s socio-economic status is important in understanding student academic attainment, there is no unified agreement as to how it should be measured or articulated. For purposes of this study, SES was operationally defined as a student’s eligibility for the federally-funded Free or Reduced-Price Lunch program (FRL). This proxy was chosen based on the accessibility of this data as well as its popular use in educational research. The Free or Reduced-Price Lunch program sets eligibility requirements based on a student’s family income relative to the poverty level. However, the program is voluntary, and students who may be eligible may choose not to participate. Therefore, caution should be exercised when drawing conclusions based on this data.

In order to measure the potential effects of the MATH 180 program on students from these special populations, it was important to understand their levels of progress or engagement (an aspect of implementation) with the program, in addition to their changes in academic achievement. The level of engagement with a program is a key characteristic in understanding the program’s impact on student outcomes.

There were two elements of program implementation that were examined for this analysis: software and teacher-facilitated instruction progress. Student software progress through the self-paced instructional blocks was recorded and analyzed to determine the number of blocks the students completed.  Teacher-facilitated group instruction was monitored by the examination of mSkills assessments completed. After each direct instructional block, students were required to take an online mSkills assessment. Therefore, it was assumed that if an mSkills assessment took place, it was likely the teacher completed that block.

The Math Inventory is a computer-adaptive test (CAT) which provides information on a student’s readiness for mathematics instruction.  As a computer-adaptive test, Math Inventory delivers test items targeted to a student’s ability level across six content strands (Numbers Sense, Numerical Operations, Geometry, Algebra and Algebraic Thinking, Data Analysis and Probability, and Measurement). Math Inventory quantifies a student’s path up through high school mathematics using a vertical scale aligned to the Quantile^® Framework for Mathematics. The test takes approximately 30–50 minutes to complete. Scores on Math Inventory can be used for progress monitoring and as a component for placement decisions.

Student progress through the program was assessed by examining the number of blocks completed through teacher-facilitated instruction and the number of blocks completed  by students in the self-paced software component of theprogram. Students varied considerably with respect to the number of blocks completed in the software. The average progress through the instructional software blocks by the end of the year for those designated as SPED, EL, and FRL was 3.43, 4.34, and 3.88, respectively. On average, teacher-facilitated instruction progressed through Block 4, and teachers were actively working on Block 5 when the school year ended.Therefore, both students and teachers were able to complete approximately half of the program.

The academic outcomes of students from special populations who participated in the MATH 180 early outcomes research study were compared to the academic outcomes of similar students receiving some other form of supplemental math intervention. The analyses focused on students from three unique but not mutually exclusive groups based on SPED, EL, and FRL designations provided by the participating schools or districts. The educational outcomes discussed for each of these groups are based on their performance on the Math Inventory. Score equivalency was investigated in three areas. The initial investigation focused on whether the fall scores for the non-MATH 180 and MATH 180 students were equivalent at the start of the study. Second, the difference from fall scores to spring scores was analyzed for significant growth for the non-MATH 180 and MATH 180 groups individually. The final comparison examined the growth from fall-to-spring for the non-MATH 180 group and compared it to the fall to spring growth exhibited by the students in the MATH 180 program.

Special education status was determined by the students’ classification in their schools or districts. There were N=61 students in the non-MATH 180 condition and N=83 in the  MATH 180 program. This sample represented 17.5% of the control group and 14.4% of the MATH 180 students. Figure 1 summarizes Math Inventory fall and spring scores for both the non-MATH 180 as well as MATH 180 schools for students designated as special education. Results indicate students in the control schools had slightly elevated initial fall Math Inventory scores (470Q) compared to students in the MATH 180 schools (438Q); this difference, however, was not significant. When we examine growth over the course of the school year, students in the non-MATH 180 condition did not demonstrate a significant change from fall to spring (-10Q). Students in the MATH 180 program, by comparison, did experience significant growth over the course of the year, moving from an average score of 438Q in the fall to 527Q in the spring for a growth of 89Q. This change in Math Inventory score for the MATH 180 students over the course of the year was statistically significant. Not surprisingly, when we compare the level of growth across the non-MATH 180 and MATH 180 groups, MATH 180 students demonstrated significantly more growth, with a difference of 99Q.

Language status was defined as any student categorized as an English learner (EL) by their schools or districts. There were N=102 students identified as English learners in the non-MATH 180 condition and N=176 EL students in the MATH 180 program. This sample represented 29.2% of the comparison group and 30.6% of the MATH 180 students. Figure 2 summarizes Math Inventory fall and spring test scores for both the non-MATH 180 as well as MATH 180 schools for students designated as English learners.

Students in either group had approximately the same  average score in the fall, with 496Q for the non-MATH 180 and 486Q for the MATH 180 students. These baseline scores were not significantly different from one another. When we examine growth over the course of the school year, students in both conditions experienced significant growth. Students in the non-MATH 180 condition grew by moving from an average score of 496Q in the fall to 544Q in the spring for an increase of 48Q. Students in the MATH 180 condition also grew over the course of the year, moving from an average score of 486Q in the fall to 609Q in the spring, for an increase of 123Q.

This difference in growth between the non-MATH 180 EL students and MATH 180 EL students was also statistically significant. When we examine level of growth across the non-MATH 180 and MATH 180 program, the MATH 180 students demonstrate significantly greater gains than the non-MATH 180 students, with a difference of 75Q.

There were N=283 students in the non-MATH 180 condition and N=457 students in the MATH 180 program who were categorized as eligible for free or reduced-price lunch. This sample represented 81.1% of the non-MATH 180 group and 79.5% of the MATH 180 students. Figure 3 summarizes Math Inventory fall and spring scores for both the non-MATH 180 as well as MATH 180 schools for students qualifying for free or reduced-price lunch.

Students in the non-MATH 180 condition initially had a slightly higher average fall Math Inventory score than the students in the MATH 180 program with scores of 536Q and 518Q, respectively. The difference in these baseline scores, however, was not statistically significant. When we examine growth over the course of the school year, students in both conditions experienced significant growth. Students in the non-MATH 180 condition over the course of the year grew from an average score of 536Q in the fall to 570Q in the spring for an increase of 34Q. Students in the MATH 180 program also experienced significant growth over the course of the year, moving from an average score of 518Q in the fall to 621Q in the spring for an increase of 103Q.

This difference in growth between the non-MATH 180 school FRL students and MATH 180 FRL students was also examined to see if there was differential growth between the conditions. When we compare the level of growth across the non-MATH 180 and MATH 180 program, the MATH 180 students demonstrated significantly greater gains in Math Inventory performance than the students in the non-MATH 180 condition, with an average growth differential of 69Q.

The positive gains experienced by all three groups (SPED, EL, and FRL) examined in this study translated to gains in proficiency categorization as well. Figure 4 summarizes  the movement experienced by each of these groups. A significant majority of students in each of these subgroups originally tested into the Below Basic proficiency level in the fall, with SPED at 92%, English learners also at 92%, and free-or-reduced-price-lunch-eligible students at 87%. By the spring semester, many of these students moved up in proficiency level, with the percent of SPED students testing at Basic increasing from 6% in the fall to 30% in the spring. For students classified as EL and FRL, the same trend emerged with the percentage of EL students initially testing at Basic moving up from 7% in the fall to 30% in the spring and the percentage of FRL students testing at Basic moving up from 12% in the fall to 33% in the spring. Moving students out of the Below Basic category is significant because it represents the widest score bands in the proficiency table. For example, Below Basic for students in the 6th grade spans up to 640Q (EM-640Q), 7th grade spans 700Q (EM-700Q), and 8th grade spans 800Q (EM-800Q). For many students this is an enormous first step toward moving upward through the proficiency levels.

In order for students to experience the full effect of an intervention program, it is important that the program be delivered as it was originally intended—with fidelity.  Students in this study completed slightly less than half of the blocks in the software, and teachers were able to provide instruction for slightly more than half of the blocks.

The results of this one-year study indicate that the benefits of the MATH 180 program on student academic outcomes hold for varying special populations of students. These findings are of particular importance because students with these designations tend to be at greater risk of negative outcomes. When examining the demographic composition of the students in this study, it is apparent that the intervention populations of many school districts likely contain significant percentages of these groups. Many of these students were several years behind with respect to their math progress. For many students, successes observed in this program represent the first experienced in a long time. The road to grade-level proficiency is a long one for many of these students, and it is difficult at this time to determine if student growth will continue and whether the rate of growth will be adequate to achieve algebra readiness by the end of their 8th-grade year. However, results indicated that, although most of the students began the program performing at Below Basic levels, a significant number of them have begun that climb to proficiency. It is critical that these students continue to receive the support necessary to maintain this upward trajectory.