Into Math: K–8 Early Outcomes Study

National education legislation has emphasized the importance of all U.S. students achieving mathematics proficiency. However, students in the United States continue to fall behind their international peers (NCTM, 2014). Out of 79 countries participating in the most recent Program for International Assessment (PISA, 2018), 37 countries had higher mathematics literacy scores as compared to the U.S. (OECD, 2019). Moreover, the United States’s average math score was statistically significantly below the OECD average. Furthermore, on the 2019 National Assessment of Educational Progress (NAEP) math assessment, only 41% of fourth-grade students and 34% of eighth-grade students scored at or above the proficient level (U.S. Department of Education, 2019). These statistics are concerning, as early deficits can have enduring and devastating effects on later education outcomes, career success, and long-term quality of life (Gersten et al., 2009; Lubinski, Benbow, & Kell 2014).

For today’s students to successfully meet the demands of rigorous mathematics standards, they need to not only demonstrate a mastery of computational skills and procedures; students must also foster a deep understanding of mathematical thinking.

To those ends, Houghton Mifflin Harcourt’s (HMH) Into Math © 2020 program was designed to provide a tailored learning experience that keeps students engaged in mathematics by allowing all students to experience success. HMH Into Math was built on the idea that long-term success is rooted in global mathematical reasoning abilities, not discrete competencies. Grounded in this premise, the solution emphasizes the development of a conceptual understanding that will allow students to transfer their knowledge to new situations and apply it to new contexts. The lessons are intentionally structured to ensure that procedural practice functions to reinforce conceptual knowledge. As they move through the solution, students build procedural fluency by drawing connections between new material and previously learned concepts (National Mathematics Advisory Panel, 2008).

HMH Into Math incorporates whole-class, small-group, and computer-assisted instruction into clear, purposefully structured lessons, empowering teachers to create a learning community where students can experience success and persevere through mistakes. This model aligns to research that indicates students benefit from exposure to varied experiences that incorporate teacher-guided and small-group learning with digital math programs that allow them to practice independently, at their own pace (Cheung & Slavin, 2013). A further advantage of individually tailored programs, such as HMH Into Math, is that by providing students with material that is appropriate to their skill level, students develop a growth mindset that inspires self-confidence and allows them to see beyond moments when they struggle (Pasquale, 2005).

With the implementation of the Every Student Succeeds Act (ESSA), education agencies expect that new educational programs are supported by empirical evidence of their effectiveness. To gather preliminary data on the potential impact associated with Houghton Mifflin Harcourt’s Into Math™, JEM & R, LLC conducted an impact study on the HMH Into Math program during the 2019–2020 school year (see Resendez, 2020). Of note, the study was interrupted due to the outbreak of COVID-19 and the subsequent nationwide school closures beginning in mid-March 2020. This Research Results paper describes the results of the implementation of HMH Into Math during the shortened 2019–20 school year.

Five schools districts that had already purchased the program were recruited for study participation. Participating teachers were asked to implement the HMH Into Math program with fidelity as part of their regular math instruction; complete the HMH Math Inventory at the beginning, middle, and end of the school year; and provide ongoing feedback on their experiences. As previously noted, due to nationwide school closures, teachers were not required to administer the end-of-year HMH Math Inventory and the current study report includes data collected from the beginning (“pre”) and midyear (“post”) administration only.

Results suggest that Into Math has a positive influence, as students made significant progress from pretest to posttest. These results were consistent across all grade levels, and at all sites. Further, students in all subgroups examined, including those eligible for free or reduced-price lunch, students with Limited English Proficiency (LEP), and those with Individualized Education Plans (IEPs) also made statistically significant progress from the beginning of the year to the middle of the year.

The full report prepared by JEM & R, LLC describes the overall implementation study across all districts, as well as detailed teacher reactions and feedback about the program (Resendez, 2020). For the purposes of this Research Results paper, the analyses focus on the short-term impact of the program on student achievement.

Five school districts and ten individual schools participated in the study from Arizona (3 schools), Florida (1 school), Illinois (1 school), Iowa (3 schools), and North Carolina (2 schools). Table 1 shows the total number of teachers and students participating in the study by grade. Of note, demographic data was not available from the Florida district. The Arizona district was distinct compared to the other districts participating in the study due to the predominant Hispanic student population, high Limited English Proficiency (LEP) (55%), and free or reduced-price lunch rates (over 90%). The remaining districts were predominantly White and with approximately a third of students receiving free or reduced-price lunch.

There were seventy-five teachers from Grades K–8 and the total student sample size was approximately 2,778 students. Only students that completed both a beginning-of-year and middle-of-year HMH Math Inventory assessment during the 2019–2020 school year were included in the analyses.

Across all the sites, 39% of classrooms were taught by Kindergarten to second-grade teachers (n = 30), 44% of classrooms were third- to fifth-grade teachers (n = 33), and the remaining 17% were sixth- to eighth-grade teachers (n = 13). Table 2 shows the total number of teachers and students participating in the study by grade.

As noted before, participating districts had previously purchased the HMH Into Math program and all participating teachers already had full access to the print and digital resources during the study period. Teachers were also provided with implementation guidelines that outlined the essential program components and “Theory of Action” of the solution, as well as program-specific training to ensure that all teachers had sufficient knowledge and skills to successfully implement HMH Into Math. The trainings offered to all sites included the following:

• Summer/Fall 2019—Getting Started in Person Program Overview
• Summer/Fall 2019—Follow-Up Course (Virtually Presented)
• Fall 2019/Winter 2020—Follow-Up Course (in Person)
• Fall 2019/Winter 2020—Follow-Up Course (Virtually Presented)
• 2019-2020 School Year—4 Virtual Team Coaching Sessions (30 minute sessions)

Various methods were employed to collect data as part of the study including descriptive information, program implementation data, teacher feedback, and outcome data.

Fidelity and Implementation

Study teachers logged a bi-weekly implementation report on each lesson completed as well as the components and resources utilized. The Implementation Teacher Logs allowed the researchers to monitor fidelity of HMH Into Math usage during the study period. These logs were developed as a Google^® Worksheet to provide teachers with easy “anytime” access in a user-friendly format. Online implementation logs were completed by most teachers in Arizona, Florida, Illinois, and Iowa until school closures in March. Teachers in North Carolina did not utilize lesson components as presented in the HMH Into Math program and were exempted from completing the logs.

Student Assessment

The HMH Math Inventory assessment from the HMH Ed online learning platform was administered to students at the beginning of the school year to obtain baseline data. The online assessment was administered again in January to measure growth. As previously indicated, the end-of-year administration was cancelled due to nationwide school closures. The HMH Math Inventory is an adaptive assessment that measures math achievement and growth from Kindergarten to Algebra II, using national norms and the Quantile^® Framework. It can be administered in a group setting in around 30 minutes up to 3 times per year.

Teacher logs (n=44) were reviewed and analyzed to determine fidelity of implementation during the regular school year (prior to school closures). Implementation fidelity categories were determined by comparing the number of lessons/modules in which they used each of the essential (and recommended) HMH Into Math program components out of the estimated total lessons/modules that were completed over the school year.

Teachers reported teaching math each day for a minimum of 54 minutes and a maximum of 89 minutes for core math instruction (M = 74.0, SD = 5.0).

If teachers used the majority of essential components with 70% or more of the total completed lessons/modules, they were classified as "high" implementers, 50%-69% were categorized as "moderate", and less than 50% usage were categorized as “low” implementers. It should be noted that teachers from North Carolina were not included in this analysis as they were exempted from completing implementation logs.

Across the remaining districts, there were 10 teachers classified as high implementers, 15 moderate, and 16 low. Overall, the HMH Into Math program components that the majority of teachers used with high fidelity included Warm-Up, Build Your Understanding, Step It Out, Check Your Understanding, Differentiated Options, and Module Review. The program components that the majority of teachers used with low fidelity were Wrap-Up, Module-Opener Task, Are you Ready, and Turn and Talk.

Teachers were also asked to report which lessons they completed (and intended to complete) by the end of the school year via the end-of-year teacher survey. On average, teachers reported covering 4 units of the program over the school year. Generally, primary elementary teachers covered less (2-4 units) as compared to upper elementary and middle school teachers (2-8 units).

In order to determine whether significant learning gains were observed in the content taught before school closures, student beginning-of-year (pre) and mid-year (post) HMH Math Inventory Quantile scores were analyzed via paired sample t-tests. The results are summarized in Table 3 and Figure 1. As shown, students demonstrated a 14-point increase from pre to post testing, and this growth was statistically significant. Figure 2 shows the average pre- and post-Quantile score across all HMH Into Math students that provided both pre- and post-test scores.

When analysis was conducted by grade level, results showed significant learning gains across all grade levels K–8, p<.05. Grade levels that demonstrated the greatest gains were Kindergarten (29.1 points), sixth grade (22.9 points), and fifth grade (17 points).

The HMH Math Inventory Quantile scores were analyzed by subpopulations in order to determine whether significant growth was observed among different types of HMH Into Math students via paired t-tests. In addition, to determine whether there was a significant relationship between subgroup status and growth (i.e., differential growth rates), repeated measures ANOVA were conducted with subgroup as a between subjects variable.

Results showed significant learning gains among all subgroups of Into Math students. That is females, males, students receiving free/reduced-price lunch and not, students receiving special education services and those not, Limited English Proficient (LEP) students and non-LEPs, and gifted students all demonstrated significant learning gains from pre to post testing, p<.05. In addition, significant learning gains were also observed among students of all three baseline performance levels (see Appendix).

As noted in the Implementation Section, teachers were rated as high, moderate, or low implementers based on reported usage in the implementation logs. Repeated measures ANOVA followed by paired t-test analyses were used to examine the relationship between implementation levels and math learning.

As shown in Figure 3, significant learning gains were present for each implementation level, p<.05. While there was no significant relationship between implementation fidelity and student growth, when moderate and high fidelity teachers are combined, differential learning gains are observed with teachers who used the program with more fidelity having students demonstrating higher levels of math gains, F(1, 779) = 4.218, p<.05.

Overall, teachers reported mostly moderate to high levels of program implementation and fidelity. The components that the majority of teachers used with high fidelity included Warm-Up, Build Your Understanding, Step It Out, Check Your Understanding, Differentiated Options, and Module Review. In terms of unit coverage, on average teachers reported covering 4 units of the program over the school year. Generally, primary elementary teachers covered less (2–4 units) as compared to upper elementary and middle school teachers (2–8 units).

When examining student performance as measured by the HMH Math Inventory, statistical analysis at all grade levels showed significant growth in math skills from pre to post testing (midyear). Moreover, statistically significant growth was observed across all different types of students including females, males, students receiving free/reduced-price lunch, students receiving special education services, gifted students, and students of limited English proficiency. Significant gains were also observed among various levels of implementation but were particularly high for teachers implementing the solution with greater levels of fidelity; this suggests that using the program as intended leads to greater math gains.

In sum, HMH Into Math was found to be useful to teachers and associated with significant math learning gains among students in K–8, even during a shortened study period (fall to midyear). While the results of the study are promising, more rigorous, experimental research is necessary to determine the efficacy of the program when compared to other math solutions.

Boaler, J. (2016). Mathematical mindsets: Unleashing students’ potential through creative math, inspiring messages and innovative teaching. Wiley Digital

Cheung, A.C. & Slavin, R.E. (2013). The effectiveness of educational technology applications for enhancing mathematics achievement in K-12 classrooms: A meta-analysis, Educational Research Review, 9, 88–113

Gersten, R., Beckmann, S., Clarke, B., Foegen, A., Marsh, L., Star, J. R., & Witzel, B. (2009). Assisting students struggling with mathematics: Response to Intervention (RtI) for elementary and middle schools. Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.

Lubinski, D., Benbow, C. P., & Kell, H. J. (2014). Life paths and accomplishments of mathematically precocious males and females four decades later. Psychological Science, 25, 2217–2232.

National Council of Teachers of Mathematics (2014). Principles to actions: Ensuring mathematical success for all. Reston, VA: Author.

Organization for Economic Cooperation and Development (OECD), 2019. "Comparing countries’ and economies’ performance in mathematics", in PISA 2018 Results (Volume I): What Students Know and Can Do, PISA, OECD Publishing, Paris, https://doi.org/10.1787/ad5d244c-en.

Resendez, M. A. (2020, July). Houghton Mifflin Harcourt Into Math K-8 impact study: Final report. Yuma, AZ: Author.

U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics (2019). National Assessment of Educational Progress (NAEP), various years, 1990–2019 Mathematics and Reading Assessments. Washington, D.C. : Author.