English 3D: Hemet Unified School District Quasi-Experimental and Correlational Studies

The Hemet Unified School District (HUSD) in Hemet, California, is located in the Riverside County community. The HUSD served between 21,414 and 22,025 K-12th grade students in the 2014–2015 through 2018–2019 school years¹. HUSD serves a diverse population of students with a range of ethnic backgrounds averaging: African American (9%), Asian/Filipino (<1%), Caucasian (25%), Hispanic (60%), Native American (<1%), Native Hawaiian/Pacific Islander (<1%), and students with multiple ethnic backgrounds (4%). In the 2014–2015 through 2018–2019 school years, 79–83% of students in the district were socioeconomically disadvantaged, 11–14% were English learners (ELs), and 11–15% were classified as Students with Disabilities (SWD).

^{1 http://www.ed-data.org/district/Riverside/Hemet-Unified}

HUSD implemented English 3D^® beginning in the 2014–2015 school year with 4th through 8th grade EL students and expanded to include 4th through 12th grade EL students in the 2016-2017 school year. Not all HUSD EL students participated in English 3D instruction; rather, it was used as a targeted intervention for EL students most struggling with English proficiency. Students with long-term EL status or determined at risk of long-term EL status (enrolled in a U.S. school for 6 or more years but not progressing in English language skills) were placed in English 3D instruction in addition to a regular English Language Arts (ELA) class using Collections ELA materials. Students continued English 3D instruction until achieving the minimum score needed for EL reclassification on the state English language proficiency exam. These EL students received regular 45-minute ELA instruction as well as 60-minute English 3D instruction daily. In this Single Period Instruction Model, students progressed through English 3D lessons used flexibly by the teacher to build essential skills.

During each school year, Dr. Kate Kinsella, English 3D author, and her team, provided approximately ten sessions of coaching support for HUSD EL teachers. Implementation materials included a teaching guide outlining the three essential routines (partner and group interactions, response frames, and setting up and monitoring tasks), key EL instruction techniques, use of English 3D assessment and differentiated learning materials, and a planning guide. Throughout these school years, the English 3D team also provided demonstration lessons.

Thirteen elementary schools, three K-8 schools, seven middle schools, and five high schools in the Hemet Unified School District serving students in Grades 4 through 12 utilized English 3D as a Tier 2 English Language Development intervention during the 2014-2015 through 2018-2019 school years.

Study 1: Quasi-experimental Study Participants

By year and by outcome measure, English 3D students were matched to non-English 3D EL students within the Hemet Unified School District for a matched sample comparison analysis. All students comprising the matched sample did not participate in English 3D instruction in any year between 2014–2019. Matched students were an exact match on disability status and then chosen using nearest neighbor propensity score matching based on the following covariates: grade, gender, SES, ethnicity, school, and teacher. Matched students were an exact match on baseline score or expected growth (a measure of both baseline score and time between first and last test date) when a sufficient sample size (>350) was produced or otherwise it was included as a covariate (see details in results by outcome measure). Each group in each year comprised students from at least two teachers and two schools to avoid confounds and no covariate exhibited a large imbalance (>.25SD of pooled mean).

Student ethnic backgrounds varied by school year (see Table 1), but students were predominantly Hispanic (93.6%–96.4%). Of these students, approximately half were male and half were female, the majority were low income (92.1%–98.9%), and one-sixth to one-fourth (16.0%–27.4%) were classified as Students with Disabilities.

Study 2: Longitudinal Correlational Analysis Participants

All students identified by the school district as enrolled in English 3D and not simultaneously participating in other HMH reading interventions were included in the longitudinal analysis (total N=1475; 55 students did not complete both a pretest and posttest on any outcome measure). Student ethnic backgrounds varied by school year (see Table 2), but students were predominantly Hispanic (86.5%–96.5%). Of these students, approximately half were male and half were female, the majority were low income (92.3%–98.2%), and one-fifth to one-third (18.4%–37.5%) were classified as Students with Disabilities.

Measures of Academic Progress (MAP) Reading

Measures of Academic Progress (MAP^®) is a computer-adaptive interim (Fall, Winter and Spring) assessment developed by the Northwest Evaluation Association that measures reading, language usage, and mathematics. The primary score produced is the RIT (Rasch Unit) scale, which is a stable equal-interval vertical scale. This grade-independent RIT score indicates the level of question difficulty a given student can answer correctly about 50% of the time. Percentile ranks for the MAP are also provided. HUSD students in Grades 2–8 and 10 completed the MAP Reading assessment at least twice each year (fall and winter, fall and spring, or winter and spring).

Reading Inventory

The HMH Reading Inventory^® measures reading comprehension proficiency for students in Grades K–12. The Reading Inventory uses adaptive technology to determine a student’s reading comprehension level on the Lexile Framework for Reading; the higher the Lexile^® score, the more challenging reading material the student can comprehend. Test item difficulty ranges from items appropriate for developing readers to items requiring a reading proficiency indicating preparedness for college-level texts, allowing measurement of skill growth regardless of the students’ initial ability. Assessment results include a Lexile score that indicates reading ability at a level of text complexity and a performance level of below basic, basic, proficient, or advanced, indicating achieved reading comprehension compared to grade-level expectations. Beginning in the 2016–2017 school year, HUSD students receiving English 3D instruction completed the Reading Inventory at least twice each year (fall and winter, fall and spring, or winter and spring).

Smarter Balanced Assessment (SBA) English Language Arts/Literacy (ELA) Summative Assessment

The Smarter Balanced Assessment was designed to measure end-of- year achievement in English Language Arts/Literacy (ELA) and to accurately capture growth in ELA proficiency from previous years. The SBA measures ELA achievement in reading, writing, listening, and research using a computer adaptive system to deliver between 42 and 48 questions. Assessment results include a scale score between 2,000 and 3,000 and an Achievement Level Descriptor (ALD) indicating overall performance level ranging from 1 to 4, with a 3 or higher indicating grade-level ELA proficiency. HUSD students complete the SBA each spring in Grades 3–8 and 11.

California English Language Development Test (CELDT)

The California English Language Development Test was designed to identify students as English learners and to annually assess progress towards English language proficiency. Aligned to the 1999 English-Language Development Standards for California Public Schools, the CELDT assesses progress in the four domains of listening, speaking, reading, and writing English. Assessment results include scale scores and proficiency levels for each domain (beginning, early intermediate, intermediate, early advanced, or advanced) as well as an Overall Student Performance Level created from the average of the four domain scale scores. Students remain in EL services until a “proficient” level of English is achieved, as determined by an Overall Student Performance Level of early advanced or higher and a performance level in each domain (listening, speaking, reading, writing) of intermediate or higher. HUSD students designated as EL completed the CELDT annually through the 2016–2017 school year.

English Language Proficiency Assessment for California (ELPAC)

The English Language Proficiency Assessment for California was designed to measure achievement of the California State Board of Education’s 2012 ELD standards. The ELPAC assesses the domains of reading, writing, listening, and speaking. Assessment results include a scale score (that can be compared across grades) for each domain with a domain performance level (Beginning to Developed, Somewhat to Moderately Developed, Well Developed). Listening and Speaking scores combine into an Oral Language composite while Reading and Writing scores combine into a Written Language composite. The Oral Language and Written Language composites average into an overall ELPAC score and these are associated with Overall and Composite Score performance levels (Minimally developed, Somewhat developed, Moderately developed, Well developed). Students remain in EL services until a “Well developed” Overall performance level is achieved on the ELPAC. HUSD students designated as EL completed the ELPAC each year beginning in the 2017–2018 school year.

An independent evaluator from Forge Research Group analyzed student academic achievement using data provided by the Hemet Unified School District and Houghton Mifflin Harcourt. English 3D students’ ELA performance was examined pre- and post-English 3D usage using multiple independent measures including scores from multiple timeframes within the school year (HMH Reading Inventory and MAP), the statewide end-of-year summative assessment (SBA), and the English proficiency exam (CELDT or ELPAC). English 3D students demonstrated statistically significant gains in ELA proficiency during the 2014–2015 through 2018–2019 school years on each of these reading measures and demonstrated accelerated gains compared to matched samples of non-English 3D EL students on the MAP Reading assessment and the CELDT English proficiency exam.

Measures of Academic Progress (MAP)

In the 2014–2015 school year, matched students were an exact match on disability status and then chosen using nearest neighbor propensity score matching based on the following covariates: expected MAP growth^₂, grade, gender, SES, ethnicity, school, and teacher. In school years 2016–2017 through 2018–2019, matched students were an exact match on disability status and expected MAP growth and then chosen using nearest neighbor propensity score matching based on the following covariates: grade, gender, SES, ethnicity, school, and teacher. English 3D students who completed the MAP Reading test at two timeframes within the school year (fall and winter, fall and spring, or winter and spring) demonstrated larger statistically significantly actual to expected² MAP Reading RIT score gains compared to a matched sample of non-English 3D EL students overall and in the 2014–2015 school year (see Figure 1).

^{2 As compared to 2020 NWEA MAP Growth Normative Data}

Smarter Balanced Assessment (SBA) English Language Arts/Literacy (ELA) Summative Assessment

Matched students were an exact match on disability status and then chosen using nearest neighbor propensity score matching based on the following covariates: previous year’s SBA ELA scale score, grade, gender, SES, ethnicity, school, and teacher. Overall, both English 3D and non-English 3D students who completed the SBA in both comparison years demonstrated a statistically significant gain in SBA ELA scale scores from one school year to the next. These average SBA ELA scale score gains were statistically equivalent for both groups (see Figure 2).

From spring 2017 to spring 2018, the percentage of students meeting or exceeding SBA ELA grade-level standards remained stable for both English 3D students and non-English 3D students (see Figure 3). From spring 2018 to spring 2019, the percentage of students meeting or exceeding SBA ELA grade-level standards increased by 70% (from 4.8% to 8.0%) for English 3D students and by 57% (from 9.8% to 15.4%) for non-English 3D students.

California English Language Development Test (CELDT) / English Language Proficiency Assessment for California (ELPAC)

Matched students were an exact match on disability status and then chosen using nearest neighbor propensity score matching based on the following covariates: previous year’s CELDT or ELPAC composite score, grade, gender, SES, ethnicity, and school. Overall, both English 3D and non-English 3D students who completed the CELDT and ELPAC in both comparison years demonstrated a statistically significant gain in composite score from one school year to the next (see Figure 4). From spring 2016 to spring 2017, English 3D students demonstrated statistically significantly larger CELDT gains as compared to non-English 3D students. From spring 2018 to spring 2019, the average ELPAC score gains were statistically equivalent for both groups.

Importantly, a larger percentage of English 3D students were reclassified out of the EL program compared to the matched sample of non-English 3D EL students, with reclassification of 19.2% compared to 14.3% in the 2016–2017 school year, 29.4% compared to 13.5% in the 2017–2018 school year, and 6.1% compared to 5.3% in the 2018–2019 school year (see Figure 5).

Measures of Academic Progress (MAP)

English 3D students completed the MAP Reading test in the fall, winter, and spring of each school year. Overall, students who completed the MAP Reading test at two timeframes within the school year (fall and winter, fall and spring, or winter and spring) demonstrated a statistically significant overall increase in RIT Scale Scores, averaging an 8-point (2014–2015), 5-point (2015–2016), 7-point (2016–2017), 7-point (2017–2018),and 2-point (2018–2019) gain from first testing to last testing within the designated school year (see Figure 6).

When results were disaggregated by gender, both males and females achieved statistically significant MAP Reading RIT scale score gains (see Appendix Table 1 available in the Full Report download at the top of the page). Though students in this sample were predominantly Hispanic, both Hispanic and non-Hispanic students achieved statistically significant MAP Reading RIT Scale score gains in school years 2014–2015, 2016–2017, 2017–2018, and 2018–2019. Likewise, both students classified as SWD and those not classified as SWD achieved statistically significant MAP Reading RIT scale score gains in the 2014–2015 through 2017–2018 school years (see Figure 7).

In contrast to national EL student trends, HUSD English 3D students met or exceeded national MAP Reading growth averages in four of the five school years included in this study (see Figure 1). When examined by grade level, elementary school English 3D students demonstrated the most consistent MAP Reading RIT score growth each year, averaging 6- to 9-point gains. Of note, these gains were statistically significantly greater than average student growth³ in school years 2014–2015, 2016–2017, and 2017–2018 (see Figure 8). Middle school students also demonstrated statistically significant growth each year, averaging 2- to 7-point gains. These gains were statistically significantly greater than average student growth in school years 2014–2015 and 2016–2017, but less than average student growth in school year 2018–2019. High school students met growth expectations in school years 2016–2017 and 2018–2019 but averaged nonsignificant -1.5-point losses in school year 2017–2018. These growth patterns are similar to expected national high school student MAP Reading growth, in which average growth ranges from -5.8 to 9.8.

^{3 As compared to 2020 NWEA MAP Growth Normative Data}

Reading Inventory

English 3D students completed the Reading Inventory in the fall, winter, and spring of each school year beginning in 2016–2017. Overall, students⁴ who completed the Reading Inventory at two timeframes within the school year (fall and winter, fall and spring, or winter and spring) demonstrated a statistically significant overall gain in Lexile (L) scores, averaging a 78L (2016–2017), 96L (2017–2018), and 74L (2018–2019) increase from first to last testing in the school year (see Figure 9). Further, the increase in average Reading Inventory scores was statistically significant for elementary, middle school, and high school students with sufficient sample sizes (n>25).

^{4 As a previous READ 180 Universal gains analysis has shown that students who decrease more than 100L on the Reading Inventory from fall to spring demonstrate poor test motivation and produce unreliable longitudinal achievement scores, those scores that decreased more than 100L were excluded from this implementation analysis. The Reading Inventory scores of some HUSD students decreased more than 100L in 2016-2017 (2%/n=8), 2017-2018 (7%/n=10) and 2018-2019 (16%/n=48).}

When results were disaggregated by gender and ethnicity, both males and females and Hispanic and non-Hispanic students achieved statistically significant Reading Inventory Lexile score gains (see Appendix Table 2 available in the Full Report download at the top of the page). Likewise, both students classified as SWD and those not classified SWD yearly achieved statistically significant Reading Inventory Lexile score gains (see Figure 10).

Overall, English 3D students achieved equivalent or greater than average Reading Inventory growth⁵ in each of the three school years included in this study (see Figure 11). When examined by grade level, middle school English 3D students met growth expectations in school years 2016–2017 and 2018–2019 and achieved statistically significantly greater than average student growth in school year 2017–2018. High school English 3D students’ growth was less consistent, as they met growth expectations in school year 2016–2017, achieved statistically significantly greater than average student growth in school year 2017–2018, but did not meet growth expectations in school year 2018–2019.

^{5 As compared to 2017 Reading Inventory Estimated Average Annual Growth Data}

Of note, in the 2016–2017 school year, 50% of students met or exceeded average Reading Inventory growth, with 33% of students increasing 1, 13% of students increasing 2, and 3% of students increasing 3 or more grade levels in reading proficiency (see Table 3). In the 2017–2018 school year, 58% of students met or exceeded average Reading Inventory growth, with 36% of students increasing 1, 14% of students increasing 2, and 6% of students increasing 3 or more grade levels in reading proficiency. In the 2018–2019 school year, 38% of students met or exceeded average Reading Inventory growth, with 28% of students increasing 1, 12% of students increasing 2, and 5% of students increasing 3 or more grade levels in reading proficiency.

English 3D student Reading Inventory performance levels also reflected these increases in reading proficiency (see Figure 12). The percent of students achieving at least a proficient Reading Inventory performance level increased from first to last testing by 50% (from 24% to 36.0%) in the 2016–2017 school year, by 101% (from 20% to 40.3%) in the 2017–2018 school year, and by 54% (from 3.7% to 5.7%) in the 2018–2019 school year. Further, the percent of students scoring at a below basic performance level decreased from first to last testing by 18% (from 49% to 40%) in the 2016–2017 school year, by 41% (from 54.1% to 32.1%) in the 2017–2018 school year, and by 16% (from 76% to 64.2%) in the 2018–2019 school year. The increase in students’ performance levels on the Reading Inventory was statistically significant each school year.

Smarter Balanced Assessment (SBA) English Language Arts/Literacy Summative Assessment

HUSD English 3D students completed the SBA ELA summative assessment, measuring end-of-year ELA skills, in the spring of each school year in Grades 3–8 and 11. Overall, English 3D students who completed the SBA in both comparison years demonstrated a statistically significant gain in SBA ELA scale score from one school year to the next, averaging a 31-point increase from spring 2017 to spring 2018 and a 37-point increase from spring 2018 to spring 2019 (see Figure 13). These average SBA ELA scale score gains were statistically significant for both elementary and middle school students.

When results were disaggregated by gender and ethnicity, both males and females and Hispanic and non-Hispanic students achieved statistically significant SBA ELA scale score gains (see Appendix Table 3 available in the Full Report download at the top of the page). Likewise, both students classified as SWD and those not classified as SWD achieved statistically significant SBA ELA scale score gains in the 2017–2018 through 2018–2019 school years (see Figure 13). These results indicate that English 3D students of every demographic category achieved ELA proficiency gains.

English 3D student SBA ELA performance levels also reflected increases in reading proficiency (see Figure 14). The percent of students achieving a level 2 (nearly met standard) SBA ELA Achievement Level Descriptor (ALD) increased by 28% (from 21.6% to 27.7%) from spring 2017 to spring 2018, and by 51% (from 17.4% to 26.3%) from spring 2018 to spring 2019. Additionally, the percent of students achieving at least a level 3 ALD (met standard) increased by 70% (from 4.4% to 7.5%) from spring 2018 to spring 2019 and the overall increase in students’ performance levels from the previous school year was statistically significant.

Likewise, English 3D students demonstrated a statisticallysignificant gain in ELPAC composite scores from one school year to thenext, averaging a 23-point increase from spring 2018 to spring 2019 (see Figure 16). These average ELPAC composite score gains werestatistically significant for elementary, middle school, and high school students.

California English Language Development Test (CELDT) / English Language Proficiency Assessment for California (ELPAC)

HUSD used an English language proficiency examination (the CELDT until 2017 and then the ELPAC beginning in 2018) to determine student EL status and to reclassify students once proficiency standards were met.

Overall, English 3D students demonstrated a statistically significant gain in CELDT composite scores from one school year to the next, averaging a 38-point increase from spring 2015 to spring 2016 and a 35-point increase from spring 2016 to spring 2017 (see Figure 15). These average CELDT composite score gains were statistically significant for elementary, middle school, and high school students.

Likewise, English 3D students demonstrated a statistically significant gain in ELPAC composite scores from one school year to the next, averaging a 23-point increase from spring 2018 to spring 2019 (see Figure 16). These average ELPAC composite score gains were statistically significant for elementary, middle school, and high school students.

When results were disaggregated by gender and ethnicity, both males and females and Hispanic and non-Hispanic students achieved statistically significant CELDT and ELPAC composite score gains (see Appendix Table 4 available in the Full Report download at the top of the page). Likewise, both students classified as SWD and those not classified as SWD achieved statistically significant CELDT composite score gains in the 2015–2016 through 2016–2017 school years and ELPAC composite score gains in the 2017–2018 through 2018–2019 school years (see Figure 17).

Of note, 48% (2014-2015), 39.3% (2015–2016), and 57.8% (2016–2017) of English 3D students achieved at least an early advanced CELDT Overall Student Performance Level, indicating grade-level proficiency (see Table 4). Likewise, 71.7% (2017–2018) and 52.8% (2018–2019) of English 3D students achieved at least a moderately developed ELPAC Overall performance level.

English 3D students who completed the CELDT in both comparison years demonstrated statistically significant CELDT Overall Student Performance Level gains (see Figure 18). The percent of students achieving at least an early advanced performance level (indicating grade-level proficiency) increased by 100% (from 20% to 40%) from spring 2015 to spring 2016 and by 72% (from 33% to 57%) from spring 2016 to spring 2017.

English 3D students with the lowest ELPAC Overall performance levels who completed the ELPAC in both comparison years also demonstrated performance level gains. The percent of students achieving a somewhat developed (level 2) performance level increased by 37% (from 27% to 37%) from spring 2018 to spring 2019 (see Figure 19).

The increases achieved on the CELDT and ELPAC were also reflected in increasing numbers of English 3D students being reclassified (no longer needing EL services; see Figure 20). English 3D was implemented in the 2014–2015 school year with HUSD students with long-term EL status, and initially, the reclassification rate was 5% (17/320). As students continued using the English 3D program, the reclassification rate rose to 8% (8/104) in the 2015–2016 school year, 22% (130/594) in the 2016–2017 school year, 26% (162/624) in the 2017–2018 school year, and 15% (67/440) in the 2018–2019 school year.

When examined by domain, English 3D students achieved statistically significant increases in each of the CELDT domains (see Figure 21) of reading (40-point increase from 2016 to 2017 and 34-point increase from 2017 to 2018), writing (30-point increase from 2016 to 2017 and 32-point increase from 2017 to 2018), listening (46-point increase from 2016 to 2017 and 37-point increase from 2017 to 2018), and speaking (37-point increase from 2016 to 2017 and 37-point increase from 2017 to 2018).

Likewise, English 3D students achieved statisticallysignificant increases in each of the ELPAC composite scores (see Figure22) of written language (16-point increase from 2018 to 2019) and orallanguage (29-point increase from 2018 to 2019). Additionally, 51%(157/308) of English 3D students increased a performance levelin at least one ELPAC Domain from the 2017–2018 to the 2018–2019 schoolyear: 22% in speaking, 13% in listening, 25% in reading, and 7% inwriting.

Multiple independent measures of academic English language skills support the idea that students classified as English learners or long-term English learners who received English 3D instruction made significant improvements in English Language Arts and Literacy achievement.

HUSD targeted long-term EL students who were not making gains for multiple years to participate in English 3D instruction. Despite beginning at a disadvantage, English 3D students made gains equal to (SBA, ELPAC) or statistically significantly greater than (MAP, CELDT) a matched sample of non-English 3D EL students during the 2014–2015 through 2018–2019 school years. Further, these gains were reflected in a greater EL reclassification rate for English 3D students as compared to non-English 3D EL students.

In a correlational analysis across a five-year time span, HUSD students in Grades 4–12 who received English 3D instruction demonstrated yearly statistically significant increases in Measures of Academic Progress (MAP) Reading assessment RIT Scale Scores, HMH Reading Inventory Lexile scores, Smarter Balanced Assessment (SBA) ELA scale scores, California English Language Development Test (CELDT) composite scores, and English Language Proficiency Assessment California (ELPAC) composite scores. Of note, these gains were achieved by students in each demographic category of grade level, gender, ethnicity, and Students with Disabilities classification.

Further, English 3D students demonstrated statistically significantly greater than national average growth on the MAP Reading assessment and the HMH Reading Inventory. Importantly, 45%–56% of English 3D students increased at least one grade level in reading comprehension each year, with 5%–40% reading at or above grade level.

English 3D students also increased in assessment performance levels: 20%–23% of students moved up an Academic Level Descriptor on the SBA each year; 17%–42% of students moved up a performance level on The Reading Inventory each year; and 39%–72% of students achieved at least a “proficient” CELDT Overall Student Performance Level or “moderately developed” ELPAC Overall performance level, indicating an ability to access grade-level curriculum. This study demonstrates that English 3D is an effective English language development program for EL students struggling to achieve grade-level ELA proficiency.