System 44: Charles Armstrong School 2018–2019

Read 180 now incorporates the comprehensive foundational literacy skills scope and sequence from System 44.

Students who do not meet or exceed benchmark scores on state and national tests are less likely to graduate from high school, are less likely to persist in or successfully complete future academic and workplace training endeavors, and are overall less likely to be on track for future academic and workplace success (CCRSC, 2013). Compared to other students, students with language-based learning differences such as dyslexia are 2 ½ times more likely to fail grade-level benchmark tests by 4^th grade and three times more likely to fail tests by 8^th grade (National Center for Education Statistics, 2013). The Charles Armstrong School identified the need to provide an additional reading intervention program to accelerate Grades 3 to 7 students who have been diagnosed with dyslexia to grade-level reading proficiency.

System 44 is a reading intervention program designed to build foundational reading and decoding skills for the most challenged readers. This blended learning solution includes explicit, systematic instruction in foundational literacy skills through whole-group instruction, small-group targeted differentiated reading instruction, personalized learning through the software, and an independent reading library. Materials feature modeled and independent reading with age-appropriate books that target decoding skills and strategies to promote comprehension, build vocabulary, and increase content-area knowledge. Using adaptive technology, students complete a personalized learning progression through five strands that include The Code, Word Strategies, Sight Words, Success, and Writing.

The Charles Armstrong School in Belmont, California, is a private school designed to educate students with language-based learning differences such as dyslexia. The Charles Armstrong school serves approximately 240 second through eighth grade students¹. It serves students with a range of ethnic backgrounds: African American (2%), Asian (2%), Caucasian (78%), Hispanic (4%), Native American (2%), and students with multiple ethnic backgrounds (12%). In the 2018–2019 school year, 25% of students received financial aid.

All students who completed System 44 instruction and a pretest and posttest (n=45) were included in the analysis of Phonics Inventory scores. An additional 18 students did not complete a Phonics Inventory posttest; these students are included in analyses involving school-collected reading metrics. Student demographics for the entire sample (N=63), which do not vary markedly from the demographics of the smaller sample used for Phonics Inventory analyses, are below (see Table 1; see Appendix Table 1 for the demographics of the smaller sample).

Charles Armstrong School students in Grades 3 to 7 who were identified as struggling with foundational reading skills completed the Phonics Inventory and placed in the System 44 intervention. Students completed approximately 20 minutes in the personalized online student application between two and five days a week. Students continued to receive standard core reading instruction in the classroom (utilizing the Wilson curriculum and program materials) in addition to System 44 instruction.

Teachers met with two HMH consultants for an overview of the System 44 program and ongoing instructional coaching sessions while implementing System 44 during the 2018-2019 school year.

Student software usage data was collected as students used the online student application during System 44 instruction. Software usage data included number of completed series, number of completed topics, number of completed sessions, average time spent in each session, and number of sessions averaged per week.

The HMH Phonics Inventory measures proficiency in the foundational reading skills of phonological decoding and sight word reading for students in Grades 3 to 12. The Phonics Inventory is used to identify whether students with low reading comprehension achievement also lack the skills needed to decode new words (leading to placement in System 44) or are best served by an intervention to develop reading comprehension strategies, text analysis skills, and background knowledge (leading to placement in other interventions). Assessment results include an accuracy score (range of 0–60) based on accurately reading sight words and nonsense words, a fluency score (range of 0–60) based on reading accurately as well as quickly, and a decoder status (pre-decoder, beginning, developing, or advancing). Phonics Inventory subscores are provided for Letter Recognition (range of 0–11), Nonsense Word Accuracy (range of 0–30), Sight Word Accuracy (range of 0–30), Sight Word Fluency (range of 0–30), and Nonsense Word Fluency (range of 0–30). Charles Armstrong students completed the Phonics Inventory before beginning instruction in System 44, and again at the end of the school year.

The Read Naturally Reading Fluency Progress Monitor is an efficient, valid, and reliable assessment to measure a student's progress when reading text aloud. The teacher listens to a student read a leveled passage at his/her instructional level (Grades 1–8) for one minute and determines the student's words correct per minute. Scores are graphed throughout the year to track student progress and make educational decisions.​ Charles Armstrong School students were assessed for Oral Reading Fluency using the Reading Fluency Progress Monitor multiple times during the 2018–2019 school year.

Track My Progress is an online, computer-adaptive test designed to assess math and reading skills aligned to the Common Core State Standards in kindergarten through eighth grades. Track My Progress tests have been shown to be both reliable and valid measures of Common Core State Standard proficiency, normed to a nationally representative US sample. Student progress in meeting skills is tracked as students take four 20-minute tests each year, and teachers have access to reports on specific subject and domain proficiency. Charles Armstrong School student data included a spring 2019 percentile score (range 0–99) for Reading Comprehension. A percentile score above 41 is considered to reflect grade-level ability.

The Word Identification Spelling Test is a nationally standardized, individually administered diagnostic test that assesses students' fundamental literacy skills. Three subtests (Word Identification, Spelling, and Sound-Symbol Knowledge) and a composite score called the Fundamental Literacy Index are reported. The Word Identification subtest measures word reading accuracy which includes (a) students' sight recognition of familiar words and their ability to apply word attack skills in order to decode unfamiliar words and (b) their sight recognition or orthographic memory of high-frequency words with one or more irregularities. The Spelling subtest assesses students' ability to spell words correctly from dictation and specifically measures students' (a) recall of correct letter sequences for familiar words or one's ability to apply sound/symbol relationships and rules of English orthography in order to spell unfamiliar words and (b) their recall of letter order in high-frequency words with one or more irregularities. The Sound-Symbol Knowledge subtest assesses a student's ability to associate sound(s) (i.e., phonemes) with specific letter(s) (i.e., graphemes). The Word Identification Spelling Test is both valid and reliable for its stated purposes. Charles Armstrong School student data included a spring 2019 percentile score (range 0–99) for Word Identification, Spelling, Sound-Symbol Knowledge, and the Fundamental Literacy Index.

The HMH Reading Inventory measures reading comprehension proficiency for students in Grades K to 12. The Reading Inventory uses adaptive technology to determine a student’s reading comprehension level on the Lexile Framework for Reading; the higher the Lexile score, the more challenging reading material the student can comprehend. Test item difficulty ranges from items appropriate for developing readers to items requiring a reading proficiency indicating preparedness for college level texts, allowing measurement of skill growth regardless of the students’ initial ability. Assessment results include a Lexile scale score that indicates reading ability at a level of text complexity and a performance level of below basic, basic, proficient, or advanced, indicating achieved reading comprehension compared to grade-level expectations. Charles Armstrong School students completed the Reading Inventory in the spring of 2019, following System 44 instruction.

Charles Armstrong School teachers began implementation of System 44 at different time points throughout the 2018–2019 school year (see Figure 1), and as such, length of student participation varied greatly. Third grade students began in February 2019, 4^th grade students began in March 2019, 5^th grade students began in April 2019, 6^th grade students began in November and December 2018 and others started in March and May 2019, and 7^th grade students began in November 2018 and some in May 2019.

Students who participated in System 44 during the 2018–2019 school year (see Table 2) completed an overall average of 2.9 System 44 series (SD=2.7) and 34.2 topics (SD=24.4) over an average of 28 total sessions (SD=18.8), averaging 2.1 (SD=0.6) sessions per week and totaling 521.7 minutes (SD=408.3) in the System 44 online student application.

An independent evaluator from Forge Research Group analyzed student academic achievement using test score data provided by the Charles Armstrong School and program usage data provided by Houghton Mifflin Harcourt. System 44 students’ English Language Arts (ELA) performance was examined pre- and post-implementation using multiple independent measures of reading. This analysis included scores on the Phonics Inventory and the Read Naturally Reading Fluency Progress Monitor. System 44 students demonstrated significant gains in ELA proficiency during the 2018–2019 school year on each of these reading measures. Notably, because of the self-paced nature of the program and the variability in participation, individual student gains were inconsistent.

System 44 students who completed the Phonics Inventory multiple times demonstrated statistically significant overall gains in Phonics Inventory fluency scores. Students increased from an average of 8.8 on first assessment to 11.2 on the second assessment (see Figure 2). When disaggregated by grade level, 3^rd and 4^th grade students demonstrated a significant overall gain in Phonics Inventory fluency scores. Fifth grade students completed the least amount of time in System 44 (M=121 minutes compared to the overall M=522 minutes) and had 2 outlier students with large pretest to posttest decreases, resulting in non-significant overall decreases in Phonics Inventoryfluency scores (mean change was -0.40 points when outliers were removed, indicating that participation was not long enough to increase reading fluency). Sixth and 7^th grade students demonstrated notable gains in Phonics Inventory fluency scores, however, inconsistent time spent using System 44 resulted in inconsistent student gains, decreasing the overall statistical significance. Three students’ grade level was not identified and so are only included in the overall results.

In addition to Phonics Inventory fluency score gains, System 44 students also achieved increased pretest to posttest decoder status on the Phonics Inventory (see Figure 3). The percentage of students achieving an “Advancing” decoder status on the Phonics Inventory, indicating readiness to move to a more complex reading intervention, increased overall from 4.4% to 11.1%, while the percentage of students scoring at a “Pre-Decoder” decoder status, indicating little or no foundational reading skills, decreased from 2.2% to 0% during the 2018–2019 school year.

Of note, approximately 25% of the System 44 students who began the program with a Developing decoder status progressed to an Advancing decoder status, indicating readiness to graduate from the System 44 intervention. Large differences in mean topics completed and time spent in the software (see Table 3) distinguished between students who progressed from Developing to Advancing (mean topics completed=74.5, mean software time=995.5 minutes) and students who remained as Developing (mean topics completed=42.2, mean software time=526.6 minutes).

Phonics Inventory subscores also demonstrated significant literacy gains during the 2018–2019 school year (see Figure 4). At pretest, Charles Armstrong School students demonstrated proficiency in Letter Name Accuracy (mean score 10.87 out of 11), an indication that students had the prerequisite skills to build decoding skills. From pretest to posttest, System 44 students achieved statistically significant gains in both Sight Word Accuracy (2.33 points), and Nonsense Word Accuracy (2.40 points), indicating an increase in decoding skills. Students also achieved statistically significant gains in both Sight Word Fluency (1.18 points) and Nonsense Word Fluency (1.28 points), indicating progress in foundational reading skills. Importantly, the gains in accuracy and fluency for both sight words and nonsense words indicate that System 44 is an effective intervention for students with dyslexia.

Students with dyslexia often fit one of two identified scoring trends², having one of the following reading profiles: 1) possessing sight word reading skills but not nonsense word reading skills, indicating compensated memorization of words and a lack of true decoding skills; or 2) possessing neither sight word nor nonsense word reading skills, indicating both a lack of decoding skills and little experience with reading the irregular words and vocabulary specific to the English language. According to the pretest subscores, 7 students demonstrated the first profile and 18 students demonstrated the second profile. By posttest, 6 of the 7 profile 1 students increased in nonsense word accuracy and 17 of the 18 profile 2 students increased both sight word and nonsense word accuracy, indicating that all students increased decoding skills rather than continuing to compensate with memorization.

Further, increased use of the System 44 online software was a statistically significant predictor of Phonics Inventory Fluency score growth, both before and after correcting for selection bias (see Appendix Table 3 for details). On average, System 44 students who completed more topics in the System 44 online software³ also achieved higher gains in the Phonics Inventory Fluency score after accounting for initial Fluency scores (see Figure 5).⁴

Notably, disaggregation of the data indicated that System 44 was associated with significant Phonics Inventory Fluency Score gains for most categories of students during the 2018–2019 school year (see Figure 6). When results were disaggregated by gender, both males and females achieved statistically significant Phonics Inventory gains from pre- to post-instruction. When results were disaggregated by ethnicity, Caucasian students achieved statistically significant Phonics Inventory gains. Although the sample was too small (n=6) to capture the true significance of gains statistically, multiracial students gained an average of 4.7 points on the Phonics Inventory, in line with other students’ gains. Other categories of students had fewer than 4 members and so results were suppressed.

In addition to increases on the Phonics Inventory, System 44 students demonstrated statistically significant growth in Oral Reading Fluency (ORF) on the Read Naturally Reading Fluency Progress Monitor. From the beginning to end of the 2018–2019 school year, System 44 students averaged a 60% increase in words correct per minute (wcpm) on the ORF test (see Figure 7). When disaggregated by grade level, 3^rd, 4^th, and 5^th grade students all demonstrated significant gains in the ORF, gaining 34, 29, and 30 wcpm, respectively.

System 44 students completed the Reading Inventory at the end of the school year in May 2019. The Reading Inventory is sensitive enough to capture below-grade-level ELA ability and growth not usually captured by other assessments (including negative Lexile scores). System 44 students who completed the Reading Inventory in the spring averaged a 410 Lexile (L) Score (see Figure 8), with extreme variation between students in different grades (from 180L to 784L).

Charles Armstrong School System 44 students also completed multiple reading metrics at the end of the school year. In addition to the HMH Phonics Inventory and Reading Inventory, students completed the Word Identification Spelling Test (WIST), the Track My Progress Reading Comprehension assessment, and the Read Naturally ORF assessment. Spring 2019 scores on each of the valid, nationally normed school reading metrics demonstrated a statistically significant correlation with each of the HMH reading metrics. While pretest and posttest scores were not available to directly test the relationship between gains on HMH reading metrics and the WIST or Track My Progress assessments for the 2018–2019 school year, this indicates that System 44 students’ increases in HMH reading metrics will likely be correlated with increases in school reading metrics in subsequent years.

Multiple independent measures support the idea that students who received System 44 instruction made statistically significant improvements in English Language Arts and Literacy achievement. After one year of instruction, students in Grades 3 to 7 at the Charles Armstrong School demonstrated statistically significant increases in Phonics Inventory fluency scores. Students also increased in assessment performance levels; the percentage of students achieving an Advancing decoder status on the Phonics Inventoryincreased overall from 4% to 11%. Students achieved statistically significant increases in Phonics Inventory subscores measuring both sight word and nonsense word fluency, indicating growth in decoding skills rather than compensated memorization, as is often seen in students with dyslexia. Disaggregation of the data by gender indicated that use of System 44 was associated with significant gains in Phonics Inventory fluency scores for both male and female students. Students worked through varying amounts of the self-paced, personalized online instruction components – increased completion of System 44 topics was a significant predictor of Phonics Inventory fluency score gains during the 2018–2019 school year. In addition to gains measured by HMH metrics, System 44 students also demonstrated statistically significant gains in Oral Reading Fluency from beginning to the end of school year as measured by the Read Naturally Reading Fluency Progress Monitor. This study demonstrates that using System 44 to provide a reading intervention is an effective method of increasing literacy for students struggling to achieve grade-level ELA proficiency.

1. Charles Armstrong School Quick Facts found at www.CharlesArmstrong.org.
2. Scholastic Phonics Inventory Data Interpretation Guide accessed at http://52.1.239.6/product-supp....
3. Graph excludes 2 multivariate outliers (students who completed a large number of topics and showed large score decreases).
4. Of note, the Charles Armstrong School System 44 student data showed a strong correlation between pretest score and number of topics completed, with students with the lowest pretest scores also completing the fewest topics. This pattern was found because 3rd and 4th grade students had lower pretest scores than 6th and 7th grade students (as would be expected), and the minimal and moderate implementation groups primarily included 3rd and 4th grade students while the high implementation group primarily included 6th and 7th grade students. Further confounding the relationship, students in lower grades and with lower pretest scores generally make more gains that students in higher grades and with higher pretest scores. As such, the relationship between fluency gains and number of topics completed (M=3 for each implementation group) is more similar across implementation group than would be expected in a population in which students in all grades participated in System 44 for equal amounts of time.

College and Career Readiness and Success Center (CCRSC). (2013). Predictors of Postsecondary Success. Washington, DC: American Institute for Research.

National Center for Education Statistics. (2013). The Nation’s Report Card: A First Look: 2013 Mathematics and Reading (NCES 2014-451). Institute of Education Sciences, U.S. Department of Education, Washington, D.C.