At a glance
- Program: Amira Learning
- Subjects: Assessment, Early Learning, Literacy Curriculum
- Report Type: Efficacy Study, Study Conducted by Third Party
- Grade Level: Elementary
- Region: Southeast
- Population: Students with Disabilities, Students with Specific Learning Disabilities
- Race/Ethnicity: Asian, Black, Hispanic, White, Multiracial
- District Urbanicity: Urban
- District Size: Large
The Savannah-Chatham County Public School System (SCCPSS), the tenth largest district in Georgia, enrolls approximately 19,000 elementary students. District enrollment is 57% Black, 23% white, and 12% Hispanic.
Amira Learning is an artificial intelligent (AI) reading tutor that listens to students read out loud, assesses mastery, screens for dyslexia, and delivers personalized one-on-one tutoring for kindergarten through fifth grade students who are learning to read. As the child reads aloud, Amira Learning uses AI to decipher what obstacles the young reader is facing, selecting in real-time from over 40 reading science-based micro-interventions to bridge specific reading skills gaps. Students use Amira Learning independently without the help of a teacher. In addition, every interaction with Amira Learning generates data for teachers, administrators, and parents with information to shape their real-world reading support and instructional time. This report describes the associations between Amira Learning usage and student literacy development during a one-year implementation In Savannah-Chatham County Public Schools.
During the 2020-21 academic year, kindergarten through third grade SCCPSS students participated in Amira Learning. Beginning in mid-September and continuing through May, schools and teachers were asked to encourage students to complete three to five practice sessions per week, equivalent to approximately 30 minutes a week. Students initially engaged with Amira Learning in a remote setting due to the COVID-19 pandemic, but transitioned back to in-person, classroom settings mid-year.
While approximately 5,000 first and second grade SCCPSS students participated in Amira Learning at some point during the school year, there was significant variation in student usage, which we discuss in more detail below. Our analytic sample, which is limited to students who took a series of fall, winter, and spring assessments and engaged in at least one practice session, includes 2,305 first and second grade students in 29 schools (see Table 1). Our sample was over 50% Black, roughly one-quarter white, and 13% Hispanic, suggesting that Amira Learning participants were representative of the SCCPSS student population in terms of racial/ethnic background. Slightly fewer than 10% of Amira Learning students received special education services, and approximately 7% of Amira Learning students were at risk for dyslexia.
Literacy Outcomes
We examine the associations between Amira Learning usage and five literacy outcomes, including Oral Reading Fluency, Vocabulary Size, Sight Recognition, Phonological Awareness (PA), and Lexile score. Oral reading fluency (ORF) is a student's ability to read text accurately, at an appropriate rate, and with expression. Oral reading fluency is measured through Words Correct Per Minute (WCPM), which captures the number of words a student can read accurately within the one-minute timeframe. We use the adjusted WCPM score, which accounts for differences in passage difficulty. Vocabulary Size estimates the number of words likely present in a student’s expressive vocabulary. Sight Recognition uses the Estimated Sight Recognition Inventory (ESRI) to estimate the percentage of sight words a student has mastered. Phonological Awareness, measured through Phoneme Segmentation Fluency, captures a student’s ability to accurately produce phonemes within words. Finally, Amira Learning produces a Lexile score based on the ORF. Lexile is an outcome of reading ability with a higher Lexile score indicating that a student is capable of reading and understanding more challenging texts. We standardized (z-scored) all outcomes within grade. All assessments were administered three times throughout the year (fall, winter, and spring).
Amira Learning Usage Indicators
To capture Amira Learning usage, we use separate continuous measures of the number of weeks students completed at least one Amira Learning practice session during the fall-to-winter, winter-to-spring, and fall-to-spring periods. We also created categorical versions of these measures indicating very low-usage (1-9 weeks), low-usage (10-14 weeks), medium-usage (15-19 weeks), and high-usage (20 or more weeks) for the full fall-to-spring period. For the shorter fall-to-winter and winter-to-spring periods, the categorical measures indicate low-usage (0-4 weeks), medium-usage (5-9 weeks), and high-usage (10 or more weeks). Similarly, we explore Amira Learning usage with both a continuous measure of the number of practice sessions a student completed and a categorical version of the same variable (1-9 practice sessions, 10-19 practice sessions, 20-29 practice sessions, 30-39 practice sessions, and 40 or more practice sessions for fall-to-spring; 0-4 practice sessions, 5-9 practice sessions,10-19 practice sessions, 20-29 practice sessions, and 30 or more practice sessions for fall-to-winter and winter-to-spring). A practice session is considered any day a student logs on to the platform and reads one or more practice stories.
Students were encouraged to complete multiple practice stories in a given session and multiple sessions per week, from mid-September through May, for a total of approximately 30 weeks and 30 minutes per week. Actual usage rates, however, were lower than expected, with the modal student participating in a practice session for 13 weeks throughout the school year (see Table 2). Roughly 60 percent of students participated in 14 or fewer weeks of practice sessions, meaning the majority of students received less than half of the intended treatment. Ultimately, only 18 percent of students participated for 20 weeks or more. Further, the average number of minutes read per week was nine, well below the recommended 30 minutes per week. Lastly, there are differences in average minutes read and number of sessions completed between usage categories. By definition, high- and medium-usage students not only participated in Amira Learning for more weeks, but during those weeks they also read for more minutes and completed more practice sessions than their low-usage peers.
Results suggest that each additional week of Amira Learning usage was associated with an increase of 0.60 WCPM (0.019 SD) during the fall-to-winter period (p<.001). Using a categorical indicator of usage, students in the low-usage category (10-14 weeks) gained on average 5.9 WCPM (0.187 SD) more than very-low-usage students (1-9 weeks), holding all else constant (p<.001). This is equivalent to a shift from roughly the 50th to the 57th percentile. This linear trend continues with medium-usage students gaining, on average, 8.5 WCPM (0.270 SD) more than their very-low-usage counterparts (p<.001), representing a move from the 50th to the 61st percentile, and high usage associated with a 10.1 WCPM (0.321 SD) average advantage (13 percentile points; p<.001).
Results suggest that each additional practice session completed was associated with an increase in literacy development of 0.16 WCPM (0.005 SD; p<.001). Using a categorical indicator to compare literacy growth between students who completed 10-19 practice sessions, 20-29 practice sessions, 30-39 practice sessions, and 40 or more practice sessions to those students who completed 1-9 practice sessions, there continues to be a positive relationship with number of practice sessions and adjusted WCPM. Students who completed 20-29 practice sessions experienced a 4.0 WCPM (0.126 SD) average advantage over their same-school peers who completed nine or fewer sessions (p<.001). This advantage increased somewhat for students in the 30-39 sessions category (7.1 WCPM; 0.226 SD), and the 40+ sessions category (9.0 WCPM; 0.284 SD; p<.001).
The results are similar and consistent when exploring the relationship between Amira Learning usage and our other literacy outcomes: vocabulary size, ESRI, PA, and Lexile scores (see Graph 1).
Female and male students appear to benefit similarly from Amira Learning usage. However, there is some variation across racial/ethnic groups. Hispanic students gained more for each additional week of Amira Learning usage compared to white students on the ESRI and PA outcomes. Similarly, Black students gained more than white students on the PA and Lexile outcomes for each additional week of usage.
Estimates of the associations between Amira Learning usage and literacy development for the fall-to-winter and winter-to-spring periods suggest that, on average, each week of usage during the fall-to-winter period was associated with 0.95 additional WCPM growth (0.030 SD; p<.001), and a 0.35 WCPM advantage during the winter-to-spring period (less than one percentage point or 0.011 SD; p<.05). It is important to note that the links between usage and literacy development were much stronger during the fall period. See Graphs 2 and 3.
GRAPH 1. Amira Usage and Fall-to-Spring Literacy Outcomes
All differences with very-low usage students significant at least at the p<.01 level.
GRAPH 2. Amira Usage and Fall-to-Winter Literacy Outcomes
All differences with low-usage students significant at least at the p<.01 level.
GRAPH 3. Amira Usage and Winter-to-Spring Literacy Outcomes
~p<.10; *p<0.05; ** p<0.01; ***p<0.001. Significance tests compared to low-usage students.
This report explored the implementation of Amira Learning in Savannah-Chatham County Public School System during the 2020-21 academic school year. The results described above provide promising evidence of the effectiveness of Amira Learning. We consistently found increased usage of Amira Learning to be positively associated with student literacy development for first and second graders. These promising trends were present across the three time periods we examined, although the Amira Learning estimates were generally larger during the fall. The Amira Learning advantage appears to be constant across gender, but did vary somewhat across racial/ethnic groups. Hispanic and Black students benefited more for each additional week of Amira Learning usage compared to white students in the same class with two of the five literacy outcomes.
Finally, it is important to stress that these may be conservative estimates, as the vast majority of students did not receive the recommended Amira Learning dosages. Additionally, the study took place during the 2020-21 school year, one of the most disruptive periods in the history of U.S. education due to the COVID-19 pandemic. Overall, these analyses cannot identify the causal impact of Amira Learning on student development. More robust experimental and quasi-experimental studies are warranted to provide contemporary evidence of the extent to which Amira Learning helps all students achieve reading proficiency. That being said, these results clearly suggest that fidelity of Amira Learning implementation is critical: students who received a greater percentage of the recommended dosage consistently gained more than those students who received smaller dosages.