Waggle: Research Evidence Base

The COVID-19 pandemic has changed how K–12 districts in the U.S. plan for teaching and learning. Due to this pandemic, districts realized they must prepare for multiple options—onsite learning, distance learning, and possibly a hybrid solution. Now more than ever, technology-based instructional solutions will play an essential role in teaching and learning. Districts need research-based solutions that take advantage of the science of learning.

Waggle is a supplemental mathematics and English language arts digital solution for Grades 2–8 that offers adaptive, personalized, guided practice and instruction, while also providing ongoing formative assessment. Waggle leverages the latest findings in educational research and learning science to maximize student proficiency and growth. The intelligent adaptive engine analyzes multiple data points (academic and behavioral) to provide students with the most appropriate learning experience at the right time. Embedded hints and feedback support student mastery and retention of essential knowledge and skills, while also helping students develop higher order thinking and problem-solving skills and a growth mindset. Waggle also fosters persistence and engages students in productive struggle. The system empowers teachers by providing flexibility and real-time data to support instructional decision making.

Waggle offers teachers several options for assigning content. Teachers can assign content to students on a weekly basis, as a supplement to in-class instruction. Alternatively, teachers can have Waggle automatically sequence the content over the course of the school year, following its research-based progression of skills and standards. A third option is for teachers to have students begin by taking the HMH Growth Measure, which can serve as an initial screener for placing students. With all three options, Waggle makes continual adjustments to each student’s learning sequence based on ongoing performance.

This paper summarizes the research base for key elements of Waggle’s design, and explains how Waggle aligns to the research recommendations, including:

• Personalized learning through a continuously adaptive digital learning system;
• Effective retrieval practice to support student long-term mastery and application of knowledge and skills;
• Fostering student persistence and productive struggle; and
• Teacher use of formative assessment and ongoing, in-class performance data to inform instructional decision making.

An adaptive, personalized learning system continuously monitors key learner characteristics to make adjustments to “the objectives, pathways, and pace” of instruction to enhance learning (SRI International, 2018, p. 7). Researchers define learning systems as adaptive to the extent that:

• The design is “based on data about common learner challenges.”
• Pedagogical changes are “based on psychological measures of individual learners.”
• The system “interactively responds to learner actions” (Aleven et al., 2017, p. 523).

A robust body of research indicates that formative assessment data based on student performance on learning tasks can be used to modify instruction and personalize learning, resulting in increased student learning (Aleven et al., 2017; SRI International, 2018).

Some researchers use the term intelligent tutoring systems (ITS) to refer to adaptive, personalized learning systems. A meta-analysis found that student use of an ITS resulted in superior learning outcomes, compared to students experiencing “teacher-led, large-group instruction,” other computer-based instruction, and “textbooks or workbook instruction.” Significant, positive effects were found for ITS at all levels of education and in almost all subject domains evaluated. Results for learning from ITS were similar to learning from individualized human tutoring and from small-group instruction (Ma et al., 2014, p. 901). Earlier studies also found that ITS can be “as effective as one-on-one human tutoring” (SRI International, 2018, p.16).

Research and research-based learning theories strongly suggest that adaptive personalized, learning systems should:

• Include “well-defined student models, knowledge tracing, and detailed hint, help, and feedback support” (SRI International, 2018, p. 29);
• Present learning challenges that are “difficult enough to engage [students] in deep cognitive work but not so difficult that …they become frustrated” (SRI International, 2018, p. 29);
• Adapt to student knowledge by modifying instruction based on the student’s prior knowledge and growth in knowledge, and “based on data about knowledge demands of the domain” (Aleven et al., 2017, p. 549);
• Respond to the specific learning path a student takes through a learning activity including the “student’s solution strategy, specific errors, and requests for help” (Aleven et al., 2017, p., 549, citing VanLehn, 2006, 2011, 2016);
• Adapt to learner affect and motivation (Aleven et al., 2017, p. 539).

HOW WAGGLE ALIGNS TO THE RESEARCH

Waggle is a continuous adaptive learning system that offers personalized practice and instruction to meet each student’s individual needs.

As a student works through Waggle skills-based practice, the system assesses knowledge and skill mastery in real-time. Students’ learning objectives, pathways, and pace are individually optimized through constant analysis of their unique performance and behavior to deliver the right content at the exact right time.

Waggle uses the Knewton recommendation engine¹ to continuously evaluate each student’s performance on learning goals based on a student learning model that includes 13 data points organized in three categories—Learner’s Knowledge, Learner’s Behavior, and Learning Progressions—to provide a detailed picture of what students know and how they learn.

• Learner’s Knowledge data points address learner proficiency on specific skills, the amount of practice needed to become proficient, the need for remediation on prerequisite skills, and which skill(s) to target within the current learning goal after proficiency on the current skill has been reached.
• Learner’s Behavior data points address students’ pace on practice items, their need for repeated practice for content they struggle with, and the learning strategies they use.
• Learning Progressions data points track each student’s progress through grade level sequences of skills that comprise a learning goal within a curriculum strand.

Based on analysis of these data points, and teacher input, Waggle works as an intelligent tutoring system. It identifies an individual student’s strengths, weaknesses, and learning patterns. Based on these, the system places the student on a learning path and monitors the learning environment on an ongoing basis to make appropriate adjustments to better support and enhance learning every time the student interacts with Waggle. These real-time adjustments ensure that students are provided the appropriate level of challenge based on their ongoing learning experience.

Like a good tutor, Waggle offers students detailed hints, help, and feedback support when they get stuck, based on the student’s solution strategy and specific errors.

Waggle also continuously tracks student engagement, and Waggle’s intelligent tutoring system measures and encourages student effort and persistence throughout the learning activities.

When Waggle is used as a supplemental resource, teachers can provide essential whole class and small group instruction while also offering one-to-one-tutoring through Waggle’s intelligent tutoring system.

Retrieval practice refers to learning activities involving deliberate attempts to recall and articulate previously-taught information. It can take a form similar to assessment, such as taking a quiz, but it is intended as a learning strategy rather than as a pure assessment of learning (Argawal & Bain, 2019, p. 48).

Research indicates that retrieval practice can greatly increase learning, and many researchers consider it to be the most important component of learning (Argawal & Bain, 2019, pp. 30–34). Research in various classroom settings has demonstrated that retrieval practice improved student learning across different subject areas, for varying time delays, and at different developmental stages (from preschool to adult)—and it has been shown to boost skill learning, higher-order thinking, and transfer of knowledge (Agarwal & Bain, 2019, pp. 41–42). Based on their research synthesis of the science and practice of learning, The Committee on How People Learn II recommend “practicing retrieval of memorized information, rather than just studying the information again” (National Academies of Sciences, Engineering, and Medicine [NASEM], 2018, p. 55).

Research supports incorporating the following for effective retrieval practice:

• Spaced retrieval practice—that is, practicing retrievals multiple times as well as over a period of time, rather than all at the same time (Agarwal & Bain, 2019, p. 93). Research demonstrates that spacing out learning and retrieval results in improved long-term performance and memory (Agarwal & Bain, 2019, pp. 94, 100 citing Rohrer & Taylor, 2006; Dudai et al., 2015; NASEM, 2018, p. 55).
• Interleaving—intermixing the focuses of retrieval practice (e.g., practicing a math problem of type A, then one of type B, then one of type C), in contrast to blocked practice, where content is grouped with like items presented next to each other (e.g., practicing a series of math problems of type A, then a series of type B, etc.) (Agarwal & Bain, 2019, p 106). Research has established that interleaving retrieval practice “can increase (and even double) student learning and retention” (Agarwal & Bain, 2019, pp. 106-109).

The Importance of Feedback During Retrieval Practice

Frequent feedback during retrieval practice has been shown to support metacognition, so that students become aware of “what they know and don’t know” (Agarwal & Bain, 2019, p. 125). Research indicates that there is an “illusion of fluency” and overconfidence when learning activities are too easy, resulting in students being unable to “accurately predict their own learning.” However, when learning is more appropriately challenging and students are informed about their performance along the way, their accuracy in making predictions about their learning improves (Agarwal & Bain, 2019, p. 128).

Feedback is regarded by many researchers as essential to improving knowledge and skill acquisition (Shute, 2008, p. 153). Reviews of 12 meta-analyses indicate that “the average effects of feedback are among the highest we know in education”² (Hattie & Mark, 2010, p. 475, citing Hattie, 2009). Feedback has also been found to motivate learners (Shute, 2008, p. 153).

• Provide feedback on incorrect and correct answers because when students are positive, they know something and they are wrong, they are then more likely to remember the information in the future (Agarwal & Bain, 2019, p. 133).
• Prioritize elaborated feedback, which is especially useful for knowledge transfer to new contexts (Agarwal & Bain, 2019, p. 134, citing Finn et al., 2018). Elaborated feedback is response-specific and typically explains why the correct answer is correct and “may explain why the selected response is wrong” (Shute, 2008, p. 159). A meta-analysis of studies on feedback found that elaborated feedback produced larger effects than other types and was more effective for higher order learning outcomes (Van der Kleij et al., 2015, p. 475). Other research supports providing feedback that includes both verification of correctness and elaboration (Shute, 2008, p.158).
• Provide scaffolding feedback supports that enable “learners to do more advanced activities and to engage in more advanced thinking and problem solving than they could without such help.” Examples of scaffolding include simplifying tasks to make them more manageable, providing direction to help the student stay focused on the goal, showing the difference between student work and the desired solution, and modeling and clearly defining expectations (Shute, 2009, p. 163, citing Bransford, Brown, & Cocking, 2000).
• Provide feedback immediately or after a delay, as both strategies can be effective. (Agarwal & Bain, 2019, p. 135).
• Encourage students to make mistakes, because retrieval practice with errors plus elaborated feedback can improve learning (Agarwal & Bain, 2019, p. 136, citing Metcalfe, 2017).

HOW WAGGLE ALIGNS TO THE RESEARCH

Waggle was designed for retrieval practice, enabling teachers to assign learning goals to stimulate recall and application of previously taught facts, concepts, and skills.

Teachers can search for and assign follow-up skills practice within Waggle by learning goal or by curriculum standard. After an assignment is completed within Waggle, it remains available for spaced follow-up practice. Students can choose to return to these assignments, review their previous work, and engage in additional practice toward reaching advanced proficiency. Teachers have the option of assigning additional practice on a completed assignment.

Waggle provides interleaving of skills within a learning goal. For example, in English language arts practice, a student might read a passage, and then answer questions associated with three different skills. When a student is having difficulty with a mathematics problem-solving strategy, the system can offer up practice on related prerequisite skills, including skills taught in earlier grades.

Waggle provides robust hints and feedback for students who are having difficulty with practice activities.

Waggle offers detailed hints as scaffolding support for students who are “stuck,” so they can better understand the skill, apply prior knowledge related to the skill, and build confidence in their ability to apply the skill.

For example, an English language arts practice item presents a comprehension question regarding making inferences while reading. A sequence of available hints in provided for each Waggle lesson. If students need additional help, they can click for another hint. Waggle lessons have anywhere from 2 to 5 hints available to support students' learning.

In this example, the first hint that would appear on the lesson screen is:

• Hint 1: An inference is a thoughtful guess based on clues from the text and your background knowledge.

In mathematics, for example, in a practice item comparing fractions, the sequence of available hints that would appear on the lesson screen are as follows:

• Hint 1: Look at the bottom number, the dominator, in both fractions. Both fractions have 4 in the denominator, so both wholes are divided into 4 equal parts.
• Hint 2: Since the denominators are the same and the whole circles are the same size, compare the fractions by comparing the numerators.
• Hint 3: The model for ¾ has 3 parts shaded. The model for ¼ has 1 part shaded. Which model shows the greater number of shaded parts?

If Waggle detects that a student is going too fast and performing poorly, the student is encouraged to slow down and try the hints.

If a student gets a practice item wrong, Waggle offers elaborated feedback indicating why the answer was incorrect and/or helping the student approach the problem in a different way, rather than immediately revealing the correct answer. The student can then try tackling the problem again. As appropriate, Waggle sometimes offers a sequence of graduated levels of explanation. As needed, Waggle offers optional instructional support, such as video lessons to help students develop conceptual understanding. And after several incorrect tries, Waggle explains the correct answer.

Waggle also provides scaffolded support for English learners.

Text and audio support is available for challenging language, academic vocabulary, idioms, cognates, and possibly unfamiliar cultural references. Translanguaging strategies leverage students’ knowledge of their primary language to build metalinguistic awareness.

Research evidence suggests that students who persist when faced with challenging learning tasks tend to perform better academically (Darling-Hammond et al., 2019, p. 26). Related to persistence is the concept of productive struggle—figuring out something that is within reach: not too challenging to create frustration but challenging enough that effort is needed to move from an idea that is “comprehensible but not yet well formed” (Hiebert & Grouws, 2007, p. 387, citing Hiebert et al., 1996). In a review of research on teaching for conceptual understanding in mathematics, researchers found that “the engagement of students in struggling or wrestling with important mathematical ideas …consistently facilitates students’ conceptual understanding” (Hiebert & Grouws, 2007, p. 391).

Two approaches to encouraging academic persistence are supporting intrinsic motivation and fostering a growth mindset. Research on each of these approaches is presented below.

Intrinsic motivation is the human “inherent tendency to seek out novelty and challenges, to extend and exercise one's capacities, to explore, and to learn” (Ryan & Deci, 2000, p. 70). According to the research-based Self-Determination Theory and Cognitive Evaluation Theory, conditions that support intrinsic motivation include experiences that build a sense of competence (e.g., “optimal challenges,” supportive feedback, and “freedom from demeaning evaluations”) and experiences of autonomy (Ryan & Deci, 2000, p. 70; also see NASEM, 2018, p. 115). Research shows a positive relationship between intrinsic motivation and academic achievement (Lemos & Verissimo, 2014, p. 931).

• Engage students in learning challenges that are manageable and approachable—within the learner’s zone of proximal development (Darling-Hammond et al., 2019, p. 26; NASEM, 2018, p. 109; Ryan & Deci, 2000, p. 70, Stipek; 2002, p. 99).
• Offer supports to students so they “feel safe in attempting to wrestle” with a learning task (Darling-Hammond et al., 2019, p. 26, citing Lee, 2017). Helpful support includes “clear, specific, and informative feedback” (Stipek, 2002, p. 103; also see Ryan & Deci, 2000, p. 70). Encourage students to ask for help when they experience problems they can’t solve on their own (Stipek, 2002, pp. 106-107).
• Provide students with choices over their learning. When students are offered choices to give them greater control over their learning, they are more likely to show persistence in difficult task completion. Evidence suggests that even small choices, if meaningful, can support autonomy and motivation resulting in greater learning. Having choices while learning can provide a buffer against the inhibiting effects of negative feedback and may foster higher levels of engagement (NASEM, 2018, p. 117; also see Ryan & Deci, 2000, pp. 70-71).
• Appeal to student interests. Challenging learning activities can be more motivating when they are connected to issues that students believe are important, relevant to their lives, and focus on problems that are realistic and interesting (Darling-Hammond et al., 2019, p. 26).
• Emphasize mastery goals over grades and performance goals (Darling-Hammond et al., 2019, p. 27). Set short-term goals, and differentiate the goals for individual students based on their knowledge, skills, and ability (Stipek, 2002, p. 103).

According to Dweck and Leggett’s social-cognitive framework, students who have—or can develop—a growth mindset (seeing intelligence as malleable) are more motivated and persevere, resulting in more successful learning, compared to students who have a fixed mindset (seeing intelligence as a fixed state) and who more quickly give up when faced with challenges (Sarrasin et al., 2018, p. 22, citing Dweck & Leggett, 1988). Research shows that students who see their intelligence as fixed view academic challenges as a sign that they may lack intelligence, and this compromises their resilience, even among high-achieving students (Yeager & Dweck, 2012, p. 302).

Students who have a growth mindset tend to adopt mastery goals, whereas those that see intelligence as fixed focus on demonstrating competence and adopt performance goals. Research suggests that students who adhere to mastery goals are more likely to enjoy new and challenging activities, expend more effort, use higher-order cognitive skills, and are more persistent in their learning (NASEM, 2018, p. 119). A recent meta-analysis of the research concluded that having a growth mindset is “a predictor for academic achievement,” associated with higher gains in mathematics, language arts, and in general (Sarrasin et al., 2018, pp. 22–23).

A review of cross-sectional and longitudinal studies confirmed that helping students develop a growth mindset is positively correlated with their adopting mastery rather than performance goals, and with their display of effort and use of positive strategies when facing challenging learning tasks (Sarrasin et al. 2018, p. 22). Research suggests that a growth mindset can be fostered via learning and cultural influences that help students change their self-attributions and learn to view intelligence as malleable, which in turn can lead to more successful learning (NASEM, 2018, p. 111, citing Blackwell et al., 2007) and promote resilience (Yeager & Dweck, 2012, p. 306).

• Emphasize students’ goals aimed at increasing their ability, rather than proving their ability to others (Sarrasin et al., 2018, p. 22).
• Share the research-based theory that intelligence is malleable and not fixed (NASM, 2018, p. 119; Sarrasin et al., 2018 p. 22, citing Dweck & Leggett, 1988).
• Encourage the belief that effort can lead to positive outcomes (Sarrasin et al., 2018, p. 22).
• Encourage “mastery oriented responses to failure”, such as, displaying effort and positive strategies when faced with difficulty or failure (Sarrasin et al., 2018, p. 22).

HOW WAGGLE ALIGNS TO THE RESEARCH

The design of Waggle supports intrinsic motivation in multiple ways.

Waggle ensures that students are appropriately challenged without becoming frustrated through its continuously adapting, personalized practice. Waggle keeps each student in their zone of proximal development.

As described previously, Waggle offers scaffolded hints, support for English learners, explanatory feedback, and optional instructional lessons, which create an environment in which students feel safe to wrestle with the learning activity at hand. Students feel comfortable making mistakes along the way because help is always available, and they are encouraged to ask for hints when necessary.

Waggle offers students choices over their learning. The student dashboard allows them to choose to work on their assigned practice activities toward their current learning goal, return to a previous assignment, complete a skill boost (quick assessment), or play a learning game. When students experience difficulty through several consecutive practice items, they can decide whether to access a lesson or take a break.

Waggle engages students, captures their interest, and provides a sense of ownership of their experience from the start. When students log into Waggle, they select an avatar that reflects their own personalities. Then Waggle offers a choice of age-appropriate “worlds” (i.e., “Hero City”, “Oasis Outpost,” or “Planet Zebus”) and missions in which to work, based on their interests, and they can move among these worlds freely at the conclusion of a mission, lesson, goal, or learning game.

Waggle activities appeal to student interests with diverse characters, scenarios, and images that allow students to “see themselves” as they engage in learning experiences. Real life applications of concepts and skills make learning relevant to students’ lives. For example, in a mathematics lesson on place value, this concept is presented in the context of astronomy and space science.

As students work through their assignments within Waggle, their student dashboard emphasizes mastery by tracking their progress toward mastery of specific skills and learning goals.

Waggle also encourages student motivation through teacher support, even when students and teachers are working in a remote environment. For example, the Teacher Dashboard allows teachers to manage their class and easily recognize all students’ efforts, regardless of whether students are working, below, at, or above proficiency.

Waggle fosters a growth mindset.

Waggle encourages the idea that effort leads to successful learning. An on-screen character praises students for continuing to make an effort if they initially experience difficulty with a skill.

Waggle includes a text messaging system for teacher shout outs to students, and teachers are encouraged to praise students for their effort.

Students can also track their completed assignments, accomplishments, and previous work on their Student Dashboard, including recently completed skill practice activities.

Students can also see how their persistence, effort, and proficiency have unlocked various achievements. For instance, the screen below reports 1,110 points earned for persistence and effort, 6 different gems for hot streaks of proficiency, and badges for performance and mastery.

Waggle was designed with the theory of malleable intelligence in mind. This theory is discussed with teachers during Waggle professional development, and teachers are encouraged to share this idea with students. During professional development, teachers are also introduced to the features of Waggle that emphasize learning through effort.

The Waggle Mindset Report provides teachers with data on student engagement, documenting the student’s proficiency level, the number of attempts on an item, student use of hints, and the length of time spent on an item, so teachers can identify which students are exhibiting positive learning behaviors and which students might be disengaged and in need of additional teacher support.

Overall, Waggle encourages persistence and promotes a growth mindset by building students’ confidence organically and incrementally—increasing their desire to continue with a challenging activity and heightening engagement with content that sparks their interest. Waggle develops students’ understanding, and equally important, builds students’ perceptions of themselves as capable learners, who can tackle difficult content.

Formative assessment includes any student activity that provides information that can be used to adapt teaching and learning activities (Black & Wiliam, 1998a/2010). An extensive literature review found strong evidence that use of student data to adapt instruction to individual student needs is an essential component of classwork and can improve learning (Black & Wiliam, 1998a/2010, p. 1–2; Black & Wiliam, 1998b, p. 7). Strengthening formative assessment practices can produce substantial learning gains with large effect sizes³ and was found to help low achieving students more than other students, resulting in a reduction of the achievement gap while increasing achievement overall (Black & Wiliam, 1998a/2010, p. 4).

A robust meta-analysis of 21 studies found a greater effect size for studies where teachers used explicit procedures for reviewing data and adapting instruction, compared to studies where teachers had greater discretion about how to incorporate data into instruction⁴ (Black & Wiliam, 1998b, p. 15, citing Fuchs & Fuchs, 1986). This meta-analysis also found that when teachers created graphs of each student’s progress to guide their teaching, there were larger mean gains than when this did not occur⁵ (Black & Wiliam, 1998b, p. 15, citing Fuchs & Fuchs, 1986).

A growing body of research suggests that teacher-mediated instructional interventions informed by adaptive assessment data are effective in improving learning. A review of seven randomized controlled trials found that data-driven personalized literacy instruction was effective in preventing serious reading difficulties⁶ (Connor, 2019, p. 89). These trials featured adaptive assessments that automatically selected new items based on a student’s previous responses, and that provided teachers with scores immediately so they could adapt instruction and create student groups based on skill levels (Connor, 2019, p. 98). Research investigating the impact of a formative assessment system for mathematics determined that it helped teachers to differentiate instruction and had positive effects on student achievement and motivation (Faber, Luyten, & Visscher, 2017, p. 83).

Recommendations below are based on a What Works Clearinghouse (WWC) practice guide (Hamilton et al., 2009) and research cited above:

• Collect and prepare a wide variety of data about students, including classroom assessment data and non‑achievement data, to provide immediate feedback to teachers on student learning (in addition to annual and interim assessment data) (Hamilton et al., 2009, p. 12).
• Interpret data and develop hypotheses about how to improve learning, including analysis of whole class and individual strengths and weaknesses to adapt assignments, feedback, and instruction (Hamilton et al., 2009, p. 14). Such data interpretation might include creation or review of graphs of student progress (Black & Wiliam, 1998b, p. 15, citing Fuchs & Fuchs, 1986).
• Modify instruction to test hypotheses and adapt instruction by spending more time on topics or essential skills with which students are struggling, grouping and regrouping students to focus on specific skills, and trying new ways of teaching complex concepts based on best practices (Hamilton et al., 2009, p. 15).
• Provide teachers with guidance on reviewing student data and adapting instruction (Black & Wiliam, 1998b, p. 15, citing Fuchs & Fuchs, 1986).
• Utilize an adaptive assessment system, with reporting features that support differentiated instruction and organizing students for small group instruction (Connor, 2019; Faber et al., 2017).

HOW WAGGLE ALIGNS TO THE RESEARCH

Waggle reports on a wide variety of student data in real-time that gives teachers the information they need to plan and adjust instruction on an ongoing basis.

Waggle reports on student proficiency at the skill level, taking into account not only percent correct but also student learning behavior within the system.

The Waggle Teacher Dashboard’s Student Profile: Summary View provides a real-time overview of student proficiency level(s) on assigned skills and progress on skill boosts, learning goals, lessons, and learning games. Teachers can see who has completed assignments, who is making progress, and who has not started yet.

The Waggle reporting system interprets the data in ways that support improvement of learning.

The Waggle Teacher Dashboard also offers actionable insights. The teacher can immediately view the performance of their class, including which students are demonstrating proficiency and which students could benefit from skill-specific support.

The Students Need Help screen below lets teachers review the skills that challenge one or more students. Then teachers can use the Find Content tool for any one of these skills to review and assign available resources to classes, groups, or individuals.

The Teacher Dashboard also tracks instances where a student’s level of proficiency is low but productive struggle (Mindset) is high, an indication that intervention by a teacher is required.

As noted previously, the Waggle Mindset Report tracks each student’s productive struggle, taking into consideration the student’s proficiency level, the number of attempts a student has made working on the practice items or games, the number of hints accessed across all practice items and games, and the amount of time spent working on a skill.

The Waggle Growth Report provides a holistic view of the class's growth, including the class's current average proficiency level and time spent across domains. This report also includes easy-to-interpret visualizations that represent current proficiency and trends toward higher or lower levels of proficiency at a glance.

Digging deeper, the teacher can see positive or negative trends in proficiency for each assignment, along with names of the students at each proficiency level, helpful in planning follow-up small group instruction.

Waggle provides teachers with guidance on how to review students’ performance data and use this to determine next steps.

The Waggle Getting Started Guide and implementation resources introduce teachers to the Waggle Teacher Dashboard and essential reports, as well as guidance on how to use these tools to make data-informed instructional decisions. Waggle professional learning support guides teachers through the process of making instructional assignments and decisions about small group instruction based on real-time data.

Waggle is both an adaptive assessment system and an adaptive learning system.

Every Waggle session a student completes contributes to an ongoing formative assessment picture. The results are automatically and continuously used to determine the most appropriate, personalized guided practice and instruction for each student, while also providing teachers with the data and analysis they need to make good decisions about whole class and small group instruction.

Waggle is a digital mathematics and English language arts solution based on educational research and learning science to supplement and support the work of dedicated teachers.

Waggle offers personalized practice and instruction through a continuously adaptive digital learning system. Based on a student learning model that includes multiple data points and teacher input, the system develops a detailed picture of each student’s knowledge and approach to learning. Waggle employs this analysis to provide an intelligent tutoring system built on each student’s strengths, weaknesses, and learning patterns. In these ways, Waggle aligns with research-based recommendations on effective adaptive learning systems.

Waggle provides retrieval practice to support long-term mastery and application of knowledge and skills. Consistent with the research on effective retrieval practice, Waggle enables teachers to assign spaced, interleaved practice that incorporates detailed hints as scaffolding support for students who get “stuck,” and elaborated and corrective feedback for students who answer incorrectly.

Waggle fosters student persistence and productive struggle by supporting intrinsic motivation and encouraging a growth mindset. Following research-based recommendations, Waggle supports intrinsic motivation through its continuously adaptive practice that challenges students without frustrating them, by offering choices over their learning activities, by appealing to student interests and making learning relevant to their lives, and by emphasizing mastery of specific skills and learning goals. The system also adheres to research and expert opinion on fostering a growth mindset by encouraging the idea that effort leads to successful learning and by rewarding students to persevere when they initially experience difficulty with a skill.

Finally, Waggle is both an adaptive learning system and an adaptive, formative assessment system. In alignment with research on formative assessment and data-based decision making, the Waggle Teacher Dashboard and reports provide real-time student data that inform teachers’ planning, and offer insights enabling teachers to make ongoing adjustments to instruction. Teachers get the data and analysis they need to make solid decisions about whole class and small group instruction.

1. Knewton, Inc.
2. Average effect size was d = 79
3. Effect sizes ranged from 0.4 to 0.7.
4. Mean effect size was 0.92 for teachers using explicit procedures versus 0.42 for teachers with greater discretion.
5. Mean effect size was 0.70 for teachers creating graphs versus 0.26 for teachers not creating graphs.
6. An effect size of 0.7 was found for gains over a three-year span from Grades 1 through 3 .

Jay Sivin-Kachala, Ed.D., Executive Vice President, Interactive Educational Systems Design

Bonnie Louque, Director, Content & Curriculum, Waggle, Houghton Mifflin Harcourt

Ryan Parker, Product Owner, Waggle, Houghton Mifflin Harcourt

Renee Behring, Ed.M., Education Research Director, Houghton Mifflin Harcourt

Danielle Forbes, Ph.D., Education Research Manager, Houghton Mifflin Harcourt

Agarwal, P. K., & Bain, P. M. (2019). Powerful teaching: Unleash the science of learning. San Francisco: Josey-Bass.

Aleven, V., McLaughlin, E. A., Glenn, R. A., & Koedinger, K. R. (2017). Instruction based on adaptive learning technologies. In R. E. Mayer & P. Alexander (Eds.), Handbook of research on learning and instruction (2nd ed., pp. 522–560). New York: Routledge.

Black, P. & Wiliam, D. (1998a, October/2010, September). Inside the black box: Raising standards through classroom assessment. Phi Delta Kappa, 92(2),81-90. https://doi.org/10.1177/003172171009200119

Black, P. & Wiliam, D. (1998b). Assessment and classroom learning. Assessment in Education: Principles, Policy & Practice, 5(1), 7-74.

Connor, C. M. (2019). Using technology and assessment to personalize instruction: Preventing Reading Problems, 20(1), 89–99. https://doi.org/10.1007/s11121-017-0842-9

Darling-Hammond, L., Flook, L., Cook-Harvey, C., Barron, B., & Osher, D. (2019). Implications for educational practice of the science of learning and development, Applied Developmental Science, 1–44. https://doi.org/10.1080/10888691.2018.1537791

Faber, J. M., Luyten, A. J., & Visscher, A. J. (2017, March). The effects of a digital formative assessment tool on mathematics achievement and student motivation: Results of a randomized experiment. Computers & Education, 106, 83–96. https://doi.org/10.1016/j.compedu.2016.12.001

Hamilton, L., Halverson, R., Jackson, S., Mandinach, E., Supovitz, J., & Wayman, J. (2009). Using student achievement data to support instructional decision making (NCEE 2009-4067). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. https://ies.ed.gov/ncee/wwc/Docs/PracticeGuide/dddm_pg_092909.pdf

Hattie, J., & Mark, G. (2010). Instruction based on feedback. In R. E. Mayer & P. A. Alexander (Eds.), Handbook of research on learning and instruction. New York: Routledge.

Hiebert, J., & Grouws, D. (2007). The effects of classroom mathematics teaching on students’ learning. In F. K. Lester (Ed.), Second handbook of research on mathematics teaching and learning (pp. 371–404). Greenwich, CT: Information Age. https://pdfs.semanticscholar.org/3b2e/2aabd07c64bb65408a3891902be4b7277cd6.pdf

Lemos, M. S., & Verissimo, L. (2014). The relationships between intrinsic motivation, extrinsic motivation, and achievement, along elementary school. Procedia – Social and Behavioral Sciences, 112, 930–938

Ma, W., Adesope, O. O., Nesbit, J. C., & Liu, Q. (2014). Intelligent tutoring systems and learning outcomes: A meta-analysis. Journal of Educational Psychology, 106(4), 901–918. https://doi.org/10.1037/a0037123

National Academies of Sciences, Engineering, and Medicine (NASEM). (2018). How people learn II: Learners, contexts, and cultures. Washington, DC: The National Academies Press. https://doi.org/10.17226/24783

Ryan, R. M. & Deci, E. L. (2000, January). Self-Determination Theory and the facilitation of intrinsic motivation, social development, and well-being. American Psychologist, 55(1), 68-78

Sarrasin, J. B., Nenciovici, L., Foisy, L.-M. B., Allaire-Duquette, G., Riopel, M., & Masson, S. (2018). Effects of teaching the concept of neuroplasticity to induce a growth mindset on motivation, achievement, and brain activity: A meta-analysis. Trends in Neuroscience and Education, 12, 22–31. https://doi.org/10.1016/j.tine.2018.07.003

Shute, V. J. (2008). Focus on formative feedback. Review of Educational Research, 78(1), 153–189.https://doi.org/10.3102/0034654307313795

SRI International (2018). Using technology to personalize learning in K–12 schools. SRI International, Menlo Park, CA. https://www.sri.com/sites/default/files/publications/using-technology-personalize-learning-k-12-schools.pdf

Stipek, D. (2002). Motivation to learn: From theory to practice (4th ed). Needham Heights, MA: Allyn & Bacon.

Van der Kleij, F. M., Feskens, R. C. W., & Eggen, T. J. H. M. (2015). Effects of feedback in a computer-based learning environment on students’ learning outcomes. Review of Educational Research, 85(4), 475–511. https://doi.org/10.3102/0034654314564881

Yeager, D., & Dweck, C. (2012). Mindsets that promote resilience: When students believe that personal characteristics can be developed. Educational Psychologist, 47, 302–314. https://doi.org/10.1080/00461520.2012.722805