Study at a Glance:
The COVID-19 pandemic led to substantial unfinished learning in math, exacerbating longstanding equity gaps in student math achievement. Educators now face the notable challenge of how best to teach grade level content while also helping their students catch up. Digital math products that supplement core teacher instruction are a promising option with multiple studies showing that the use of these products can improve students’ math achievement. Yet, there is still much to learn about the efficacy of two different instructional approaches that these products can take to help students catch up. The ReSolve Math Study aims to accomplish two learning goals:
(1 ) Provide evidence about which instructional support approach is most effective in helping to catch up elementary math students who start the year behind.
(2) Learn which instructional support approach works best for which students. For example: students who begin the year especially behind, students of color, English learners, and students with disabilities.
The U.S. Department of Education has commissioned MDRC and its partners at Digital Promise, Public Strategies, RAND, and Westat to conduct the ReSolve Math Study. The study team is currently developing partnerships with eligible districts and elementary schools to implement one of two digital math products with 4th and 5th grade students for two years and to participate in research activities aligned with the learning goals described above.
There will be two cohorts of schools participating in the study. Both cohorts will implement the products and engage in data collection for two school years. The second cohort will start in Fall 2024.
The study aims to provide much needed evidence on a debate in mathematics education: whether students’ mathematics proficiency—in both the short and long-term—is better served by instruction that systematically supports building mastery in unfinished learning or by instruction that is highly targeted to unfinished learning in just the prerequisite concepts thought to be essential to succeed in grade level content. By assessing these two different instructional approaches as delivered via digital math products, the study will provide much needed evidence about whether both approaches are equally effective and whether their effectiveness varies for different types of students or the extent of the students’ unfinished learning.
All public schools that meet the following criteria are eligible for the study:
- Serve grades 4 and 5, the grade levels in which the digital math products will be implemented.
- Have high academic need in math: The study defines “high academic need” as having 50% or more of 4th graders scoring below proficient in state math assessment OR at least 30 students in 4th grade who scored below proficient on the state math test.
The study’s development team will work directly with the district to identify and select schools that meet the study’s eligibility criteria and requirements and that have the capacity to participate in the study’s data collection activities. The study team will work with district administrators and selected schools to ensure that the schools have the appropriate infrastructure to support the use of digital math products (e.g., student computer access and reliable internet access) and can meet the study’s expectation for 60 minutes of student usage per week.
Digital Math Products:
The products selected for the study are Curriculum Associates’ i-Ready and Personalized Instruction and Renaissance Learning’s Star Math Assessment and Freckle. The products were selected through a competitive process that involved careful review from a panel of math education and education technology experts. In addition to other criteria, the products were selected for their promise in addressing unfinished learning in math and their alignment with the following core principles:
- Deliver math content covering key 4th and 5th grade standards and prerequisite skills and knowledge from previous grades.
- Frequent evaluation of unfinished learning to adapt content to each student’s need. Products have a methodology for identifying or diagnosing students’ unfinished learning in math, directing each student to math content that addresses their specific needs, reassessing if learning needs were addressed, and directing each student to the next appropriate content area.
- Use of instructional practices that align with the What Works Clearinghouse Practice Guide for Assisting Struggling Students with Mathematics (Fuchs, 2021), provide engaging and motivating content to improve each student’s math achievement, and provide them with timely feedback on their performance.
- Inclusion of progress monitoring systems that continuously collect student data and provide dashboards or reports on individual student, classroom, and school usage and progress.
Products are designed to seamlessly complement teacher-led instruction and do not require that teachers change what or how they teach their core curriculum. Both products can deliver the two differentiated instructional approaches described above – all with the goal of helping struggling students to catch-up to grade level standards as quickly as possible.
Participating schools will receive free access to the digital math product for two years for all of their 4th and 5th grade students.
District administrators know their schools’ and students’ needs best. Participating districts get to express their preference for the use of either Curriculum Associates’ i-Ready and Personalized Instruction or Renaissance Learning’s Star Math Assessment and Freckle. The study team will attempt to give districts their first-choice product. All participating schools in a district will use the same product.
4th and 5th grade teachers in study schools will receive up to six hours of in-person or virtual training from their district’s selected vendor on how to use the product. Additionally, up to two school-level administrators with math curriculum and instruction oversight will receive training on how to support the use of the product in their schools. The initial trainings will take place in summer and fall 2024, and will be coordinated by the product vendor and the study team with each participating district. A second training will be provided in summer 2025.
The focus of the trainings will be on the expectations for product usage and will not ask teachers to change their approach to core instruction – other than making time during their math instruction period for students to use the digital math product. The trainings will include familiarization with platform and content, software use and troubleshooting, as well as administrative tasks. The trainings will also include instruction on how to access information on students’ usage and progress on the product, including recommendations for how to support students who may not be actively participating with the product or who may not be successfully progressing through the material.
Additionally, each product will offer opportunities during the school year for teachers to access additional training and technical assistance.
Each product vendor will provide support to school and/or district IT staff regarding integration of the product into district and school platforms so that it is usable by students. The vendor will conduct virtual product orientation sessions with district and school leaders prior to the start of the 2024 school year and will be available to provide technical assistance as needed.
The product will be used during the regular classroom math instructional time for approximately 60 minutes per week. There is flexibility for the teacher to decide how to distribute product time over the course of the week.
These products are intended to complement teacher instruction, not replace it. These products can be seen as a tool to make teachers’ lives easier in the classroom by providing high quality personalized math content to accelerate each student’s learning. Furthermore, the data produced by the product assessments can support teachers in more effectively using data to monitor student progress and plan instruction.
Research Activities:
This study is testing the relative effectiveness of two different approaches to supporting students' math learning. The main research questions are as follows:
(1 ) Are the two approaches to catch-up instruction equally effective at improving struggling students’ learning?
(2) Are the products and learning approaches particularly effective for certain groups of students?
(3) Were the products implemented with fidelity?
(4) Are the products cost effective?
The study will consider effects on all 4th and 5th grade students but is particularly focused on students performing below grade level at the start of the school year.
To evaluate which of the approaches to supporting students’ math learning is most effective and in which ways, the study will use a lottery to assign all 4th and 5th grade students in the study schools to receive one of the two instructional approaches to catching students up to grade level standards that the digital product offers. The study will compare the outcomes of students receiving each approach to each other. The study will consider outcomes for all 4th and 5th grade students in participating schools but have a special focus on the outcomes of students who start the year below grade level.
All students in a math classroom will be using the same product. The standards-aligned content and instructional activities that the products draw from are the same for all students. The products are designed to be adaptive and to personalize the student learning based on each student’s needs. As a result, each student’s pathway may be unique, but in a strategic way that is meant to be optimized to their individual needs and aligned with the study’s research goals to test different ways of catching students up to grade level standards.
The study plans to collect the following types of data about students and staff in all participating schools:
- School administrator survey: Survey of school staff member(s) most knowledgeable about math instruction and education technology usage at the school to understand school context.
- Teacher survey: Survey of 4th and 5th grade teachers to understand math instructional practice and mindset.
- District student records: The study team will work with districts to collect information about 4th and 5th grade students in participating schools including their demographic characteristics and their performance on state standardized math tests before the study started (SY 2023-2024) and at the end of each implementation year. The study may also request follow-up data, to the extent available, on these students’ math outcomes in later years including state standardized test scores and math course-taking in middle and high school.
- Data from the digital math product: The study team will work with the digital math product vendors to collect information about how students and teachers use the product.
Data collected by the study team will only be used for research purposes. All information will be kept strictly confidential. We will follow IES confidentiality and data protection requirements (Education Sciences Reform Act of 2002, Title I, Part E, Section 183). The public reports prepared for the study will summarize findings across the participating schools and districts and will not associate responses with a specific school or individual. We will not provide information that may identify any individual study participant to anyone outside the study team, except as required by law. Any potential reports on district-specific findings would only be shared with the relevant district, would be marked confidential, and would not permit identification of individual students. Participation in study-related data collection activities will be voluntary. Moreover, we will follow both MDRC’s and school districts’ research office procedures that provide guidance and regulation regarding the rights of study participants and confidentiality. Our encryption, data-access, and other security procedures often exceed these requirements.
Study Participation:
- Two years of free access to an adaptive digital math product for all 4th and 5th grade students in all study schools.
- Free training for 4th and 5th grade teachers and school leaders in study schools on how to use the product. They will also have access to technical assistance from the product vendors and Digital Promise to support high-quality implementation of the digital math products with students.
- Ongoing implementation and technical support from the product vendors and Digital Promise to support high-quality implementation of the digital math products with students.
- Funds to compensate teachers in study schools for their time spent in training.
- Funds for districts and schools to help coordinate study activities, including data collection. Districts can receive between $4,000 and $15,000 per year (depending on the number of participating schools) to offset costs associated with efforts to facilitate study activities (e.g., scheduling trainings; collaborating with the study team to support data collection).
- Each participating school will receive $1,000 per year to compensate for coordination of data collection activities, and individual participants in data collection activities (e.g., teacher surveys) will be offered a small stipend for participation.
- Support schools’ participation in the data collection activities (e.g., surveys, if needed).
- Agree to the lottery assignment of 4th and 5th grade students to different product modes.
- Partner with the vendors and study team to encourage and support participating schools to implement the products as intended (e.g., student use of the product during the school-day for 60 minutes per week).
- Agree to set aside >= 60 minutes weekly during math class time for each student to work with the product.
- Agree to the student level lottery assignment of 4th and 5th grade students to one of the two product modes.
- Agree to the project’s data collection activities.
- Agree to the project’s training activities for teachers in the program group schools.
You can contact the ReSolve Math Study team at ReSolveMath@mdrc.org or reach out directly to the Deputy Project Director, Dr. Barbara Condliffe at barbara.condliffe@mdrc.org or the Project Director Dr. William Corrin at william.corrin@mdrc.org.
