Research Summary
September 2023
Summary of Research on onebillion’s Learning Software
A Substantial Body of Rigorous Evidence
At least nine randomized controlled trials (RCTs) have been conducted on onebillion‘s learning software in different countries, languages, and settings (e.g., in school, out of school, and refugee camps). These RCTs have consistently shown that onebillion’s software produces positive and significant learning gains in literacy and numeracy. The attached Table 1 summarizes key details of these studies.
Imagine’s Research
Imagine Worldwide conducted five of the nine RCTs mentioned above; detailed results are presented in Table 2. We built upon the early proof-of-concept RCTs conducted by the University of Nottingham over 8 weeks and 14 weeks to investigate whether the observed short-term impacts would be sustained over longer periods of time (i.e., over 8 months and over 2 school years). Imagine’s interest is not just in producing a statistically significant increase in learning that might translate into a few more letters, words, or numbers, but ultimately in helping children to become readers and to attain a similar skill level in mathematics (i.e., attaining end of 4th-grade standards). We also sought to test whether the tablet program could produce positive learning impacts in refugee settings, which include some of the most marginalized children. Finally, we replicated studies conducted in Malawi, in government schools and a refugee camp, in Tanzania. Over 4 years, our RCTs used three different versions of onebillion’s software, with results indicating increasing impacts with successive software versions. Further, our studies to date have shown that girls benefit at least as much as boys through the tablet program.
Although average efficacy impacts have been consistently positive and significant, we also found that a substantial proportion of learners remained non-readers despite using onebillion’s software. About 40% of learners in both the Global Learning XPrize RCT and Imagine’s 2-year RCT remained non-readers after using the tablet program for 15 months and 13 months, respectively. Being a “non-reader” means not being able to read a single word of connected text (that is, text presented as sentences in a paragraph). To understand factors influencing non-progress in reading, we conducted an initial exploratory study that showed that children’s visual working memory skills uniquely predicted high versus low progress in reading over and above some other known predictors of early literacy development (such as home literacy and learning environment and age). We investigated instructional strategies that have been shown to be effective in addressing working memory deficits, and shared these with onebillion to inform software improvements.
As we conclude our initial efficacy research phase, we are now focusing our research on supporting continuous improvement--of the software and of implementation practices in the field. We will be extending our research on non-progress in reading during the next couple of years and will conduct additional research to understand better for whom the software is working, for whom it is not, and why. We are also conducting implementation research to understand barriers to and enablers of quality program implementation as we expand.
Research Reports and Presentations
Imagine’s studies have met rigorous design and analysis standards. Our studies were shaped and reviewed by our founding research advisory group of highly respected experts as well as by our research partners. All of our RCTs have met baseline equivalence standards for the final analytic samples, supporting the causal validity of the findings.
Imagine’s Research
“Tablet-based Learning for Foundational Literacy and Math: An 8-month RCT in Malawi.” (2020). Imagine Worldwide Technical Report. Link.
“2-year RCT of EdTech in Malawi.” (2022). Imagine Worldwide Research Brief, No. 1. Link.
“2-year RCT of EdTech in Malawi.” (2022). Imagine Worldwide Research Brief, No. 1. Link.
“Technology-enabled Learning in a Second Language: A Proof-of-Concept RCT with Refugee Children in Malawi” [Conference presentation]. CIES 2021. Available on request.
“Autonomous Learning for Supplemental Education in Tanzania (ALSET): Independent Efficacy RCTs.” (2023). Summary of Findings. Available on request.
“RCT evidence and continuous improvement of an EdTech intervention” [Conference presentation]. mEducation 2023. Available on request.
“An exploratory analysis of divergent patterns in reading progression during a tablet-based literacy program.” In Frontiers in Education, 23 February 2023. Link.
Selected Additional Research
- Pitchford (2015). “Development of early mathematical skills with a tablet intervention: a randomized control trial in Malawi.” Available here.
- Outhwaite et al. (2017). “Closing the gap: Efficacy of a tablet intervention to support the development of early mathematical skills in UK primary school children.” Available here.
- Pitchford et al. (2018). “Interactive Apps Promote Learning of Basic Mathematics in Children With Special Educational Needs and Disabilities.” Available here.
- Outhwaite et al. (2019). “Raising Early Achievement in Math with Interactive Apps: A Randomized Control Trial.” Available here.
- King et al. (2019). Global Learning XPRIZE Data Summary. RTI International presentation.
Table 1. Rigorous Studies of onebillion's Software
| RCT Brief Name | Research organization | Year concluded | Country | Setting | Treatment duration | Target Pop. | Software version* | Subject | Language | Daily Dosage | Effect sizes (statistically significant) | ||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Math | Literacy | ||||||||||||
| 1 | Proof of Concept Math | U Nottingham | 2013 | Malawi | In school | 8 weeks | Grades 1-3 | Proc. v1 | Math | Chichewa | 1 hour/day | Overall math = .63 | - |
| 2 | Proof of Concept Literacy | U Nottingham | 2017 | Malawi | In school | 14 weeks | Grades 1-3 | Proc. v1 | Literacy | Chichewa | 1 hour/day | - | Overall literacy = .42 |
| 3 | 8-month RCT | Imagine | 2019 | Malawi | In school | 8 months | Grade 2 | Proc. v1 | Literacy or Math | Chichewa | 40 min/day literacy OR math | Early math = .15 - .29 | Overall literacy = .34 |
| 4 | Global Learning XPrize | RTI | 2019 | Tanzania | Out of school | 15 months | 9-11 year-olds | Community | Literacy & Math | Swahili | Unlimited | Math various = .56 - 1.09 | Literacy various = .32 - .95 |
| 5 | Refugee Camp Math RCT | Imagine | 2020 | Malawi | Refugee camp-informal school | 5 months | Preschool 3-Grade 3 | Proc. v1 | Math | English | 20 min/day | Early math = .36 | - |
| 6 | Multi-country Study | U Nottingham | 2021 | Ethiopia, Kenya, Malawi | In school | 12 weeks | Grade 1 | Proc. v1 | Math | Multiple | 30 min/day | Overall math = .28 - .39 | - |
| 7 | 2-year RCT | Imagine | 2021 | Malawi | In school | 13 months | Grades 2-3 | Proc. v2 | Literacy or math | Chichewa | 40 min/day literacy OR math | Overall math = .54 | Overall literacy = .37 |
| 8 | Refugee Camp RCT | Imagine | 2022 | Tanzania | Summer break | 8 weeks | Grades 1-3 | Adaptive v1 | Literacy & math | Swahili | 40 min/day combined | Overall math = .44 | Overall literacy = .20 |
| 9 | 5-month RCT | Imagine | 2022 | Tanzania | After school | 5 months | Grades 1-3 | Adaptive v1 | Literacy & math | Swahili | 40 min/day combined | Overall math = .26 | Overall literacy = .34 |
*Procedural v1 was the original self-paced version; v2 had more content. Community was created for XPrize. Adaptive v1 had the same content as Procedural v2 but used diagnostic tests to deliver content at the right level.
References:
1 Pitchford (2015). Development of early mathematical skills with a tablet intervention: a randomized control trial in Malawi. Frontiers in Psychology, 6:485. 2 Pitchford, Hubber, and Chigeda (2017). Unlocking Talent through Technology: Improving Learning Outcomes of Primary School Children in Malawi. Report prepared for Vountary Service Overseas. Author. 3 Levesque, Bardack, and Chigeda (2020). Technical Report: Tablet-based Learning for Foundational Literacy and Math: An 8-month RCT in Malawi. Imagine Worldwide: Research Brief No. 1. https://www.imagineworldwide.org/wp-
Table 2. Imagine RCTs of onebillion's software: Impact results
| Government primary schools | Refugee camp schools | |||||
|---|---|---|---|---|---|---|
| Setting | 1 | 2 | 3 | 4 | 5 | |
| Study # | 8-month | 2-year | TZ-Host | Dzaleka | TZ-Camp | |
| RCT short name | 2 | 2 | 1 | 1 | 1 | |
| Number of schools in study | 674 | 578 | 300 | 317 | 300 | |
| Number of learners in study(1) | ||||||
| Intervention description | Malawi Government | Malawi Government | Tanzania Government | Malawi Private Proc. v1 | Tanzania Private Adaptive v1 | |
| Country | Proc.v1 | Proc.v2 | Adaptive v1 | Proc.v1 | Adaptive v1 | |
| School authority | 8 months | 13 months | 5 months | 5 months | 8 weeks | |
| Software version(2) | School day Chichewa | School day Chichewa | After school Kiswahili | School day English Math only | School break Kiswahili | |
| Intervention duration | Read or Math | Read or Math | Read + Math | Read + Math | Read + Math | |
| Implementation model | 40 min | 40 min | 40 min | 20 min | 40 min | |
| Software language | ||||||
| Subject(s) | ||||||
| Daily tablet usage time | 40 min | 40 min | 40 min | 20 min | 40 min | |
| Total time on task(in hours) per subject for treatment group | 56 | 101 | 20 | 19 | 8 | |
| Literacy impact results | ||||||
| Intent-to-Treat | ||||||
| Effect size for overall literacy (EGRA average % correct) | 0.34* | 0.37* | 0.34** | - | 0.20+ | |
| % added value(3) | 66% | 32% | 51% | - | 56% | |
| Added value in weeks or months(4) | 5.3 mos | 4.2 mos | 2.6 mos | - | 4.5 wks | |
| Rate of advancing on reading benchmark vs. control | 1.50x | 1.50x | 1.80x | - | 1.53x | |
| % of treatment group attaining emergent or fluent status | 9% | 29% | 11% | - | 12% | |
| Treatment-on-the-Treated | ||||||
| Percent of treatment group attending at least 50% of time | 88% | 77% | 71% | - | 67% | |
| Effect size (s.d.’s)-EGRA average % correct | 0.40* | 0.47* | 0.49** | - | 0.29+ | |
| Mathematics impact results | ||||||
| Intent-to-Treat | ||||||
| Effect size for overall math (EGMA average % correct) | 0.07(5) | 0.54** | 0.26* | 0.02(5) | 0.44*** | |
| % added value(3) | 7% | 28% | 67% | 5% | 239% | |
| Added value in weeks or months(4) | 0.6 mos | 3.6 mos | 3.4 mos | 0.2 mos | 19 wks | |
| Rate of advancing on math benchmark vs. control | 1.40x | 1.40x | 1.92x | 1.30x | 1.14x | |
| % of treatment group attaining emergent or fluent status | 19% | 72% | 12% | 46% | 79% | |
| Treatment-on-the-Treated | ||||||
| Percent of treatment group attending at least 50% of time | 88% | 83% | 71% | 70% | 67% | |
| Effect size (s.d.’s)-EGMA average % correct | 0.07 | .65*** | 0.37* | 0.03 | 0.65*** |
+Statistically significant at p<0.10 level, * at p<0.05 level, ** at p<0.01 level, and *** at p<0.001 level.
(1)Studies 1&2 each had two treatment groups (literacy and math) and one control group resulting in three groups of (roughly) equal size, while studies 3-5 each had one treatment group and one control group of (roughly) equal size. (2) "Procedural v1" was the original self-paced version; v2 had more content. "Adaptive v1" had the same content as Procedural v2 but used diagnostic tests to deliver content at the right level. (3) % added value is derived by dividing the treatment effect by the control group effect. Control group effects = (average control group post-test minus the average control group pre-test) divided by the average standard deviation of the pre- and post-test scores. (4) Added value in terms of weeks or months is derived by multiplying the % added value by the number of weeks or months of the intervention.