The results could open a new front in the evolving discussion over how to use AI in schools 鈥� and how closely humans must watch it when it鈥檚 interacting with kids.

In a involving 165 British secondary school students, ages 13鈥�15, the ed-tech startup put a small group of expert human tutors in charge of a , or LLM, offered by Google鈥檚 . As it tutored students on math problems via Eedi鈥檚 platform, it drafted replies when students needed help. Before the messages went out, the human tutors got a chance to revise each one to the point where they鈥檇 feel comfortable sending it themselves.

Students didn鈥檛 know whether they were talking to a human or a chatbot, but they had longer conversations, on average, with the 鈥渟upervised鈥� AI/human combination than simply with a human tutor, said Bibi Groot, Eedi鈥檚 chief impact officer. 

In the end, students using the supervised AI tutor performed slightly better than those who chatted online via text with human tutors 鈥� they were able to solve new kinds of problems on subsequent topics successfully 66.2% of the time, compared to 60.7% with human tutors.

The AI, researchers concluded, was 鈥渁 reliable source鈥� of instruction. Human tutors approved about three out of four drafted messages with few to no edits.

Students who got both human and AI tutoring were able to correct misconceptions and offer correct answers over 90% of the time, compared to just 65% of the time when they got a 鈥渟tatic, pre-written鈥� response to their questions.

And the AI only 鈥渉allucinated,鈥� or offered factual errors, 0.1% of the time 鈥� in 3,617 messages, that amounted to just five hallucinations. It didn鈥檛 produce any messages that gave the tutors pause over safety.

The results suggest that 鈥減edagogically fine-tuned鈥� AI could play a role in delivering effective, individualized tutoring at scale, researchers said. Interestingly, students who received support from the AI were more likely to solve new kinds of problems on subsequent topics. 

The key to the AI鈥檚 success, said Groot, was that researchers gave it access to detailed, 鈥渆xtremely personalized鈥� information about what topics students had covered over the previous 20 weeks. That included the topics they鈥檇 struggled with and those they鈥檇 mastered. 

鈥淲e know what topics they’re covering in the next 20 weeks 鈥� we know the curriculum. We know the other students in the classroom. We know whether they’re putting effort into their questions. We know whether they’re watching videos or not 鈥� we know so much about the student without passing any personally identifiable information to the AI.鈥�

That guided the AI鈥檚 strategy about whether students needed an extra push or just more support 鈥� something an 鈥渙ut-of-the-box, vanilla LLM鈥� can鈥檛 do, she said.

鈥淭hey don’t know anything about what the teacher is teaching in the classroom,鈥� Groot said. 鈥淭hey don’t know what misconceptions or what topics the students are struggling with and what they’ve already mastered, so they’re not able to dynamically change how they address the topic, as a human tutor would.鈥�

Human tutors, she said, generally have 鈥渁 really good sense of where the student struggles, because they have some sort of ongoing relation with a student most of the time. An LLM tutor generally doesn’t.鈥�

All the same, even master tutors typically don鈥檛 go into a session knowing a student鈥檚 comprehensive history in a course, including their misconceptions about the material. 鈥淎ll of that is too much information for a human tutor to read up on and deal with while they’re having one conversation鈥� with a student, Groot said.

And they鈥檙e under pressure to respond quickly 鈥渟o that the student is not left waiting. And that’s quite an intensive experience for tutors that leads to a bit of cognitive overload,鈥� she said. The AI doesn’t suffer from that. It needs less than a millisecond to read all of those contexts and come up with that first question.鈥�

Even with their personal connection to students, human tutors can鈥檛 be available 24/7. Groot said Eedi employs about 25 tutors across several time zones who are available to students from 9 a.m. to 10 p.m. every day, but to give students broader access would require hiring 鈥渁n army of tutors,鈥� she said.

The new findings could encourage schools to use AI as a kind of 鈥渇ront line鈥� tutor, with humans intervening when a student is 鈥渄erailing the conversation, or they have such a persistent misconception that the AI can’t deal with it,鈥� said Groot. 鈥淲e think that would be an interesting way to collaborate between the AI and the human, because there is still a really important role for a human tutor. But our human tutors just cannot have conversations with thousands of students at once.鈥�

The new study, published last week on Eedi鈥檚 site and scheduled to appear in a peer-reviewed journal next year, differed in one important way from recent studies that looked at AI tutoring. Researchers at in October 2024 examined AI-assisted human tutoring, in which tutors primarily drove the conversation. But in that case, the AI acted as a kind of assistant, providing suggestions behind the scenes. In the Eedi study, it was the other way around, with AI driving the conversation and humans overseeing it.

Robin Lake, director of the at Arizona State University, said the study is important in and of itself, but also in the context of broader findings elsewhere suggesting that, with proper training and guidance, 鈥淎I can be an incredibly powerful tool 鈥� and certainly has a potential to take tutoring to scale in ways that we’ve never seen before.鈥�

Under controlled circumstances, she said, it鈥檚 also 鈥渙utperforming humans 鈥� that’s really important.鈥�

Lake noted a from Harvard researchers that examined results from 194 undergraduates in a large physics class. They presented identical material in class and via an AI tutor and found that students learned 鈥渟ignificantly more in less time鈥� using the tutor. They also felt more engaged and motivated about the material.

Liz Cohen, vice president of policy for 50CAN and author of the recent book , said the study provides 鈥渧aluable evidence鈥� about new kinds of tutoring. 

But one of its limitations, she said, is that it relied on 13-to-15-year-olds. 鈥淪o immediately I have a lot of questions about if the findings are applicable for younger students, especially using a chat based model,鈥� which may not be a good one for such students.

She also noted that there are many questions around student persistence with AI tutors, including what happens when students get frustrated or aren鈥檛 sufficiently engaged in the work? 

鈥淚 still mostly think that entirely AI tutoring programs are biased towards students who want to do the work or are interested in learning,鈥� Cohen said, 鈥渁nd it鈥檚 pretty easy to see that students who aren鈥檛 bought in or are frustrated are going to give up more readily with an AI tutor.鈥�

She noted that her 12-year-old daughter has experienced problems persisting in an AI-powered math tutoring program. 鈥淪he gets frustrated if she can鈥檛 get the answer and then she doesn鈥檛 want to do it anymore, so I think we need to figure out that piece of it.鈥�