The homework went faster and the answers got better. But when the students had to show their knowledge in tests, the results were significantly worse. That was the main message in a study of nearly 27,000 Chinese schoolchildren, carried out by researchers at Stockholm University and the University of Hong Kong.
According to the study, homework time was reduced by about 30 percent when students started using generative AI. At the same time, the results on homework rose by 18 percent. On the tests in the classroom, however, the negative effect was around 20 percent within six months.
The conclusions were widely disseminated internationally and were noticed by, among others, The Economist and the New York Times. On the World Bank's blog, the study was described as "thorough and rigorous" - and a wake-up call for parents and teachers.
Now suspicions are being directed at the basis for the results themselves.
Recently, an anonymous whistleblower contacted Stockholm University and questioned whether the study's data is genuine. In a detailed report, the person points to several patterns that are difficult to explain if the data comes from real students and tests.
Among other things, students are said to have started using AI before the named tools were available. Results in different school subjects also appear to have a remarkably weak relationship. Students who perform well in one subject often do so in others as well - but in the study's data, this connection is barely visible, according to the informant.
In addition, not a single test result is missing in parts of the extensive material. The complainant questions how it is possible that no students, for example, were ill at any time of the exam.
The anonymous person claims to work as a national economist at the AI company Anthropic and writes himself that this may involve a conflict of interest.
Last week, Stockholm University reported the research to the Board for Examination of Improper Research in Research as "suspected forgery".
The application has been made by Mårten Palme, professor of economics at the Department of Economics at Stockholm University. He is also head of department at the department. In his report, he describes the tip as "concrete and well-founded". According to the report from Mårten Palme, especially the weak connection between the students' results in different subjects has also led the co-author at Stockholm University to suspect that the data is not genuine.
The study was published in June. It has not yet been published in any scientific journal, but has thus been widely disseminated in both Swedish and international media.
David Strömberg, professor of economics at Stockholm University, is one of the authors behind the criticized study. He tells DN that he thinks it is good that it is reported.
- We are trying to sort this out and see what has happened.
Do you have any explanation for how this could have happened?
- I don't want to comment on that. We are investigating it and I don't know.
Who has been in charge of the data collection?
- I don't want to go into it. We have received data from an authority in China and we have documents from there that certify authenticity. I don't want to go into further how it happened.
How serious do you think this is?
- It is very serious. After all, the essay has not been published. But we have posted it as a research report and that research report has been circulated before publication.
