Artificial intelligence (AI) based on large language models (LLMs) has demonstrated high accuracy in identifying the reliability of scientific data, outperforming even experienced neuroscientists.
A study conducted by scientists from University College London (UCL) and published in the journal Nature Human Behaviour showed that LLMs are able to recognize fake data and assess the validity of scientific results. The experiment used the BrainBench tool, which compares pairs of scientific abstracts: one containing real results, the other – modified but plausible. 15 language models and 171 neuroscience experts participated in the test.
The results showed that the AI achieved an average accuracy of 81%, while the experts achieved only 63%. The specialized BrainGPT model, developed on the basis of the open VMM Mistral and adapted to neurobiological data, showed an accuracy of 86%, which is superior to the general-purpose models. Unlike humans, the confidence of the AI in its conclusions correlated with the correctness of the answers, increasing trust in such tools.
Scientists note that AI can help not only in the analysis of existing data, but also in the design of new experiments. According to Professor Bradley Love from UCL, the ability of models to predict results indicates that scientific work is often based on already known patterns, which makes them predictable. In the future, this will allow to optimize scientific research, reducing the time and resources required to test hypotheses.
The developers plan to create an AI-based platform that will help researchers evaluate and improve their experiments before they are even conducted. They also emphasize that the approach used in BrainBench can be adapted to other scientific fields, expanding the impact of AI in accelerating scientific progress.
The study received support from leading organizations, including Microsoft and the UK's Economic and Social Research Council (ESRC). It involved scientists from the universities of Cambridge, Oxford and other world research centers, which emphasizes the global significance of this breakthrough.