AI and systematic literature reviews: potential benefits and use cases
Good quality evidence is the hallmark of any successful roadmap to access. Systematic literature reviews (SLRs) and meta-analysis are generally considered to be among the most robust forms of evidence; their adherence to strict methodological guidelines helps negate the inherent bias of researchers. SLRs are important for synthesizing and analyzing often vast amounts of research findings in various domains. They are used in research and policymaking to inform evidence-based decisions and practice. They differ from traditional literature reviews in that they typically adhere to strict methodological guidelines such as those issued by the Cochrane Collaboration (Cochrane, 2019) and the Centre for Reviews & Dissemination (Centre for Reviews and Discussion, 2009).
Conducting SLRs can be quite labour-intensive, requiring extensive manual efforts from researchers to identify and screen large volumes of published literature. However, with the recent boom in Artificial Intelligence and Machine Learning (AL/ML) technologies, the way SLRs are conducted may be about to change considerably.
In this paper, we discuss the roles of AI/ML to date as well as the challenges that may be faced in using these in literature reviews.
Benefits
One of the primary benefits of using AL/ML while conducting an SLR is the potential reduction in human time and effort required, especially during the citation screening phase of the review. One recent review reported a 77% reduction in human screening time (Van Dijk, 2023), however authors did acknowledge that a substantial period of time was required for researchers to become familiar with using the AI tool in the first place. While the exponential growth of publications in the field of medical science presents a challenge for human reviewers, AL/ML algorithms can excel here as they do not suffer from fatigue while dealing with such large data sets. As a result, it is possible that the use of AI/ML will not only decrease time required to conduct the review, but also increase accuracy in screening.
Types of AI for SLR
There are currently several forms of AI/ML available to support researchers with conducting SLRs.
Two of the more commonly used types — those discussed in the Cochrane guidelines — are: machine learning tools used to classify studies; and natural language processing tools used to determine inclusion and exclusion of studies.
Classifiers
Machine learning models (or ‘classifiers’) are often used to identify reports of randomised controlled trials (RCTs) from data sets containing a heterogenous variety of clinical trials designs. Tools such as the Cochrane “RCT classifier,” which is built on a large dataset of hundreds of thousands of records screened by Cochrane Crowd (Cochrane, n.d.), can be used to automatically exclude all non-RCTs from a data set prior to manual screening. This tool has the obvious benefit of removing non-eligible studies from the data set prior to manual screening, meaning that the human reviewer can focus on determining eligibility of the smaller subset of remaining articles.
AI for screening purposes
By using a combination of natural language and active learning algorithms, some tools can be trained to apply eligibility criteria within individual review to semi-automate the process of screening. In practice, after a period of training the tool will rank and prioritize studies to be screened. Studies with higher relevance are presented to human reviewers first. When the human reviewer is confident that the training has been sufficient, the AI/ML can be used to screen the remainder of the batch. This approach could allow for one review by AI/ML and one review by human, reducing the screening burden by 50%.
AI for extractions
Although perhaps less commonly used at present, AI/ML can also help with data extraction as part of systematic reviews. Information extraction algorithms are trained to recognize and extract specific types of information from text, such as study design, sample size, participant demographics, intervention details, outcomes, and results. These algorithms can be customized to extract data relevant to the specific requirements of the SLR. While these tools are in an earlier stage of development and at present would still require a high degree of human review, the potential for time savings is also significant.
Challenges of AI/ML
While the integration of AI/ML into the field of SLRs and evidence generation holds tremendous potential to streamline processes by enhancing efficiency and perhaps, the advancement is not without its challenges. One such significant challenge in employing AI/ML for SLRs lies in the quality and bias of the underlying data used to train the tools. It is often difficult to determine in advance what size and quality of training data set is required before it is safe to allow tools to screen automatically, and so careful calibration and validation of the tools is required before they can be safely used.
An additional concern is the opaque nature of the algorithms. The lack of transparency and interpretability in how algorithmic decisions are made could lead to concerns regarding the reproducibility and trustworthiness of review results.
Many governments and regulatory bodies across the world are moving to implement new regulations to govern the development and use of AI and ML. For instance, the European Union’s AI Act aims to create a framework that categorizes AI applications based on their risk levels, imposing stricter requirements on high-risk applications (European parliament, 2023). Similarly, the United States is considering frameworks that emphasize AI ethics, aiming to prevent bias and ensure AI systems are transparent and understandable (Intel.gov). These regulations often require developers to provide clear documentation, conduct impact assessments, and implement robust data governance practices.
A significant concern with the use of AI/MI in SLRs as part of Health Technology Assessment (HTA) submissions is acceptance by HTA bodies themselves. We reviewed guidance documents from a selection of some of the larger HTA agencies (including NICE, SMC, HAS, GBA, CADTH, NCPE and PBAC). Of the seven agencies reviewed, six provided no guidance and the National Institute of Clinical Excellence (NICE) stated that they support the use of machine classifiers if sufficiently validated – however NICE guidelines stopped short of recommending tools used to determine eligibility.
| HTA agency | Recommendation |
| NICE | NICE supports the use of machine classifiers but does not recommend the use of tools for determining eligibility |
| SMC | Refers to NICE guidelines |
| HAS | No recommendation provided on the use of AI in SLRs |
| GBA | No recommendation provided on the use of AI in SLRs |
| CADTH | No recommendation provided on the use of AI in SLRs |
| NCPE | No recommendation provided on the use of AI in SLRs |
| NICE | No recommendation provided on the use of AI in SLRs |
| PBAC | No recommendation provided on the use of AI in SLRs but refers to the Cochrane guidelines for conducting SLRs |
The Cochrane handbook discusses in detail both AI used for screening data sets and for use as classifiers (i.e. identification of RCTs from a data set for example). The use of AI for automatic exclusion of studies is currently not recommended for Cochrane reviews.
Summary
Impending changes to the European Union (E.U.) HTA procedures will encourage member states to conduct joint clinical assessments (JCAs) of health technologies and may bring significant challenges to health technology developers and to the agencies supporting them in their submissions. One of these challenges is the 90-day timeframe within which to conduct the SLRs and any accompanying meta-analyses. These challenges make the potential benefits of AI/ML all the more appealing.
In the future, AI/ML has the potential to adapt and mold all aspects of how researchers conduct SLRs. In addition to helping with study selection and screening, future tools will likely be developed that allow algorithms to assess the quality of included studies by analyzing various indicators such as study design and sample size. Future tools may even be able to assist in assessing the various sources of bias, such as selection bias, performance bias, detection bias, attrition bias, and reporting bias. However, for now, it is crucial not to overlook the indispensable role of human expertise in conducting SLRs. Experienced systematic review analysts bring contextual understanding of both the published literature and SLR methodology as well as disease area knowledge and critical appraisal capabilities that will complement the AI/ML tools available to them.
Despite the clear benefits AI/ML can bring to those conducting SLRs, it is critical to recognize that AI/MI is not a panacea, and that human expertise remains indispensable in ensuring the quality, relevance, and interpretability of review findings. A hybrid approach that combines the strengths of AI with human judgment and expertise will likely offer the most appropriate way forward. As the Head of Editorial Policy at Cochrane recently stated, “We’re excited at the potential of using AI to enhance the efficiency and accuracy of our review processes. However, we are also deliberately adopting a cautious approach (Cochrane, 2024).”
As with Cochrane, the primary motivation here at Clarivate will always be to ensure the robustness and reliability of our reviews and we will continue to closely monitor the use of AI/ML tools and only integrate them into our SLRs with complete transparency.
This post was written by Sunita Nair, Vice President of Evidence and Conor McCloskey, Senior consultant, Systematic Literature Reviews.
References
Cochrane, 2024. Cochrane announces new policy on AI generated content [Online]. Available: https://futurecochrane.org/newnews/cochrane-announces-new-policy-on-ai-generated-content#:~:text=Like%20them%2C%20we%20are%20open,the%20article’s%20accuracy%20and%20validity [Accessed 17/06/2024].
Cochrane. Cochrane Crowd [Online]. Available: https://crowd.cochrane.org/ [Accessed].
Centre for Reviews and Discussion, 2009. Systematic Reviews. CRDs guidance for undertaking reviews in healthcare.
EUROPEAN PARLIAMENT, 2023: EU AI Act: first regulation on artificial intelligence. https://www.europarl.europa.eu/topics/en/article/20230601STO93804/eu-ai-act-first-regulation-on-artificial-intelligence
Higgins JPT, T. J., Chandler J, Cumpston M, LI T, Page MJ, Welch VA (Editors 2019. Cochrane Handbook for Systematic Reviews of Interventions version 6.0 (updated July 2019).
INTEL.gov: https://www.intelligence.gov/artificial-intelligence-ethics-framework-for-the-intelligence-community
Van Dijk, S. H. B., Brusse-Keizer, M. G. J., Bucsan, C. C., Van Der Palen, J., Doggen, C. J. M. & Lenferink, A. 2023. Artificial intelligence in systematic reviews: promising when appropriately used. BMJ Open, 13, e072254.
