Reply to Bakker et al.
Assessing the Accuracy of the Scite Citation Classification System Requires the Same Definitions to be Used for Training as for Testing
DOI:
https://doi.org/10.18060/28018Keywords:
bibliometrics, ai, machine learningAbstract
Bakker, Theis-Mahon, and Brown1 recently presented an analysis of citations in scite.ai, a citation analysis platform. They concluded that the Scite algorithm inaccurately classified citations, particularly those they regarded as supporting previous work. While we strongly believe that independent assessments of Scite’s classifications are valuable, we argue that Bakker et al.’s assessment is incorrect, and that their conclusions are due to their definition of what constitutes supporting or contrasting citations. Additionally, Bakker, et al. restricted their analyses to citations of retracted works in systematic literature reviews, which artificially limits the types of statements that could be considered supporting or contrasting a specific claim. In our reply, we document the rationale for Scite’s classification scheme. We also provide examples of how Scite classifies different types of citations, as well as how these classifications differ from those presented in Bakker et al.
References
Bakker C, Theis-Mahon N, Brown SJ. Evaluating the accuracy of scite, a smart citation index. Hypothesis: Research Journal for Health Information Professionals. 2023 Sep 13;35(2). doi:10.18060/26528
Nicholson JM, Mordaunt M, Lopez P, Uppala A, Rosati D, Rodrigues NP, Grabitz P, Rife SC. Scite: A smart citation index that displays the context of citations and classifies their intent using deep learning. Quant Sci Stud. 2021 Nov 5;2(3):882-98. doi:10.1162/qss_a_00146
scite.ai: scite Search; [reviewed 2024 May 21; cited 024 May 21]. Available from: https://scite.ai/search?hasRetraction=true&mode=all
Lamers WS, Boyack K, Larivière V, Sugimoto CR, van Eck NJ, Waltman L, Murray D. Meta-Research: Investigating disagreement in the scientific literature. Elife. 2021 Dec 24;10:e72737. doi:10.7554/eLife.72737
Brown SJ, Bakker CJ, Theis-Mahon NR. Retracted publications in pharmacy systematic reviews. J Med Libr Assoc. 2022 Jan 1;110(1):47. doi:10.5195/jmla.2022.1280
Li W, Yu J, Liu Y, Huang X, Abumaria N, Zhu Y, Huang X, Xiong W, Ren C, Liu XG, Chui D. Elevation of brain magnesium prevents synaptic loss and reverses cognitive deficits in Alzheimer’s disease mouse model. Mol Brain. 2014 Dec;7(1):1-20. doi:10.1186/s13041-014-0065-y
Gatto RG. Molecular and microstructural biomarkers of neuroplasticity in neurodegenerative disorders through preclinical and diffusion magnetic resonance imaging studies. J Integr Neurosci. 2020 Sep 30;19(3):571-92. doi:10.31083/j.jin.2020.03.165
Zhang C, Gao X, Han Y, Teng W, Shan Z. Correlation between thyroid nodules and metabolic syndrome: a systematic review and meta-analysis. Front Endocrinol. 2021 Sep 16;12:730279. doi:10.3389/fendo.2021.730279
Kaur J. A comprehensive review on metabolic syndrome. Cardiol Res Pract. 2014 Oct;2014. doi:10.1155/2014/943162
Stavale R, Ferreira GI, Galvão JA, Zicker F, Novaes MR, Oliveira CM, Guilhem D. Research misconduct in health and life sciences research: A systematic review of retracted literature from Brazilian institutions. PLoS One. 2019 Apr 15;14(4):e0214272. doi: 10.1371/journal.pone.0214272
Tsukumo DM, Carvalho-Filho MA, Carvalheira JB, Prada PO, Hirabara SM, Schenka AA, Araujo EP, Vassallo J, Curi R, Velloso LA, Saad MJ. Loss-of-function mutation in Toll-like receptor 4 prevents diet-induced obesity and insulin resistance. Diabetes. 2007 Aug 1;56(8):1986-98. doi:10.2337/db06-1595
Downloads
Published
How to Cite
Issue
Section
Categories
License
Copyright (c) 2025 Sean Rife, Joshua Nicholson, Ashish Uppala, Domenic Rosati
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
All works in Hypothesis are licensed under a Creative Commons Attribution 4.0 International license. Authors own copyright of their articles appearing in Hypothesis. Readers may copy articles without permission of the copyright owner(s), as long as the author(s) and the Medical Library Association are acknowledged in the copy, and the copy is used for educational, not-for-profit purposes. For any other use of articles, please contact the copyright owner(s).
