Development and Validation of a Machine Learning Model for Automated Assessment of Resident Clinical Reasoning Documentation.

Publisher: Springer Country of Publication: United States NLM ID: 8605834 Publication Model: Print-Electronic Cited Medium: Internet ISSN: 1525-1497 (Electronic) Linking ISSN: 08848734 NLM ISO Abbreviation: J Gen Intern Med Subsets: MEDLINE

Publication: Secaucus, NJ : Springer
Original Publication: [Philadelphia, PA] : Hanley & Belfus, [c1986-

Clinical Reasoning* ; Documentation*; Electronic Health Records ; Humans ; Machine Learning ; Natural Language Processing ; Reproducibility of Results ; Retrospective Studies

Background: Residents receive infrequent feedback on their clinical reasoning (CR) documentation. While machine learning (ML) and natural language processing (NLP) have been used to assess CR documentation in standardized cases, no studies have described similar use in the clinical environment.
Objective: The authors developed and validated using Kane's framework a ML model for automated assessment of CR documentation quality in residents' admission notes.
Design, Participants, Main Measures: Internal medicine residents' and subspecialty fellows' admission notes at one medical center from July 2014 to March 2020 were extracted from the electronic health record. Using a validated CR documentation rubric, the authors rated 414 notes for the ML development dataset. Notes were truncated to isolate the relevant portion; an NLP software (cTAKES) extracted disease/disorder named entities and human review generated CR terms. The final model had three input variables and classified notes as demonstrating low- or high-quality CR documentation. The ML model was applied to a retrospective dataset (9591 notes) for human validation and data analysis. Reliability between human and ML ratings was assessed on 205 of these notes with Cohen's kappa. CR documentation quality by post-graduate year (PGY) was evaluated by the Mantel-Haenszel test of trend.
Key Results: The top-performing logistic regression model had an area under the receiver operating characteristic curve of 0.88, a positive predictive value of 0.68, and an accuracy of 0.79. Cohen's kappa was 0.67. Of the 9591 notes, 31.1% demonstrated high-quality CR documentation; quality increased from 27.0% (PGY1) to 31.0% (PGY2) to 39.0% (PGY3) (p < .001 for trend). Validity evidence was collected in each domain of Kane's framework (scoring, generalization, extrapolation, and implications).
Conclusions: The authors developed and validated a high-performing ML model that classifies CR documentation quality in resident admission notes in the clinical environment-a novel application of ML and NLP with many potential use cases.
(© 2022. The Author(s), under exclusive licence to Society of General Internal Medicine.)

Acad Med. 2021 Jul 1;96(7):1026-1035. (PMID: 33637657)
Med Educ. 2015 Jun;49(6):560-75. (PMID: 25989405)
Med Teach. 2020 Mar;42(3):246-251. (PMID: 31658842)
J Am Med Inform Assoc. 2019 Feb 1;26(2):172-184. (PMID: 30576561)
Diagnosis (Berl). 2019 Nov 26;6(4):335-341. (PMID: 31271549)
J Gen Intern Med. 2019 Aug;34(8):1631-1636. (PMID: 31025307)
J Am Med Inform Assoc. 2010 Sep-Oct;17(5):507-13. (PMID: 20819853)
JMIR Med Educ. 2018 Nov 12;4(2):e10306. (PMID: 30425025)
AMIA Annu Symp Proc. 2018 Dec 05;2018:1348-1357. (PMID: 30815179)
JAMA. 2019 Nov 12;322(18):1806-1816. (PMID: 31714992)
Med Teach. 2019 Sep;41(9):976-980. (PMID: 31007106)
J Allergy Clin Immunol. 2018 Jun;141(6):2019-2021.e1. (PMID: 29518424)
Teach Learn Med. 2015;27(2):163-73. (PMID: 25893938)
Ann N Y Acad Sci. 2018 Dec;1434(1):109-114. (PMID: 29766520)
BMC Emerg Med. 2018 Oct 25;18(1):36. (PMID: 30558573)
J Gen Intern Med. 2022 Feb;37(3):507-512. (PMID: 33945113)
JAMA. 1984 Mar 9;251(10):1277. (PMID: 6700018)
J Grad Med Educ. 2016 May;8(2):237-40. (PMID: 27168894)
Value Health. 2019 Jul;22(7):808-815. (PMID: 31277828)
Ann Emerg Med. 2019 Apr;73(4):334-344. (PMID: 30661855)
BMC Med. 2019 Oct 29;17(1):195. (PMID: 31665002)
Med Educ. 2015 May;49(5):476-86. (PMID: 25924123)
J Surg Educ. 2012 Jan-Feb;69(1):105-12. (PMID: 22208841)
J Eval Clin Pract. 2018 Jun;24(3):638-645. (PMID: 29237237)
Nat Med. 2019 Mar;25(3):433-438. (PMID: 30742121)
J Biomed Inform. 2019 Oct;98:103268. (PMID: 31421211)
Mayo Clin Proc. 2018 May;93(5):563-565. (PMID: 29631808)
Acad Med. 2019 May;94(5):623-625. (PMID: 30768470)
Acad Med. 2019 Mar;94(3):314-316. (PMID: 30540567)
Stud Health Technol Inform. 2017;245:447-451. (PMID: 29295134)
Acad Med. 2018 Jun;93(6):833-834. (PMID: 29538103)
Clin Teach. 2011 Sep;8(3):192-5. (PMID: 21851568)
JAMA. 2013 Dec 4;310(21):2249-50. (PMID: 24302083)
J Am Med Inform Assoc. 2014 Sep-Oct;21(5):910-6. (PMID: 24384231)
Med Teach. 2008;30(9-10):836-45. (PMID: 19117221)
Teach Learn Med. 2017 Oct-Dec;29(4):420-432. (PMID: 28497983)
J Am Med Inform Assoc. 2013 Jun;20(e1):e193-4. (PMID: 23399874)
Acad Med. 2018 Aug;93(8):1107-1109. (PMID: 29095704)
J Am Med Inform Assoc. 2011 Mar-Apr;18(2):181-6. (PMID: 21233086)
Acad Pediatr. 2017 Jan - Feb;17(1):68-73. (PMID: 27521461)
J Grad Med Educ. 2016 Oct;8(4):581-586. (PMID: 27777671)
Acad Med. 2019 Jun;94(6):902-912. (PMID: 30720527)
Appl Clin Inform. 2012;3(2):164-174. (PMID: 22577483)

Keywords: assessment; clinical reasoning; documentation; machine learning; natural language processing

Date Created: 20220617 Date Completed: 20220721 Latest Revision: 20230703

20240829

PMC9296753

10.1007/s11606-022-07526-0

35710676