Streamlining social media information retrieval for public health research with deep learning.

Publisher: Oxford University Press Country of Publication: England NLM ID: 9430800 Publication Model: Print Cited Medium: Internet ISSN: 1527-974X (Electronic) Linking ISSN: 10675027 NLM ISO Abbreviation: J Am Med Inform Assoc Subsets: MEDLINE

Publication: 2015- : Oxford : Oxford University Press
Original Publication: Philadelphia, PA : Hanley & Belfus, c1993-

Social Media* ; Deep Learning* ; Unified Medical Language System* ; COVID-19*; Humans ; Public Health ; Information Storage and Retrieval/methods

Objective: Social media-based public health research is crucial for epidemic surveillance, but most studies identify relevant corpora with keyword-matching. This study develops a system to streamline the process of curating colloquial medical dictionaries. We demonstrate the pipeline by curating a Unified Medical Language System (UMLS)-colloquial symptom dictionary from COVID-19-related tweets as proof of concept.
Methods: COVID-19-related tweets from February 1, 2020, to April 30, 2022 were used. The pipeline includes three modules: a named entity recognition module to detect symptoms in tweets; an entity normalization module to aggregate detected entities; and a mapping module that iteratively maps entities to Unified Medical Language System concepts. A random 500 entity samples were drawn from the final dictionary for accuracy validation. Additionally, we conducted a symptom frequency distribution analysis to compare our dictionary to a pre-defined lexicon from previous research.
Results: We identified 498 480 unique symptom entity expressions from the tweets. Pre-processing reduces the number to 18 226. The final dictionary contains 38 175 unique expressions of symptoms that can be mapped to 966 UMLS concepts (accuracy = 95%). Symptom distribution analysis found that our dictionary detects more symptoms and is effective at identifying psychiatric disorders like anxiety and depression, often missed by pre-defined lexicons.
Conclusions: This study advances public health research by implementing a novel, systematic pipeline for curating symptom lexicons from social media data. The final lexicon's high accuracy, validated by medical professionals, underscores the potential of this methodology to reliably interpret, and categorize vast amounts of unstructured social media data into actionable medical insights across diverse linguistic and regional landscapes.
(© The Author(s) 2024. Published by Oxford University Press on behalf of the American Medical Informatics Association. All rights reserved. For permissions, please email: [email protected].)

J Biomed Inform. 2022 Jan;125:103951. (PMID: 34785382)
Front Artif Intell. 2023 Mar 14;6:1023281. (PMID: 36998290)
J Med Internet Res. 2023 Mar 14;25:e45419. (PMID: 36812402)
J Biomed Inform. 2022 Feb;126:103983. (PMID: 34990838)
J Am Med Inform Assoc. 2022 Sep 12;29(10):1668-1678. (PMID: 35775946)
Health Commun. 2020 Jan;35(1):26-34. (PMID: 30351198)
Nucleic Acids Res. 2004 Jan 1;32(Database issue):D267-70. (PMID: 14681409)
J Med Internet Res. 2022 Oct 13;24(10):e39676. (PMID: 36191167)
Sci Rep. 2023 Jul 25;13(1):12005. (PMID: 37491443)
Soc Netw Anal Min. 2021;11(1):102. (PMID: 34697560)
Lancet Digit Health. 2021 Mar;3(3):e175-e194. (PMID: 33518503)
Science. 2020 Aug 28;369(6507):1060-1062. (PMID: 32855329)

Keywords: COVID-19; deep learning; information retrieval; named entity recognition; name  entity normalization; public health; social media

Date Created: 20240508 Date Completed: 20240620 Latest Revision: 20240622

20240622

PMC11187427

10.1093/jamia/ocae118

38718216