Using natural language processing and machine learning to replace human content coders.

Item request has been placed! ×
Item request cannot be made. ×
loading   Processing Request
Read More Add to Saved list
  • Additional Information
    • Source:
      Publisher: American Psychological Association Country of Publication: United States NLM ID: 9606928 Publication Model: Print-Electronic Cited Medium: Internet ISSN: 1939-1463 (Electronic) Linking ISSN: 1082989X NLM ISO Abbreviation: Psychol Methods Subsets: MEDLINE
    • Publication Information:
      Original Publication: Washington, DC : American Psychological Association, c1996-
    • Subject Terms:
      Natural Language Processing* ; Machine Learning* ; Psychology*/methods; Humans ; Qualitative Research ; Reproducibility of Results
    • Abstract:
      Content analysis is a common and flexible technique to quantify and make sense of qualitative data in psychological research. However, the practical implementation of content analysis is extremely labor-intensive and subject to human coder errors. Applying natural language processing (NLP) techniques can help address these limitations. We explain and illustrate these techniques to psychological researchers. For this purpose, we first present a study exploring the creation of psychometrically meaningful predictions of human content codes. Using an existing database of human content codes, we build an NLP algorithm to validly predict those codes, at generally acceptable standards. We then conduct a Monte-Carlo simulation to model how four dataset characteristics (i.e., sample size, unlabeled proportion of cases, classification base rate, and human coder reliability) influence content classification performance. The simulation indicated that the influence of sample size and unlabeled proportion on model classification performance tended to be curvilinear. In addition, base rate and human coder reliability had a strong effect on classification performance. Finally, using these results, we offer practical recommendations to psychologists on the necessary dataset characteristics to achieve valid prediction of content codes to guide researchers on the use of NLP models to replace human coders in content analysis research. (PsycInfo Database Record (c) 2024 APA, all rights reserved).
    • Publication Date:
      Date Created: 20220825 Date Completed: 20241223 Latest Revision: 20241223
    • Publication Date:
      20241224
    • Accession Number:
      10.1037/met0000518
    • Accession Number:
      36006759