Natural Language Processing and Psychosis: On the Need for Comprehensive Psychometric Evaluation

Authors

Alex S. Cohen, Louisiana State University, Department of Psychology, Baton Rouge, LA, USA.
Zachary Rodriguez, Louisiana State University, Department of Psychology, Baton Rouge, LA, USA.
Kiara K. Warren, Louisiana State University, Department of Psychology, Baton Rouge, LA, USA.
Tovah Cowan, Louisiana State University, Department of Psychology, Baton Rouge, LA, USA.
Michael D. Masucci, Louisiana State University, Department of Psychology, Baton Rouge, LA, USA.
Ole Edvard Granrud, Louisiana State University, Department of Psychology, Baton Rouge, LA, USA.
Terje B. Holmlund, University of Tromsø-The Arctic University of Norway, Tromso, Norway.
Chelsea Chandler, Emergency Medicine Residency Program-Baton Rouge Campus.
Glenn N. Jones, Family Medicine, Louisiana State University Health Sciences Center, School of Medicine in New Orleans, New Orleans, Louisiana, Baton Rouge, Louisiana.

Document Type

Article

Publication Date

9-1-2022

Abstract

BACKGROUND AND HYPOTHESIS: Despite decades of "proof of concept" findings supporting the use of Natural Language Processing (NLP) in psychosis research, clinical implementation has been slow. One obstacle reflects the lack of comprehensive psychometric evaluation of these measures. There is overwhelming evidence that criterion and content validity can be achieved for many purposes, particularly using machine learning procedures. However, there has been very little evaluation of test-retest reliability, divergent validity (sufficient to address concerns of a "generalized deficit"), and potential biases from demographics and other individual differences. STUDY DESIGN: This article highlights these concerns in development of an NLP measure for tracking clinically rated paranoia from video "selfies" recorded from smartphone devices. Patients with schizophrenia or bipolar disorder were recruited and tracked over a week-long epoch. A small NLP-based feature set from 499 language samples were modeled on clinically rated paranoia using regularized regression. STUDY RESULTS: While test-retest reliability was high, criterion, and convergent/divergent validity were only achieved when considering moderating variables, notably whether a patient was away from home, around strangers, or alone at the time of the recording. Moreover, there were systematic racial and sex biases in the model, in part, reflecting whether patients submitted videos when they were away from home, around strangers, or alone. CONCLUSIONS: Advancing NLP measures for psychosis will require deliberate consideration of test-retest reliability, divergent validity, systematic biases and the potential role of moderators. In our example, a comprehensive psychometric evaluation revealed clear strengths and weaknesses that can be systematically addressed in future research.

Publication Source (Journal or Book title)

Schizophrenia bulletin

First Page

939

Last Page

948

Recommended Citation

Cohen, A. S., Rodriguez, Z., Warren, K. K., Cowan, T., Masucci, M. D., Edvard Granrud, O., Holmlund, T. B., Chandler, C., & Jones, G. N. (2022). Natural Language Processing and Psychosis: On the Need for Comprehensive Psychometric Evaluation. Schizophrenia bulletin, 48 (5), 939-948. https://doi.org/10.1093/schbul/sbac051