Extending the usefulness of the verbal memory test: The promise of machine learning

Chelsea Chandler, Department of Computer Science, University of Colorado Boulder, CO, USA; Institute of Cognitive Science, University of Colorado Boulder, CO, USA. Electronic address: chelsea.chandler@colorado.edu.
Terje B. Holmlund, Department of Clinical Medicine, University of Tromsø - The Arctic University of Norway, Norway.
Peter W. Foltz, Institute of Cognitive Science, University of Colorado Boulder, CO, USA; Pearson, CO, USA.
Alex S. Cohen, Department of Psychology, Louisiana State University, LA, USA.
Brita Elvevåg, Department of Clinical Medicine, University of Tromsø - The Arctic University of Norway, Norway; Norwegian Centre for eHealth Research, University Hospital of North Norway, Tromsø, Norway. Electronic address: brita.elvevag@uit.no.

Abstract

The evaluation of verbal memory is a core component of neuropsychological assessment in a wide range of clinical and research settings. Leveraging story recall to assay neurocognitive function could be made more useful if it were possible to administer frequently (i.e., would allow for the collection of more patient data over time) and automatically assess the recalls with machine learning methods. In the present study, we evaluated a novel story recall test with 24 parallel forms that was deployed using smart devices in 94 psychiatric inpatients and 80 nonpatient adults. Machine learning and vector-based natural language processing methods were employed to automate test scoring, and performance using these methods was evaluated in their incremental validity, criterion validity (i.e., convergence with trained human raters), and parallel forms reliability. Our results suggest moderate to high consistency across the parallel forms, high convergence with human raters (r values ~ 0.89), and high incremental validity for discriminating between groups. While much work remains, the present findings are critical for implementing an automated, neuropsychological test deployable using remote technologies across multiple and frequent administrations.