Measuring Individual and Group Collaboration Skills using LLM-Annotated Conversational Process Data

Surina He

Abstract

This study explores the use of Large Language Models (LLMs) to automatically annotate textual group conversation data and construct rubric-based process indicators. To address the nested dependencies inherent in collaborative environments, a multilevel factor analysis model is applied to distinguish individual-level behaviours from group-level dynamics, with factor scores validated through qualitative inspection. The study illustrates how LLM-annotated conversational process data can support scalable, interpretable measurement of latent collaboration skills and complement log-based evidence in technology-enhanced assessment.