Identifying Cognitive and Meta-cognitive Aspects of Digital Inquiry from Process Data
Jesse R. Sparks & Caitlin Tenison
Abstract
Analysis of process data provides opportunities for generating novel insights into student behavior and performance in complex digital assessment tasks. The NAEP SAIL Virtual World for Online Inquiry project developed a digital platform for assessing cognitive and meta-cognitive aspects of inquiry from multiple sources of information, with attention to theoretically-grounded constructs like planning, locating and gathering information, evaluating information quality (i.e., usefulness and trustworthiness), integration, and communication – crucial aspects of digital information literacy in the 21st century. Grounded in Evidence-Centered Design (ECD), the virtual platform was designed to not only collect evidence of students’ performance on these different constructs using a suite of interactive digital tools, but to also capture - via log data - the inquiry processes students engaged in while completing a scenario-based task. The environment also included several adaptive design features (e.g., hints, prompts, and leveling) to help students regulate their inquiry processes and to encourage efficient and effective information gathering. In this workshop presentation, we will discuss our efforts to use various approaches to the analysis of process data at different timescales (e.g., timing and sequence analyses, hidden Markov models) to investigate the cognitive and meta-cognitive processes students engage in during online inquiry (e.g., time allocation, use of supports, effectiveness of hints and prompts) and explore relationships between these process-based indicators and task performance, using data from a small-scale empirical study. We will present observed patterns and relationships and focus on the implications of leveraging process data for understanding students’ digital literacy skills.