Franck Salles, Reinaldo Dos-Santos & Saskia Keskpaik

Abstract

During the last few years, France, like many other countries, has been undergoing a transition from paper-based to digital large-scale assessments in order to measure student performance in mathematics. There is a rising interest in technology-enhanced items (TEIs) which offer new ways to assess traditional mathematical competency (Stacey et al, 2013), as well as to address higher order skills using computer-based assessments (Drijvers, 2019). The rich data captured by these items, often referred to as process data (Goldhammer, in press), allows insight into how students tackle the item and their process strategies.

A theory driven methodology based on the findings of research in didactics of mathematics enabled us to provide fine insight in students’ problem solving processes captured in TEIs, in particular in the domain of numerical relationships. An analysis of tasks and conceptions at stake in TEIs was carried out, essentially based on the instrumental approach (Rabardel, 2002, Trouche, 2003, Drijvers et al, 2016) and the dual nature of mathematical conceptions (Sfard, 1991, Drijvers et al, 2016). It allowed identifying utilization schemes (Rabardel, 2002) concerning the technological artifacts embedded in TEIs, in relation with the operational and structural aspects of the conception of numerical relationships. These findings participated into defining, developing and recording meaningful process indicators within log data issued from TEIs in order to attempt to give evidence for student’s conception aspects in relation with performance (Salles et al, 2020). Didactical analysis and results of process data analysis on a sample of 9 graders in France in 2017 are then discussed. A specific item is studied as an example.

Results show the item only partly discriminates among students according to their utilization schemes and the conceptual aspect of their understanding of numerical relationship, as operated in the item. A significant number of outliers in the data made us reconsider choices made during item development, and realize that, in this specific case, process data recorded by TEIs does not always provide a complete and accurate picture of students’ mathematical activity.

Key Words: Process data; digital large-scale assessment; technology-enhanced items; validation; mathematics; instrumental approach

Video