Recordings from “Beyond Results 2021: From log data to valid inferences – Theory-based construction of process indicators”

The International Association for the Evaluation of Educational Achievement (IEA), together with the Leibniz Institute for Research and Information in Education (DIPF) and the Centre for International Student Assessment (ZIB), hosted a two-day workshop on analyzing log file and process data from large-scale international assessments in education. The event took place as an interactive webinar and online video conference on September 30th and October 1st, 2021.

Beyond Results 2021 aims at providing a platform for discussing conceptual perspectives and empirical research on how to construct process indicators that are, in some way, anchored in theory. The range of topics comprises the design of cognitive assessments providing meaningful log data, methods to extract features and process indicators from log data, strategies to validate the interpretation of process indicators and approaches to communicate insights obtained from process indicators.

In case of questions or comments, feel free to contact the organizers of the 2021 workshop:

Heiko Sibberns (IEA), Ulf Kroehne (DIPF), Frank Goldhammer (ZIB, DIPF) and Ralph Carstens (IEA)

Thursday 30 September 2021

Workshop Day One

Time
(CEST-Central European Summer Time)		 Title Speaker
13:00-13:20 From log data to valid inferences - Motivation for this workshop Frank Goldhammer (DIPF/ZIB) & Dirk Hastedt (IEA)
13:20-13:50 I see what you did there: Behavior-based prediction of diagnostic accuracy in simulated medical learning environments Matthias Stadler (LMU Munich)
13:50-14:20 Constructing strategy indicators of the diagnostic problem-solving process in the domain of car mechatronics: A theory-based approach Luca Spliethoff & Stephan Abele (TU Dresden)
14:20-14:25 (Short Break)  
14:25-15:10 Keynote: Understanding Examinees’ Item Responses through Cognitive Modeling of Response Accuracy and Response Times Susan E. Embretson (Georgia Institute of Technology)
15:10-15:25 (Break)  
15:25-15:55 Validating the Interpretation and Use of Process Data in Large-Scale Mathematics Assessments : An Example of Technology-Enhanced Mathematics Item in France. Franck Salles, Reinaldo Dos-Santos & Saskia Keskpaik (DEPP)
15:55-16:25 Leveraging log data to examine students’ progression in constructing scientific explanation Nani Teig & Fredrik Jensen (University of Oslo)
16:25-16:55 Signal to noise ratio: Lessons Learned from NAEP Process Data Ruhan Circi (AIR)
16:55-17:05 (Break)  
17:05-17:55 Debate: Derived Constructs versus Designed Constructs: A Priori and A Posteriori Approaches to Log Process Data - and who decides what to use? Matthias von Davier (Boston College) & Stephen Sireci (University of Massachusetts Amherst), Moderation: Saskia Wools (Cito)
17:55-18:00 (Short Break)  
18:00-19:00 Virtual Social Event  

Friday 1 October 2021

Workshop Day Two

Time
(CEST-Central European Summer Time)		 Title Speaker
13:00-13:05 (Welcome)  
13:05-13:35 Using process data to inform partial credit scoring in interactive assessments Daniel Duckworth (IEA)
13:35-14:05 Designing for Insights: An ECD Approach to Learning Analytics Andreas Oranje (ETS)
14:05-14:10 (Short Break)  
14:10-14:55 Keynote: Implications of a Sociocognitive Perspective on Educational Assessment with Connection to Computational Psychometrics Robert J. Mislevy (University of Maryland)
14:55-15:10 (Break)  
15:10-15:40 Identifying Cognitive and Meta-cognitive Aspects of Digital Inquiry from Process Data Jesse R. Sparks & Caitlin Tenison (ETS)
15:40-16:10 Leveraging process data to assess how students solve problems in open-ended, resource-rich environments: the PISA 2025 assessment of Learning in the Digital World Mario Piacentini (OECD)
16:10-16:40 Sensors and Traces: Investigating Assessment Response Processes with Digital Data Bryan Maddox (University of East Anglia)
16:40-16:50 (Break)  
16:50-17:20 Validity and validation of innovative educational assessments Saskia Wools (Cito)
17:20-17:50 Insights from Using Process Indicators for Formative Assessments within Learning Management Systems Ulf Kroehne (DIPF) & Carolin Hahnel (DIPF/ZIB)
17:50-18:15 Exchange on log data and process indicators in the FLIP+ community (working group) Thierry Rocher & Saskia Keskpaik](Speakers2021.md#saskia-keskpaik)
18:15-18:30 Wrap-up, final reflections