Recordings from “Beyond Results 2020: Paving the way for the use of process data”

The International Association for the Evaluation of Educational Achievement (IEA), together with the Leibniz Institute for Research and Information in Education (DIPF) and the Centre for International Student Assessment (ZIB), hosted a three-day workshop on analyzing log file and process data from large-scale international assessments in education. The event took place as an interactive webinar and online video conference on June 17th - 19th, 2020.

The workshop was the second event of a growing community dealing with process data from large-scale assessments, after a first process data conference and workshop in Dublin in May 2019 hosted by Educational Research Centre (Dublin, Ireland; ERC) and Educational Testing Service (Princeton, NJ, USA; ETS).

The first two days of the workshop in 2020 focused on managing, preparing, analyzing, and sharing insights from process data. On the third day, the focus was on practical, hands-on training using log-file data from the most recent cycle of the IEA’s Trends in International Mathematics and Science Study, TIMSS 2019.

In case of questions or comments, feel free to contact the organizers of the 2020 workshop:

Frank Goldhammer, Heiko Sibberns, and Ulf Kroehne.

Wednesday 17 June 2020

During the two days we worked in two different formats. In addition to the classic sessions with plenary lectures, we had sessions with an impulse talk to stimulate the following small group work on a specific topic.

Day One

Opening

Title Speaker
Using log data – Related challenges, desiderata, and motivation for this workshop Frank Goldhammer (DIPF/ZIB) & Dirk Hastedt (IEA)

Opening Keynote

Title Speaker
Process Data Research and some Examples from TIMSS Matthias von Davier (Boston College), Ina Mullis and Mick Martin

Concepts and Constructs

Title Speaker
The Benefits of Process Data Analysis for a Better Understanding of Complex Psychological Constructs Samuel Greiff (University of Luxembourg)
Validation and Applications of Rapid Guessing to Detect Test Taker Disengagement Steven L. Wise (nwea)
Using Log-file Based Process Measures as Indicators of Task Engagement in Digital Reading: Perspectives and Problems. Johannes Naumann (University of Wuppertal)

Data Management

Title Speaker
Preprocessing of log-file data: Adapting familiar routines to unfamiliar data Heiko Sibberns (IEA) & Ulf Kroehne (DIPF)
Small Group Work  

Closing Keynote Day One

Title Speaker
Using Paradata to Evaluate Online Performance Frauke Kreuter (University of Maryland)

Virtual Social Event

Title Performance
Take Love Easy Stefanie Hoevel & Martin Lejeune

Thursday 18 June 2020

Day Two

Log-data and Items Documentation

Title Speaker
Advances and Challenges in Logging Documentation for Educational Assessments Qiwei He (ETS)
Small Group Work  

Data Protection and Anonymization

Title Speaker
Privacy and Process-data: Lessons learned from Trusted Learning Analytics Hendrik Drachsler (DIPF)

Extraction Approaches and Tools

Title Speaker
LOGAN—An R Package for Log-file Analysis in International Large-Scale Assessments Denise Reis Costa (CEMO) & Ronny Scherer (CEMO)
Addressing the Challenges From Response Process Data: glassPy Data Analytics Solution and its Applications at Educational Testing Service Jiangang Hao (ETS)
Experience with Log-data Analysis and the Use of Big Data Technologies in Large-Scale Assessment in France Saskia Keskpaik (French Ministry of Education)
Processing and Analyzing PIAAC log files with the PIAAC LogDataAnalyzer (LDA) Carolin Hahnel (DIPF/ZIB)
Tools for Analyzing Log File Data Ulf Kroehne (DIPF)

Closing Keynote

Title Speaker
Using Computer-Based Process Data to Improve Assessment Validity: Challenges and Opportunities Stephen Sireci (University of Massachusetts Amherst)
Wrap up Frank Goldhammer (DIPF/ZIB) & Dirk Hastedt (IEA)

Friday 19 June 2020

Hands-on Training

Title Speaker
Analysis of Log-File Data using LogFSM and R Ulf Kroehne (DIPF)