In order to diagnose accurately physicians often need to work together in teams. During collaborative diagnostic processes, diagnosticians must engage in so-called collaborative diagnostic activities (CDAs) - broad process-based indicators for diagnostic competence that can be found in various collaborative diagnostic contexts. This complex behavior is highly challenging and needs to be trained. Simulation-based learning environments with embedded instructional support provide an opportunity for an effective training. However, one major challenge for educators is to adapt the support individually to the learners´needs. The aim of this study is to exploratively identify early starting points for an adaptation basis of instructional support to foster collaborative diagnostic competence based on CDAs. Therefore, we investigate whether and how fast it is possible to predict diagnostic accuracy from learners´ time engagement in CDAs based on the behavior displayed in a medical learning environment while interacting with an agent-based radiologist. Log files from 98 medical students and physicians were automatically coded according to CDAs, transformed into behavioral strings (N = 475) using time on task, and sequenced into 25 bigrams (n-gram method; Damashek, 1995). Support vector machines with linear kernel (Hsu et al., 2003), random forests (Breiman, 2001), and gradient boosting machines (Natekin & Knoll, 2013) were trained to classify whether a participant would provide the correct diagnosis based on the bigrams. The algorithms reached an acceptable overall prediction quality and a notably reliable prediction of incorrect diagnoses (specificity) only after two minutes of the observed behavior. These findings can help to identify learners who need help within a very short time to provide instructional support early in the diagnostic process. This study is an example of how user log files can be used for a performance-based assessment in higher education.