Our colleague, Dr. Yizhou Fan from the University of Edinburgh – UK, will be presenting our FLoRA work on the validity of measurement of SRL based on trace data at the AERA 2022 Symposium:
Presentation Title:
Toward Improving the Validity of Measurement of Self-Regulated Learning Based on Trace Data
Time: Fri, April 22, 11:30am to 1:00pm PDT | (Sat, April 23, 4:30 to 6:00am AEST
Description
Background. Contemporary research that looks at self-regulated learning (SRL) as processes of learning events derived from trace data has attracted an increasing interest over the past decade. Several researchers proposed trace-based measurement protocols of SRL processes which have been used in a series of subsequent studies (Siadaty et al., 2016, Saint et al., 2020, Fan et al., 2021). Because the use of trace data in measuring SRL is becoming more widespread, specific attention must be paid to whether trace data are reliable, and whether interpretations grounded in trace data are valid (Winne, 2020). However, limited research has been conducted that looks into the validity of trace-based measurement protocols.
Aims & Method. In order to fill this gap, we propose a novel validation approach (as shown in Figure 1) that combines theory-driven and data-driven perspectives to increase the validity of interpretations of SRL processes extracted from trace-data. We developed our own trace-based measurement protocols to translate raw trace data into SRL processes (such as Orientation and Monitoring); and we also used a previously developed coding scheme (Bannert, 2007) to code learners’ utterances into SRL processes. The hand-coded think-aloud data were used as “ground truth” for interpretation and validation of SRL processes derived from trace data. The main contribution of this approach consists of three alignments between trace data and think aloud data to improve measurement validity, see Figure 1. In addition, we define the match rate between SRL processes extracted from trace data and think aloud as a quantitative indicator to evaluate the “degree” of validity.
Findings & Significance. We tested this validation approach in a laboratory study that involved 44 learners who learned individually about the topic of artificial intelligence in education with the use of a technology-enhanced learning environment for 45 minutes. Following this new validation approach, we achieved an improved match rate between SRL processes extracted from trace-data and think aloud results (training set: 54.24%; testing set: 55.09%) compared to the match rate before applying the validation approach (training set: 38.97%; test set: 34.54%). By considering think-aloud data as “ground truth”, this improvement of the match rate quantified the extent to which validity can be improved by using our validation approach. It is also worth noting that, the shared theoretical background on SRL also played an essential role in informing the coding of think-aloud data and interpreting the SRL processes extracted from trace data. In conclusion, the novel validation approach presented in this study used both empirical evidence from think-aloud data and rationale from our theoretical framework of SRL, which now, allows testing and improvement of the validity of trace-based SRL measurements.
The full session details can be accessed via AERA 2022 Program Guide
Symposium Information :
Authenticating the Signal: Validating Digital Traces of Student Learning Using Concurrent, Corroborating Data Sources
Presenters in the session include:
Chair: Matthew Bernacki – University of North Carolina Chapel Hill
Discussant: Sanna Järvelä – University of Oulu
Presenting Authors: Matthew Graham (University of Oregon), Fatemeh Salehian Kia, (University of British Columbia), Matthew Graham (University of Oregon), Yizhou Fan (University of Edinburgh)