Exploring collaborative problem-solving tasks for individual-level and team-level inferences using multilevel modeling

Collaborative problem solving (ColPS) is necessary to succeed in the 21st century and has therefore been introduced into compulsory K-12 education in many Western countries. ColPS is the “capacity of an individual to: recognize the perspective of other persons in a group; participate as a member of the group by contributing their knowledge, experience, and expertise constructively; recognize the need for contributions and how to manage them; identify structure and procedure involved in resolving a problem; and, as a member of the collaborative group, build and develop knowledge and understanding” (Griffin et al., 2012, p.7). Several aspects need consideration when designing ColPS tasks, including group dependence, team size, and activities to promote collaboration and problem solving (Siddiq & Scherer, 2017). Since students are often randomly assigned to teams, observations within groups are not independent. Hence, psychometric models need to account for the hierarchical structure of the data. Wilson et al. (2017) presented an approach for analyzing ColPS tasks using multilevel models, which are widely used in educational research. The authors focused on modeling the group effects on the overall test performances. We propose focusing on group effects on individual task performance instead. We aim to classify ColPS tasks into ones with and without ignorable group effects, implying lower and higher collaboration. We also propose calibrating these tasks using a two-dimensional Rasch model to further understand the two constructs these items represent. We analyse data collected from the Norwegian Digital literacy (LDN-ICT) instrument (Siddiq et al., 2017). In total, 39 tasks were administered to 144 ninth-grade students who worked in groups of three or four. Prior to modelling, it will be helpful to use descriptive statistics in an exploratory way to establish a framing context for modelling. Preliminary findings suggest that tasks showing significant group effects can be identified based on the estimated intraclass correlation (ICC)—a commonly used measure of group dependence. ICCs closer to zero indicate ignorable group effects, while higher ICCs imply substantial group effects. The ICCs ranged between 0.00 and 0.88, with an average ICC of 0.31. We also found a significant agreement between estimated group dependence and intended group dependence. Moreover, tasks requiring reflection, evaluation, and accessing information exhibited higher ICCs than others. This study contributes to the design and analysis of ColPS assessments and illustrates tools to assess if tasks function as they were intended.