Norwegian version of this page

Process data analysis in problem-solving tasks (PhD-project) (completed)

Problem-solving is considered as one of the 21st-century skills. Understanding how people solve problems provides a solid foundation for problem-based learning. The process of approaching a problem can be inferred from the steps taken by problem-solvers. This Ph.D. project aims to gain a better understanding of problem-solving by analyzing process data. 

Illustratration photo by Colourbox.

Illustration photo: Colourbox.

Background

The assessment of problem-solving skills has become a focus area in international large-scale assessments such as the 2012 and 2015 Program for International Student Assessment (PISA) and the 2012 Program for the International Assessment of Adult Competencies (PIAAC). In these assessments, problem-solving tasks are delivered on computers, and the human-computer interaction is recorded in log files. These highly detailed data capture each step a respondent takes and the corresponding response times. Such process data provide valuable information that helps improve educational practice.

Studies

The project consists of four studies from both a substantive and methodological perspective. Sound methods provide a solid foundation for obtaining new substantive knowledge.

In Study I, we visualize action sequences and response times using networks, extract essential information (called network features) from networks of process data, and identify solution patterns of problem-solvers using clustering techniques. We use a problem-solving task from PISA 2012 as an example to illustrate our approach. In Study II, we examine the validity of generalizing process indicators for planning and non-targeted exploration across seven tasks from the PIAAC 2012 domain Problem-Solving in Technology-Rich Environments. These studies provide insights into the solution strategies and cognitive processes involved in problem-solving, which can potentially a) help problem-solvers reflect on their problem-solving processes through individual networks of process data, b) help educators tailor instructions for students with different solution patterns, and c) help test developers validate their task designs.

A joint analysis of process data and performance data, such as in Studies I and II, often includes both continuous data (e.g., response times) and discrete data (e.g., task scores and the number of actions). The analysis of such data is challenging due to a) the high dimensionality of the data, b) the dependencies between indicators, and c) the mixture of data types. Generalized linear latent variable models are suitable for this situation. However, the issue of estimation efficiency limits their use in practice. In Studies III and IV, we apply Laplace approximations to the integrals of the marginal maximum likelihood to increase the computational efficiency of generalized linear latent variable models. Specifically, in Study III, we apply second-order Laplace approximations to multi-dimensional, multi-group generalized linear latent variable models for ordinal data; in Study IV, we extend the method to a collection of continuous, ordinal, and count variables. The results suggest that Laplace approximations greatly reduce the estimation time compared to quadrature-based methods, and the higher-order Laplace approximations produce more accurate estimates and achieve a higher convergence rate compared to first-order Laplace approximations. The method can be also applied to other fields, such as ecological data including species counts and biomass and eye-tracking data including fixation counts and time.

Tags: process data, problem-solving, latent variable models, social network analysis
Published May 19, 2021 10:19 AM - Last modified Dec. 4, 2023 8:13 AM

Contact

Maoxin Zhang

Participants

Detailed list of participants