Test Process Data

Modeling Multilevel Dependence Structures for Responses and Process Data in a Bayesian Framework (PR 19-03)

by Konrad Klotzke and Jean-Paul Fox, University of Twente, Enschede, the Netherlands

Bayesian covariance structure modeling (BCSM) offers a flexible approach to modeling complex interdependences that arise when gathering test-taker data through computerized testing. In addition to the scored responses, process data such as response times or action patterns are obtained. Data from different sources may be cross-correlated; furthermore, within each data source, blocks of correlated observations may form testlet structures. In previous reports, BCSM was limited to the assumption that all test takers are part of the same group. In practice, test takers are often part of nested groups, such as schools, classrooms, and workgroups. Applied research has shown that ignoring group membership can result in biased
inferences about the latent constructs of interest. In this report, the BCSM methodology is extended to model, and control for, within- and between-group variations in the BCSM dependence structures. A simulation study illustrates the advantages of multilevel BCSM over the one-level BCSM. Limitations and future prospects of multilevel BCSM are discussed.

Back to report gallery

Request the full report

Additional reports in this collection

The Bayesian Covariance Structure Model for Testlets...

Standard item response theory (IRT) models have been extended with testlet effects to account for the nesting of items; these are well known as (Bayesian) testlet models or random effect models for testlets. The testlet modeling framework has several disadvantages. A sufficient number of testlet items are needed to estimate testlet effects, and a sufficient number of individuals are needed to estimate testlet variance. The prior for the testlet variance parameter can only represent a positive association among testlet items.

Read More

Predicting Future Academic Success: How a Bayesian...

The aim of this study was twofold: First, we investigated whether scores on an admission test administered in proctored and unproctored environments led to similar predictions of future academic success. Second, we explored how Bayesian modeling can be of help in interpreting admission-testing data. Results showed that the two modes of administering an admission test did not require the use of different models for predicting academic success, and that Bayesian modeling provides a very useful and easy-to-interpret framework for predicting future academic success.

Read More