Test Process Data

Marginalized Measurement Variance Modeling and Bayes Factor Testing (RR 16-06)

by Jean-Paul Fox and Vera Broks, University of Twente, Enschede, the Netherlands

Among the assumptions that should be met when applying an item response theory (IRT) model to the analysis of test data is measurement invariance. Measurement invariance requires that, after controlling for a test taker’s proficiency, group membership have no effect on the probability that that test taker will answer a test question correctly. Groups may be defined on the basis of many factors, including gender, race/ethnicity, and citizenship.

This research study proposes and evaluates a new method for detecting violations of the measurement invariance assumption. The method is evaluated through both data simulation and application to actual responses to an international survey. The results obtained by the proposed method are also compared to those obtained using the Mantel–Haenszel statistic, the industry standard for group membership comparisons. Promising results are reported, and plans for extended research are discussed.

Back to report gallery

Request the full report

Additional reports in this collection

The Bayesian Covariance Structure Model for Testlets...

Standard item response theory (IRT) models have been extended with testlet effects to account for the nesting of items; these are well known as (Bayesian) testlet models or random effect models for testlets. The testlet modeling framework has several disadvantages. A sufficient number of testlet items are needed to estimate testlet effects, and a sufficient number of individuals are needed to estimate testlet variance. The prior for the testlet variance parameter can only represent a positive association among testlet items.

Read More

Modeling Multilevel Dependence Structures for Responses...

Bayesian covariance structure modeling (BCSM) offers a flexible approach to modeling complex interdependences that arise when gathering test-taker data through computerized testing. In addition to the scored responses, process data such as response times or action patterns are obtained. Data from different sources may be cross-correlated; furthermore, within each data source, blocks of correlated observations may form testlet structures. In previous reports, BCSM was limited to the assumption that all test takers are part of the same group.

Read More