Due to scheduled system maintenance, LSAC customer service representatives will be available only from 8:30 a.m. to 5 p.m. ET on Thursday, March 27. We will return to our normal schedule, 8:30 a.m. to 8 p.m. ET, on Friday, March 28.

Test Process Data

The Probability of Exceedance as a Nonparametric Person-Fit Statistic for Tests of Moderate Length (RR 13-06)

by Jorge N. Tendeiro and Rob R. Meijer, University of Groningen, Groningen, the Netherlands

In this report we present a measure to identify unlikely patterns of correct/incorrect answers to test questions (commonly referred to as items). Some examples of why such patterns may occur include the misinterpretation of questions, item preknowledge, answer copying, or guessing behavior. The proposed measure is the probability of exceedance (PE). PE provides information about the probability of a correct/incorrect answer pattern, conditional on the test taker’s total score. Although this concept is not new, it is hardly if ever applied in practice.

In this report we show how the PE of a response vector can be computed and how misfitting response patterns are detected. A simulation study is conducted to investigate the robustness of this procedure.

Back to report gallery

Request the full report

Additional reports in this collection

The Bayesian Covariance Structure Model for Testlets...

Standard item response theory (IRT) models have been extended with testlet effects to account for the nesting of items; these are well known as (Bayesian) testlet models or random effect models for testlets. The testlet modeling framework has several disadvantages. A sufficient number of testlet items are needed to estimate testlet effects, and a sufficient number of individuals are needed to estimate testlet variance. The prior for the testlet variance parameter can only represent a positive association among testlet items.

Read More

Modeling Multilevel Dependence Structures for Responses...

Bayesian covariance structure modeling (BCSM) offers a flexible approach to modeling complex interdependences that arise when gathering test-taker data through computerized testing. In addition to the scored responses, process data such as response times or action patterns are obtained. Data from different sources may be cross-correlated; furthermore, within each data source, blocks of correlated observations may form testlet structures. In previous reports, BCSM was limited to the assumption that all test takers are part of the same group.

Read More