Due to scheduled system maintenance, LSAC customer service representatives will be available only from 8:30 a.m. to 5 p.m. ET on Thursday, March 27. We will return to our normal schedule, 8:30 a.m. to 8 p.m. ET, on Friday, March 28.

Test Process Data

The Use of Person-Fit Scores in High-Stakes Educational Testing: How to Use Them and What They Tell Us (RR 14-03)

by Rob R. Meijer and Jorge N. Tendeiro, University of Groningen, Groningen, the Netherlands

Several statistics used to detect inconsistent patterns of correct/incorrect answers to test questions (items) were evaluated based on data from one Analytical Reasoning (AR) and one Logical Reasoning (LR) section of the Law School Admission Test. Item score patterns were also evaluated based on gender and racial/ethnic subgroups. We showed that test takers who were consistently flagged by all statistics evaluated and for both the AR and the LR sections had relatively low scores, which may have been the result of extensive guessing. Gender group comparisons showed no inconsistent test-taking behavior between male and female test takers. However, we did find significant differences in item score patterns for one racial/ethnic subgroup compared to the other subgroups. This particular subgroup has a large proportion of test takers whose first language is not English. We conclude that the indices evaluated provide useful information that may be used to routinely monitor test-taking behavior and to enhance the interpretation of test scores.

Back to report gallery

Request the full report

Additional reports in this collection

The Bayesian Covariance Structure Model for Testlets...

Standard item response theory (IRT) models have been extended with testlet effects to account for the nesting of items; these are well known as (Bayesian) testlet models or random effect models for testlets. The testlet modeling framework has several disadvantages. A sufficient number of testlet items are needed to estimate testlet effects, and a sufficient number of individuals are needed to estimate testlet variance. The prior for the testlet variance parameter can only represent a positive association among testlet items.

Read More

Modeling Multilevel Dependence Structures for Responses...

Bayesian covariance structure modeling (BCSM) offers a flexible approach to modeling complex interdependences that arise when gathering test-taker data through computerized testing. In addition to the scored responses, process data such as response times or action patterns are obtained. Data from different sources may be cross-correlated; furthermore, within each data source, blocks of correlated observations may form testlet structures. In previous reports, BCSM was limited to the assumption that all test takers are part of the same group.

Read More