A Hierarchical Framework for Modeling Speed and Accuracy on Test Items (RR 05-02)

In the analysis of data for the Law School Admission Test (LSAT) and other similar standardized tests, a mathematical model called item response theory (IRT) is commonly used to estimate both the characteristics of the test questions (items) and the ability level of the test takers. Such analyses are based on the test takers’ correct and incorrect responses to the test items. When a test is administered in a computerized mode, the capability of recording the amount of time a test taker has spent on each item provides us with additional information about the test-taking experience of individuals as well as the characteristics of items.

The practical goal of this paper is to use response times on test items as an additional source of information in estimating the abilities of the test takers when the test is delivered in a computerized mode. It is only possible to use this additional source when we have a mathematical model that (i) relates the speed at which a test taker works to his/her ability (accuracy) on the test and (ii) separates the test taker’s speed on the test from the time intensities of the items.

A hierarchical framework of modeling is introduced that has two different levels, one for the individual test taker and one for the population of test takers. Each level includes two components: one to model speed and the other to model accuracy. At the level of the individual test taker, the framework models the test taker’s responses to items (correct or incorrect) and the time he or she spent on each item. Both component models have separate parameters for the item and person effects. At the second level, the framework has a model for the population of test takers that explains how the speed and accuracy of the test takers tend to be related. In addition, it has an item-domain model that relates the time intensities of the items to such features as their difficulties.

A method is applied to estimate all unknown parameters from the responses and times on the test items. A description of how the second level model can be used to predict a test taker's accuracy from his speed or the difficulties of the items from their time intensities is also included.