Test Assembly

A Stochastic Search for Test Assembly, Item Pool Analysis, and Design (RR 04-01)

by Dmitry Belov, Law School Admission Council; and Ronald D. Armstrong, Rutgers, The State University of New Jersey

The problems of item pool analysis and design are the subject of many recent studies. The rationale for this type of research is to increase the usability of existing item pools and to decrease the cost of designing new items. Clearly these are crucial problems for all testing agencies.

This paper demonstrates that a stochastic search technique allows for the solution of various item pool analysis and design problems. Stochastic search refers to a family of approaches that allows for the introduction of random noise into the model and/or search strategy being developed. The problems addressed in this paper are as follows:

Comparison of two pools: Given two different item pools, identify the strengths and weaknesses of the pool characteristics from the test assembly point of view.

Constraint difficulty and relaxation: Given an item pool and test assembly constraints, identify the most difficult constraint(s), that is, the constraints that prevent the assembly of additional test forms. Explore how relaxation of those constraints impacts the number of feasible forms available from the pool.

Designing new items: Given pool and test assembly constraints, determine the properties of the items that should be added to a pool in order to acquire more test forms.

The paper employs stochastic search techniques to assemble linear standardized tests where the properties of the constraints are used to guide the search. The nature of the random search provided efficient and elegant solutions to the above problems.

The theoretical results of this paper can be used either for development of paper-and-pencil tests or computerized adaptive tests (CAT). Based on methods described in this paper, software tools were developed for assembling Law School Admission Test (LSAT) forms, extracting multiple disjoint LSAT forms, and determining the characteristics that new items to the pool should have in order to obtain additional LSAT forms from the pool.

Back to report gallery

Request the full report

Additional reports in this collection

Some Practical Issues in Computerized Adaptive Testing...

Many standardized tests are now administered via computer rather than paper-and-pencil format. The computer-based delivery mode brings with it certain advantages. One advantage is the ability to adapt the difficulty level of the test to the ability level of the test taker in what has been termed computerized adaptive testing, or CAT. A second advantage is the ability to record not only the test taker’s response to each item (i.e., question), but also the amount of time the test taker spends considering and answering each item.

Read More

Review of Modern Methods for Automated Test Assembly...

Automated methods have been developed for assembling test forms, evaluating a pool of test questions (i.e., items) to determine the number of test form assemblies it can support, and designing an item pool that can most efficiently support the test form assembly process. Automated methods have greatly maintained and improved such activities, all of which are essential to the support of every testing program. This report reviews the major approaches that have been applied in the development of these methods.

Read More