Test Assembly

Robust Text Similarity and Its Applications for the LSAT (RR 13-04)

by Dmitry I. Belov, Law School Admission Council; Bernard P. Veldkamp, University of Twente, Enschede, the Netherlands; and David A. Kary, Law School Admission Council

Text similarity measurement provides a rich source of information and is increasingly being used in the development of new educational and psychological applications. However, due to the high-stakes nature of educational and psychological testing, it is imperative that a text similarity measure be stable (or robust) to avoid uncertainty in the data. The present research was sparked by this requirement. First, multiple sources of uncertainty that may affect the computation of semantic similarity between two texts are enumerated. Second, a method for achieving the requirement of a robust text similarity measure is proposed and then evaluated by applying it to data from the Law School Admission Test (LSAT). While further evaluation of the proposed method is warranted, the preliminary results were promising.

Back to report gallery

Request the full report

Additional reports in this collection

An Overview of Research on the Testlet Effect: Associated...

A mathematical model called item response theory is often applied to high-stakes tests to estimate test-taker ability level and to determine the characteristics of test questions (i.e., items). Often, these tests contain subsets of items (testlets) grouped around a common stimulus. This grouping often leads to items within one testlet being more strongly correlated among themselves than among items from other testlets, which can result in moderate to strong testlet effects.

Read More

Applications of Uniform Test Assembly for the LSAT...

In a large-scaled high-stakes testing program such as the Law School Admission Test (LSAT), it is necessary to maintain a large bank of test items to support the demand for a new test form at nearly every administration. To assure that the item bank can support the test assembly requirements, ongoing monitoring of the quality of the item bank is necessary to identify deficiencies and direct item development efforts.

Read More