LSAT Validity and ABA Standard 503

A Foundation of Validity

Beginning with the very first notion of a standardized test for admission to law school, validity was a primary focus. In his May 17, 1945 letter to the College Entrance Examination Board (CEEB) suggesting the development of such a test, Frank H. Bowles, Director of Admissions at Columbia University, stated the seven criteria listed below. (Note that Henry Chauncey, Director of CEEB, in his May 22, 1945 reply to Bowles, acknowledged that some of these criteria may need to be modified in order to achieve others.).

1. High predictive value as a measure of success in law school.
2. It should be a discrete measure of capacity for law study insofar as that capacity can be isolated.
3. High reliability.
4. Short enough to be administered in not over an hour and one-half.
5. Sensible and observable relation to the study of law.
6. Results easy to interpret.
7. Low cost to the applicant and to the university.

These foundational criteria demonstrate that validity (1, 2, and 5) and reliability (3) were priorities for the LSAT since its inception.

In the Fall of 1947, the first LSAT validity evaluation was carried out when a three-hour pretest was administered to first-year students at 9 law schools. Pretest performance was compared to participants’ first year grades at the conclusion of the 1947-1948 academic year. While these validity results were not available in time to guide development of the first LSAT form administered on February 28, 1948, they did guide the development of the second operational LSAT form, administered in February 1949.

Test validity continued to guide the evolution of the LSAT as law schools considering LSAT scores in their admission process shared first year performance data with LSAT validity researchers. A 1950 report describing results of a series of studies initiated in 1949 noted a correlation between LSAT scores and first year law school grades of .44 and a correlation of .52 when LSAT score was combined with a measure of previous academic success (Schrader & Olsen, 1950). These results are remarkably consistent with results currently observed. Predictive validity is only one aspect of an overall validity evaluation, and the experimental administration and subsequent evaluation of new LSAT question types was common in the early days of the test. Empirical evaluation of such question types, as evidenced by a 1949 research report on question types pretested in the fall of 1948 (French, 1949), led some to be adopted and others to be discarded. The composition of the LSAT evolved to best reflect the skills needed for success in law school.

In the mid-1970’s, LSAC President Frederick Hart assembled reports on approximately 105 LSAC-sponsored research studies from 1949 through 1983 into 4 volumes that are still referenced today. A review of the titles in those volumes further demonstrates the value placed on establishing and demonstrating the validity of the LSAT. About half of the reports, 53 to be exact, address the validity of the LSAT in some way (see Appendix A for a list of these reports). The validity research presented includes overall predictive validity, the evaluation of experimental question types, differential predictive validity for population subgroups, validity for repeat test takers, and technical aspects of validity analyses. Note that LSAC-sponsored research on the best representation of undergraduate grades in predicting law school performance were also noted but not included in the count of 53.

In 1965, LSAC (then known as the Law School Admission Test Council) made the evaluation of validity for individual law schools a priority by committing to provide this service free of charge to all member law schools. In June 1968, having carried out 128 validity studies for 97 law schools over the prior four years, the Council voted to make the validity study service permanent (see an excerpt from the memo to law schools in Appendix B). This service, now known as the LSAT Correlation Studies, is still provided today free of charge to LSAC member law schools. Participants receive a school-specific report detailing the predictive power of LSAT score and undergraduate grade point average, both individually and in combination, for predicting law school first year average. The results of the Correlation Studies are regularly summarized and published (see Summary of 2017, 2018, and 2019 LSAT Correlation Study Results (PDF)). Technical reports addressing repeat test takers and differential prediction for gender and racial/ethnic subgroups are also regularly shared (see LSAC Research Library)

Was LSAC required to demonstrate LSAT validity for the ABA?

In a review of the ABA Standards archives available on the organization’s website (Standards Archives, americanbar.org ), the first endorsement of LSAT usage in law school admission appears in 1969 as, “The Council is of opinion that LSAT, administered by Educational Testing Service of Princeton, New Jersey, should be required of all applicants for admission to the approved schools.” By the time this statement appeared, the LSAT had undergone numerous validity evaluations of various types to support its usage for this purpose. In fact, 20 of the 53 research studies mentioned above had been carried out by this time. LSAC had also permanently committed to providing free validity studies to all member law schools, having already provided this service for four years.

Continuing LSAT Validity Evaluation

Validity continues to be the first priority for the LSAT as it was in Mr. Bowles letter of 1945. Annual LSAT Correlation Study participation remains high with 163 law schools having participated in the latest cycle. As mentioned above, LSAT Technical Reports on various aspects of validity are regularly made available. The skills evaluated by the LSAT were also recently validated through the 2018 Skills Analysis Study (see LSAC 2018 Skills Analysis Study: Content Validity of the LSAT (RR 19-01)).

A Word on Reliability

Unlike validity, reliability is evaluated for an individual test form and can be represented as a single statistic with a theoretical ceiling (never observed in practice) of 1.0. LSAT reliability is consistently above 0.90, indicating that the LSAT is a highly reliable test.

Appendix A

Early Validity Research Reports

French, J. W. (1949). Validation of Experimental Law School Admission Test Administered in the Fall of 1948 (LSAC-49-1)

Schrader, W. B. & Olsen, M (1950). The Law School Admission Test as a Predictor of Law School Grades (LSAC-50-1).

French, J. W. (1952). Validation of the Practical Judgment and Directed Memory Experimental Sections of One Form of the Law School Admission Test (LSAC-52-1).

Johnson, A. Pemberton & Olsen, M.A. (1952). Comparative Three-Year and One-Year Validities of the Law School Admission Test at Two Law Schools (LSAC-52-2).

French, J. W. (1953). Validation of the Study of Arguments and Vocabulary Interest Test Experimental Sections of One Form of the Law School Admission Test (LSAC-53-1).

Coffman, William E. & Papachristou, J. (1955). Experimental Objective Tests of Writing Ability for the Law School Admission Test (LSAC-55-1).

Olsen, M. A. (1956). The Effectiveness of Tests of Writing Ability and of Directed Memory for Predicting Law School Grades and Ratings of Writing Ability (LSAC-56-1).

Hills. J. R. & Raine, W. J. (1959). Keying Pairs of Items for Predicting Law School Grades (LSAC-59-1).

Miller, Paul V. (1962). A Study of the Relationships Existing Among Scores on the LSAT, Writing Ability, and General Background Tests and between These Scores and Certain Other Candidate Variables (LSAC-62-1).

Pitcher, B. (1962). The Prediction of First-Year Law School Grades from Experimental Tests of Writing and Reasoning Ability and from the Law School Admission Test, 19459-1960 (LSAC-62-2).

Miller, P. V. (1963). A Repetition Study of the Relationship Existing Among Scores on the LSAT, Writing Ability, and General Background Tests and between These Scores and Other Candidate Variables (LSAC-63-3).

Winterbottom, J. A., Pitcher, B., & Miller, P. V. (1963). Report on the Readministration of the LSAT to Third-Year Students (LSAC-63-3).

Pitcher, B. (1965). The Law School Admission Test as a Predictor of First-Year Law School Grades 1962-1963 (LSAC-65-2).

Boldt, R. F. (1966). Predicting Law School Achievement with Discrepant Predictor Scores (LSAC-66-1).

Pitcher, B. & Schrader, W. B. (1866). Performance of Experimental Section (LSAC-66-2).

Schrader, W. B., Pitcher, B., & Winterbottom, J. A. (1966). The Interpretation of Law School Admission Test Scores for Culturally Deprived and Non-White Candidates (LSAC-66-3). (for the Ad Hoc Committee on Student Background Factors, Frank K. Walwer, Chairman).

Klein, Stephen P., Rock, D. A., & Evans, F. R. (1967). Predicting Success in Law School with Moderators (LSAC-67-1).

Boldt, R. F. (1968). Extension of the Discrepant Predictor Study – Final Report (LSAC-68-1).

Klein, S. P. & Evans, F. R. (1968). An Examination of the Validity of Nine experimental Tests for Predicting Success in Law School (LSAC-68-2).

Wills, Thomas A. (1968). Programmed Study Experimental Item Types (LSAC-68-4).

Boldt, R. F. (1970). Discrepant Predictor Study of LSAT and Writing Ability (LSAC-70-1)

Boldt, R. F. (1970). Discrepant Predictor Study Using Validity Study Results (LSAC-70-2)

Carlson, A. B. (1970). Factor Analysis and Validity Study of the Law School Admission Test Battery (LSAC-70-3).

Linn, R. L., Klein, S. P., & Hart, F. M. (1970). The Nature and Correlation of Law School Essay Grades (LSAC-70-4).

Schrader, W. B. & Pitcher, B. (1970). Interpreting Performance of Foreign Law Students on the Law School Admission Test and The Test of English as a Foreign Language (LSAC-70-6)

Angoff, W. H. & Herring, C. L. (1971). Study of the Appropriateness of the Law School Admission Test for Canadian and American Students (LSAC-71-1).

Schrader, W. B. & Pitcher, B. (1972). Discriminant Function Analysis in the Prediction of Law School Performance (LSAC-72-4).

Schrader, W. B. & Pitcher, B. (1972). The Interpretation of Law School Admission Test Scores for Culturally Deprived Candidates: An extension of the 1966 Study Based on Five Additional Law Schools (LSAC-72-5).

Evans, F. R. & Rock, D. A. (1973). A Study of the effects of Moderator Variables on the Prediction of Law School Performance (LSAC-73-2).

Pitcher, B. (1973). An Exploratory Study of the Possibility of Curvilinearity on Law School Validity Study Data (LSAC-73-3).

Schrader, W. B. & Pitcher, B. (1973). Predicting Law School Grades for Black American Law Students (LSAC-73-6).

McPeek, W. M., Pitcher, B. & Carlson, A. B. (1974. The Predictive effectiveness of Several Experimental Item Types and the Operational Item Types in the Law School Admission Test in 1970-71 (LSAC-74-1).

Pitcher, B. (1974). LSAT Part-Score Validity Study (LSAC-74-2).

Pitcher, B. (1974). Predicting Law School Grades for Female Law Students (LSAC-74-3).

Schrader, W. B. & Pitcher, B. (1974). Extension of the Study of the Possibility of Curvilinearity in Law School Validity Study Data (LSAC-74-7)

Schrader, W. B. & Pitcher, B. (1974). Prediction of Law School Grades for Mexican American and Black American Students (LSAC-74-8).

Linn, R. L. (1975). Test Bias and the Prediction of Grades in Law School (LSAC-75-1).

Pitcher, B. (1975). A Further study of Predicting Law School Grades for Female Law Students (LSAC-75-3).

Carlson, A. B. & Werts, C. E. (1976). Relationship Among Law School Predictors, Law School Performance, and Bar Examination Results (LSAC-76-1).

Linn, R. L. & Pitcher, B. (1976). Predictor Score Regions with Significant Differences in Predicted Law School Grades from Subgroup Regression Equations (LSAC-76-2)

Pitcher, B. McPeek, W. M., & Binkley, M. (1976). The Validity of Two Experimental Item Types – Issues and Facts, and Artificial Language (LSAC-76-3)

Pitcher, B. (1976). Subgroups Validity Study (LSAC-76-6).

Schrader, W. B. (1976). Summary of Law School Validity Studies, 1948-1975 (LSAC-76-8).

Powers. D. E. (197). Comparing Predictions of Law School Performance for Black, Chicano, and White Law Students (LSAC-77-3).

Linn, R. L. (1977). On the Treatment of Multiple Scores for Law School Admission Test Repeaters (LSAC-77-4).

Pitcher, B. (1977). The Validity of the Law School Admission Test Scores for Repeaters (LSAC-77-5).

Boldt, R. F. (1977). Evaluation of Three Methods for Treating Repeaters’ Scores (LSAC-77-6).

Pitcher, B., McPeek, W. M., & Binkley, M. (1977). The Validity of the Usage experimental Item Type (LSAC-77-7).

Rubin, D. B. (1978). Using Empirical Bayes Techniques in the Law School Validity Studies (LSAC-78-1).

Powers, D. E. (1980). A Consideration of the Possibility of Differential Predictive Effectiveness of Writing Ability Scores for Minority Groups (LSAC-80-1).

Powers, D. E. (1981). Predicting Law School Grades for Minority and Nonminority Students: Beyond the first-Year Average (LSAC-81-1).

Evans, F. R. (1982). Recent Trends in Law School Validity Studies (LSAC-82-1).

Linn, R. L. & Hastings, C. N. (1983). A Meta Analysis of the Validity of Predictors of Performance in Law School (LSAC-83-1).

Appendix B

Excerpt from Memo to Law Schools Announcing Permanent Validity Study Service

Read the Excerpt