Keeping Up to Data: November 2023

Welcome to the Keeping Up to Data^SM podcast, a space in which we discuss, analyze, and contextualize trends and perspectives in the current law school admission cycle by taking a deeper dive into the most up-to-date data and making sense of the complicated world of legal education.

SUSAN KRINSKY: Welcome back to Keeping Up to Data. I’m Susan Krinsky, LSAC’s executive vice president for operations and chief of staff.

The last time I recorded an episode of Keeping Up to Data was four months ago. The 2023 application cycle was winding down, and the U.S. Supreme Court had just announced its decision in the two cases brought against Harvard University and the University of North Carolina. I reference those cases because the decisions in those cases may explain why the 2024 application cycle has gotten off to a slow start, as many schools made changes to their applications and applicants sorted through new essay prompts on many applications.

About a month ago, applicants were down between 8% and 10%, and earlier in the application season, even more than that. But as of the middle of November, we are seeing over 17,000 applicants, and that is less than 1% fewer than last year at this time. So, although later than last year, the pace has started to pick up and applicants are applying.

Given the increase in LSAT test takers, this is not terribly surprising. As for application volume, again, as of the middle of November, applicants have submitted just over 95,000 applications, about 9% fewer than last year at this time. But a month ago, application volume was down about 16%. And it’s worth noting that during the first two weeks of November, we saw more applications submitted than in any of the previous four years.

Score release from the November test will be November 29, the week after Thanksgiving, and we’d expect to see another mini-surge of applications shortly after that date. As of today, 67 schools are showing increases in applications, 123 are seeing decreases, and six are showing no change.

Just over 40% of this year’s applicants identify as persons of color, more than last year’s percentage at this time. Currently, 55% of this year’s applicants identify as female, 1.25% as gender diverse, and 24% of this year’s applicants report that they are first-generation college students.

For the first four LSAT administrations of this cycle — August, September, October, and November — we had 10% more test takers than for the same four tests in 2022. And from July through October, 5% more purchases of CAS subscriptions than for the same period in 2022. As always, you can find the latest applicant trends and numbers on our website, which is updated daily, 365 days a year.

In 2019, LSAC entered into a settlement agreement with two blind individuals who asserted that the analytical reasoning section of the LSAT, commonly known as logic games, disadvantaged blind test takers because they could not easily draw or use diagrams to help solve these questions. Although this concern was not shared by all blind test takers, to address any concerns about diagramming, LSAC committed to research alternative methods for assessing analytical or deductive reasoning skills, as well as the extent to which those skills are assessed on other existing sections of the LSAT.

After extensive review of alternatives, LSAC has decided that the best way to continue to assess students’ reasoning skills is through the addition of a second logical reasoning section to replace the existing analytical reasoning, or logic games, section. This change will take effect starting with the August 2024 LSAT, which is the first test in the 2024-2025 testing year.

For students who plan to take the LSAT between now and June of 2024, there is no change. Their test will consist of one logical reasoning section, one reading comprehension section, one analytical reasoning section, plus an unscored section that enables us to test items for future use.

For students who plan to take the LSAT starting in August 2024, the test will consist of two scored logical reasoning sections and one scored reading comprehension section, plus an unscored section of either logical reasoning or reading comprehension.

The logical reasoning section has been a component of the LSAT for decades and was designed to assess many varieties of reasoning skills that are essential for law school success, including the deductive reasoning skills that were the focus of the logic games exercises.

Replacing the current analytical reasoning, or logic games, section with a second logical reasoning section will ensure that the LSAT continues to assess the reasoning skills that are so important to the study and practice of law while addressing the concerns that were raised about the use of diagramming. And because students are already very familiar with the logical reasoning section of the LSAT, we believe adding a second logical reasoning section at the beginning of a new testing year will have minimal impact on test takers who have already begun to prepare for the LSAT.

I’m very pleased to be joined by Anna Topczewski, who is LSAC’s director of assessment sciences, to answer some important questions about the new format. Welcome, Anna.

ANNA TOPCZEWSKI: Thank you, Susan.

SUSAN: You are the director of assessment sciences at LSAC. For those of us who are not assessment scientists, what does that mean?

ANNA: Susan, that means that I have the privilege of leading a brilliant team of hardworking individuals whose job is to ensure that we are measuring the skills of the LSAT and making sure we’re doing that well. We are the team that, after each administration, we’re analyzing all data to ensure that scores are fair, valid, and reliable. We’re making sure that the questions are the appropriate difficulty. And, while the test is built to be equivalent, there’s always slight differences, and the assessment sciences team is the one making those statistical adjustments on the back end to ensure that each and every LSAT score means the same thing despite test takers receiving different versions of the test. Because we know how important these scores are, we know that we need to get this right 100% of the time.

SUSAN: So, given this upcoming change, can you say that the LSAT is still a valid and reliable test? And how do you know?

ANNA: Yes, we can absolutely say that the LSAT is still a valid and reliable test. We have done extensive research that indicates that the revised approach to assessing reasoning skills has virtually no impact on the strong correlation between the LSAT test scores and first-year law school success. Research also indicates that the revised approach has virtually no impact on overall scores, and at the individual test-taker level, the overwhelming majority of individuals would have a score within their score band.

Our research further shows that the LSAT will remain a highly reliable test with a reliability equal to or even higher than the current version of the test. Why is all of this true? At the heart of it, the LSAT has and will continue to assess reading comprehension, reasoning, and writing skills that are vital to success in law school and the legal profession.

SUSAN: You said that overall, there is virtually no impact on scores. Does the research indicate that different groups of test takers — say, by gender or race, ethnicity or socioeconomic status — will be affected in different ways by this change?

ANNA: The research indicates that if any changes occur, they’re very small across the different groups.

SUSAN: What does “extremely small” mean?

ANNA: “Small” means point-something of a change — 0.1, 0.2. It’s a fraction of a score point.

SUSAN: Your team tested alternative approaches to measuring or assessing analytical reasoning skills on two separate occasions over the last four years. What did you learn from that research? Or, put another way, why didn’t you decide to use one of those alternative approaches that were tested?

ANNA: The 2019 settlement agreement envisioned several different options moving forward, one of them being replacing the analytical reasoning section with an additional logical reasoning section, another one being the redesigning of the analytical reasoning section, along with other options we looked at as well.

What we have done the last several years is to research all of these options. Some of them were more visible than others. Like you said, we had very public research that we did looking at the redesigned analytical reasoning section, where we had large-scale field studies in June 2022 and December 2022. Other research options were less visible — we could use existing data to look at this. In the end, we looked at all the research from all of the different options to determine which one is best.

It’s not really that one failed or one succeeded. It’s that each option had different strengths, and in the end, we wanted to choose the option moving forward that was addressing the concerns of the settlement but had very little impact on overall scores.

SUSAN: Will you use the same 120-to-180 score scale for the test starting in August? And if so, how can you have the same score scale given this change?

ANNA: The score scale depends on the skills being measured. This change in the test does not change the fact that the LSAT measures and will continue to measure reading comprehension, reasoning, and writing. The research also shows that overall, there is virtually no impact on scores. Together, these are part of the support on why we’re confident in using the same 120-to-180 score scale.

SUSAN: We’ve seen a lot of changes in the LSAT over the past four years: before the pandemic, a five-section test that was administered in person only, most recently on tablets, but before that, paper and pencil; then, a three-section test that was solely remote; then, a four-section remote-only test; and currently, a four-section test that is given both online and in person. And now, eliminating the entire analytical reasoning (or logic games) section. How should schools interpret results from these different tests?

ANNA: Susan, what’s key to remember is, through all of these changes, what has not changed. And what has not changed is the fact that the LSAT measures reading comprehension, reasoning, and writing. And throughout all of this, research has shown that the test is valid and reliable. In particular, the predictive validity research that looks at the correlation between LSAT scores and 1L GPA has found that that correlation has remained strong. And what we have found, even looking at the change for this upcoming test, is that that predictive validity will also remain strong.

SUSAN: For any given test taker, though, there could be a change, correct?

ANNA: Yes. We expect there to be changes at the individual test-taker level.

SUSAN: Does the elimination of the analytical reasoning section and replacement with a second logical reasoning section make the test easier?

ANNA: No, what we found is, again, at that overall level, the scores don’t change. So, reconciling the individual changes versus the overall changes, it’s important to remember everybody has strengths and weaknesses on the different skills tested on the LSAT. For those individuals who find analytical reasoning challenging, they might see higher scores on a test without analytical reasoning. But the reverse is also true: For those individuals who love analytical reasoning. they might see lower scores on a test without analytical reasoning. What research has shown is that these differences, when analytical reasoning is included or not, are still relatively small, and that the majority of test takers will receive a score that falls within their score band if they were to retake it with this new version.

SUSAN: And what does the addition of a second logical reasoning section really accomplish? Isn’t that kind of reasoning testing already covered in the first logical reasoning section? Why is a second logical reasoning section needed?

ANNA: Susan, test length is a very important factor in having a reliable test. Having the second scored section gives us a test length for a highly reliable LSAT, which means we are certain in the scores test takers are receiving.

SUSAN: One question we hear a lot is what a test taker should do if they’re not applying to law school this year. What advice do you have for those who are trying to decide if they should take the test before June, or wait until August?

ANNA: My advice is to know your strengths. Where are your skills at? How are you doing on the analytical reasoning section, logical reasoning section, and reading comprehension section? Is analytical reasoning your best or worst section? That is something that will help you know what to do, what’s best. But that’s not all of it. You also have to know how much time you have to study in the upcoming months. Do you have time to study and then take the test in April or June, or would it be better if you waited to study over the summer and really get prepared for August? It’s really going to depend on each person, and luckily, you have the time to ask yourself these questions and develop a plan, which is why we knew we needed to make this announcement now.

SUSAN: Thank you so much, Anna.

ANNA: Thank you, Susan.

SUSAN: Thank you to our listeners. Thank you for joining us at Keeping Up to Data. We look forward to your joining our next episode, when we will continue to take a close look at the data from the current admission cycle. Until next time, stay well, and I hope you have a peaceful holiday.

Thank you for joining us. Keeping Up to Data^SM is a production of LSAC. If you want to learn more about the current law school admission cycle and the latest trends and news, visit us at LSAC.org.

Back to Keeping Up to Data