Wednesday, September 28, 2016

The Bar Exam and "Math."


Florida, we have a problem.
Shots were fired over a year ago as to this topic. On the one side was Dean Allard of Brooklyn Law School, leading the charge - "the bar exam is too hard! It's unfair!" On the other side was the NCBE - "current students are 'less able'! Buckle down!" If anyone should be on the side of mathematical analytics, is would be Bloomberg.
As is often the case in many things, the truth is likely somewhere in the middle. Given my own participation in the debate over the years and my own leanings, it is safe to say I am still in the scamblog camp - this is, I believe the charges of bar exam difficulty are heavily weighted in the "hey, this affects my livelihood so stop looking at the data so hard" direction, not the more-politically-palatable proclamation that "diversity is super-important, how dare you try to interfere with that using an arbitrary test" direction. When the Law School Cartel has gone from laughingly-dismissing the scambloggers to calling them demons over the years, especially in the face of falling bar passage rates, I get a bit cynical. Then again, I am a product of the self-same system, so the Cartel reaps what the Cartel sows, I guess.

That said, let's talk about the major rejoinders from the Cartel, and what I find to be a fascinating debate over at the Faculty Lounge. Per my reading, they fall into three basic categories:

-more below the fold-


(1) Bar passage rates have a numerator (student ability) AND a denominator (test difficulty). Scambloggers say it is the former, we, the learned academics, say it is the latter.

(2) The bar exam does not properly measure or reflect the aptitudes necessary to be a successful lawyer.

(3) Various State Bars have been hiking difficulty either intentionally or unintentionally, but the fact remains.

As for the first argument, yes, it is safe to assert that there is more than one variable that can derive a calculated result. In fact, if we could quantify them reliably and accurately, I would bet there are several variables, not just a numerator and denominator. The mere existence of a variable does not mean that the variable is definitively in play, however. Kyle McEntee at LST, David Frakt, Paul Campos, Matt Leichter, the NCBE, various scambloggers, and many, many others have discussed and provided evidence of declining test scores and aptitude scores. Merely stating that, "well, the test could also be harder" is possible, but not proof. The current proof (test scores, performance metrics) bears statistical and historical weight, and are valuable for that reason alone. If, for example, raw MBE scores are not an "accurate" function box, what other measurement does one propose to try to draw inferences from the past to the present?

As for the second argument, yes, there could be better items to test. Deborah Merritt, who I consider a reasonably staunch law school reformer and general friend to the scamblog camp (though she may disagree), itemizes several ideas that could be tested:
 

1. Ask multiple choice questions that are a realistic measure of what every lawyer needs to know by heart before beginning practice...

2. Alternatively, maintain the current level of detail but make the MBE open book...

3. Replace essays with more MPT exercises...

4. Find a way to test for basic skills in interviewing, communicating with clients, negotiating, and other skills that everyone agrees are essential for a lawyer to have...
My only issue with these is that, well, possibly they are valid considerations, but the are not exactly "mathematical" nor tied to objective measurements. Again, the charge from the Cartel is that scores are down due to difficulty, and the subsequent statistical analysis of cut scores, equivalency mappings, harsh subjective grading, etc. generate a false negative that wrongly impugns the Cartel and student performance. Again, a worthwhile debate, but a distraction from the overall charge. Testing different metrics does not directly address what the issue is with current declining metrics, metrics that have been in place for decades.

As for the third charge, possibly State Bars have indeed made passing requirements more strict. Note that is not the fault of the MBE, for example, but decisions made by various State Boards of Bar Examiners. If Law School X believes that State Y's Bar Exam evaluation is unfair, that has little to do with the "math" of the Bar Exam. Maybe it has to do with essays, or MPTs, or the weight items carry, or something else. Perhaps Law School X, and others similarly situated, should dig in and have some real discussion with their State Bar concerning how they are interpreting the results, how they are determining passage rates, and the like. Maybe Law School X would be surprised by the results, or perhaps they would be "vindicated" and State Y will change their ways. Again, this is not necessarily "math" as far as the Bar Exam proper is concerned.

In any event, read the comments for yourselves, as the voices there articulate the issues better than myself. See what persuades you, personally.

In closing, one can always posit a theory. Demonstrating the validity of the theory requires evidence and supported inductive reasoning, which in my opinion is sorely lacking on the part of the Cartel defenders as of this date. Especially where, as here, pocketbooks are ultimately concerned.

34 comments:

  1. So 1/3 can't pass the bar on the first try... At this rate, if you graduate next year and you pass the bar on first try, you might have a healthy chance of getting that coveted JD required long term position that gives you 1/6 chance to pay back 'em loans.

    ReplyDelete
  2. Captain Hruska Carswell, Continuance KingSeptember 28, 2016 at 2:42 PM

    I started to read that same Faculty Lounge post and I usually comment on that site. However, the number of comments reached 42 and were verbose---long winded. I stopped reading. Those posters are not practicing attorneys and have no clue.

    If you can't explain a concept within five words, you lost the argument.

    I will violate Captain's Maxim #2 and will tell you in twenty words. Law School is a bad deal today because it does not provide an opportunity as it once did, for a sustainable middle class income regardless of Bar scores.

    ReplyDelete
  3. Although law school admission standards have fallen practically across the board, I think it is the near elimination of standards at the worst schools that is most responsible for the decline in bar passage rates. For example, I doubt that a decline in the median LSAT score from 165 to 162 would have much of an effect on the bar passage rate at a school like BC or Fordham. Similarly, I don’t think it’s a big deal (as far as bar passage goes) if the median LSAT a T2 school goes from 159 to 156. However, once you get down into the 140s territory, it’s a different story.

    Time was, law schools didn’t accept applicants with LSAT scores in the 140s except for a few bottom feeders like Cooley, and even those schools had some standards, usually around the 147 mark. All of that changed four or five years ago when applications and enrollment dropped like a rock. The schools needed to put asses in seats and loan dollars in their bank accounts so admission standards went out the window. A lot of T4 schools started reporting median LSATs in the 140s and 25% figures in the low 140s and even high 130s. Many of the students behind those numbers (who are now sitting for the bar) have basic deficiencies in critical areas such as reading comprehension, logical reasoning, and overall intelligence which preclude them from passing the bar. And to my mind, whatever else you want to say about the bar exam, to the extent that it keeps thousands of low aptitude graduates out of an already overcrowded "profession" - it serves a useful purpose.

    ReplyDelete
  4. Anyone who does not see the clear correlation between the lower admissions standards at most law schools three years ago and the lower bar exam scores of today ... has nothing to teach anyone.

    ReplyDelete
    Replies
    1. some 55 y.o temp my firm hired to help out during busy season told me if i'd get my law degree (to complement my cpa license) that i'd be golden. I had to refer her to ITLSS and OTLSS. it's a lost battle to tell these old fucks that the law degree is not worth it anymore except for a very small sliver of ppl.

      Delete
    2. And of course nobody knows going in whether they will be part of the "sliver," even coming out of HYS.

      Delete
  5. Other kinds of math which law school students (read: LEMMINGS) ought to know...
    1. Non-dischargeable debt with interest after leaving law school
    2. The cost of starting and running your own solo practice (go ahead, laugh at that notion, get it out of your system)

    Hm, speaking of which... sorry if this really is off-topic, but I found and wanted to share this rediscovered Internet gem: Scotty Bullock (a.k.a. "Law is 4 Losers") taking the Solo Practice University folks to task over the pipe dream of having a solo practice these days:
    http://solopracticeuniversity.com/2010/01/11/dont-be-a-victim-of-the-victim-mentality/
    (Scroll down, you'll see it. Also, notice how the lady doesn't answer or respond to his questions about what he could have done differently.)

    And while I'm at it, another lost gem of his:
    http://whydoeslawsuck.blogspot.com/2006/09/more-great-words-of-wisdom-from-jdjive.html

    ReplyDelete
    Replies
    1. Even to a Solo like me out of law school over 20 years, Solo practice is very frequently underemployment. No work for long periods of time. At least I started out my career as a gub'mint attorney and gained some seasoning and experience and then went out on my own. It was good until 2006. About 75K consistently. Then the bottom fell out and has not returned. That is correlated with the bottom ranked law schools dumping legions of newbies into the market who couldn't find jobs and beginning their careers as Solos and overcrowding "appointed work" lists. Not enough to go around for everybody. Like trying to split an Arby's Roast Beef sandwich among 20 hungry people. For them, it is functional unemployment to go to court for next to nothing. There are billboards posted in and around the Chicago area advertising $49.00 traffic ticket defense. Ask yourselves: How do you think that works out for the client?

      Delete
    2. Aye, there's the rub. There is a constant thru-put of newbies who slash fees to get business for a year or two then go under only to be replaced by a new crop.

      Delete
    3. Not sure if they still exist, but there was a chain of low cost women's clothing stores called "Dots" because the price tags were color-coded disks, e.g., all $5.00 items would be marked with an orange dot. There used to be one in my city and a female friend who had gone in to look around reported that the clothes appeared as if they would not survive three trips through the washing machine. We then discussed that the only advertising they did was to leave photocopied flyers around, concluding that you'd have to sell a whole lot of $15.00 dresses to justify an ad in even our relatively inexpensive local daily paper.

      On that theme, one has to wonder how many $49.00 traffic tickets you'd have to handle just to cover the monthly rental on the billboard and start to turn a profit.

      Delete
  6. One of the main problems is that the professors who teach the various subjects that appear on a particular state's bar are rarely licensed in that state and almost never have practiced in that area of the law in that state. Of course the academy doesn't see the value in experience but I would bet that the students who are taught bar subjects by adjuncts fare a lot better.

    ReplyDelete
  7. Florida has an inordinate number of trash heap schools. For the 2013 entering class (i.e., the people who sat for the July 2016 bar), six schools had median LSATs below 150 - Ave Maria, Barry, Florida A&M, Florida Costal, Nova Southeastern, and St. Thomas. Also, I believe Cooley has a campus in Florida and you can bet your ass that the median LSAT there is well below 150. In contrast, of the 15 law schools in New York (which is way too many), only one school has a median LSAT below 150. That would be Touro. Also, Florida does not have anything close to a T14 or near T14 school to bring up the passage rate. It has two or three schools that are halfway decent and everything else is shit.

    ReplyDelete
    Replies
    1. 6:46 A.M. UF and FSU are obviously good law schools, as good as any of them out there. UM is also a very good school, albeit way too expensive. What I find interesting is even among the Scambloggers,too many of you are truly elitists. Too many of you think that only the top 20, etc. mean anything. You who think that have no idea what it takes to practice law have probably very limited experience in practicing law, and I can assure you that it does not take a top twenty school to be an excellent lawyer. That lawyers graduating from schools below the top 20 have trouble getting jobs is only because this field is filled with dysfunctional elitists. Lawyers are aholes . . and your post 6:46 AM is just proof of the dregs who populate this profession. that's why I recommend my daughters to stay away from law school, its the absolutely horrible people who populate the profession.

      Delete
    2. It's not about élitism. As I have shown before, only 13 law schools see even half of their graduates get the sorts of jobs that pay enough to support the debt from law school. Only 16 see 40% of their graduates get those jobs. Only about a third of the law schools see 10% of their graduates get those jobs. (http://outsidethelawschoolscam.blogspot.ca/2014/12/guest-post-by-old-guy-which-law-schools.html)

      Even if the University of Horrida and Horrida State University have excellent law schools, still no one should attend them.

      Delete
    3. To some extent, I agree with you, 10:59 AM. However, it's not the scambloggers that are elitist; it's the profession itself. Don't shoot the messenger.

      Delete
    4. 10:59 thinks that FSU, Miami and U of F are "as good" as Harvard, Yale, and Stanford. I don't agree with that. He also thinks elitism is what prevents all those Barry and Florida Coastal grads from finding jobs. I don't agree with that either.

      Delete
  8. —— Ask multiple choice questions that are a realistic measure of what every lawyer needs to know by heart before beginning practice

    No. Anything that "every lawyer needs to know by heart" should be tested directly, not through multiple choice.

    —— Alternatively, maintain the current level of detail but make the MBE open book

    No. The exam should be made more difficult, not easier.

    —— Replace essays with more MPT exercises

    No. Maintain the tests of writing—and score them to a stricter standard.

    —— Find a way to test for basic skills in interviewing, communicating with clients, negotiating, and other skills that everyone agrees are essential for a lawyer to have

    No. This is just a way to smuggle "soft skills" in as a distraction.

    ReplyDelete
  9. The argument about the Florida bar being harder because the pass rate of experienced lawyers is just a few points higher than that of non-experienced lawyers is probably a sham. Most of the people taking the exam are likely to be recent graduates whose credentials are weaker than earlier graduates.

    ReplyDelete
  10. This year the entering classes at Harvard and Yale Law have lower LSAT scores than in the past. Columbia raised its LSAT scores over last year but at the expense of the 25th percentile GPA. Stanford has not yet given out its statistics.

    Given the dearth of better law schools near Stanford and Chicago and their small size, they may be more insulated from the declining quality of the application pool than the eastern law schools.

    Applicants are starting to realize that it is a bloodbath out there for lawyers. Similar to what it always was then you got a PhD at a top graduate school and then did not get tenure. You could be subject to a lifetime of working as an adjunct at $4,000 a class, with short-term appointments, no health insurance and wearing out your car to drive to your three teaching engagements in one academic year. You never know where your next paycheck is coming from.

    Law is like that now, even if you go to Harvard or Yale. It is easy to end up unemployed a little down the road with no job opportunities that use your elite law degree. Thank you ABA and not challenging up or out policies that leave especially older women and minorities and also a not insignificant number of older white men with worthless Harvard and Yale Law and other top law school degrees.

    Young people are getting the message. Unfortunately, not enough of them are getting the message before it is too late and their lives are in shambles due to underemployment and unemployment in the legal profession and being too old and indebted to be able to train for a career where the supply and demand are more balanced.

    ReplyDelete
    Replies
    1. Only about 2000 people per year get scores of 170 or better. (Just a few years ago, the number exceeded 3000.) Many of them won't go to law school at all. Of the others, many will end up at an NYU or a Michigan or even a Minnesota.

      Thus Harvard and Yale find it more difficult to draw in people with scores in this range. They dip deeper and deeper into the 160s, thereby encroaching on the territory of the Penns and the Cornells. These in turn cut into the traditional range of the UCLAs and the Emorys. The cascading effect continues all the way down, to the point that _UNT and Indiana Tech and Thomas Jefferson have to take people whose scores are scarcely better than what one would expect from random guessing.

      Delete
    2. The fact that going to an elite law school does not guarantee you any type of employment is very different from what one would expect. You assume someone with double Harvard or Princeton and Yale degrees can always find work but that has not been true for lawyers for at least the last 20 years. If you have those degrees and your law degree is from Harvard or Yale and you are unemployed or underemployed, what is going on? These are people who had extraordinary success all of their lives until middle age. The answer is that law is no longer a good career. It is too risky in terms of job security for a career to be a good bet any more. I am not talking about job losses for a few months. In these cases, losses of an in house or law firm job was a permanent job loss. Maybe the lawyer can temp after that, but even temping is competitive because of the immense lawyer oversupply.

      Bottom line - you may have wasted your opportunities by getting a Harvard or Yale Law degree if you cannot find full-time permanent work that uses a law degree or other comparable work after middle age.

      Delete
  11. The argument that a possible change in the denominator (an increase in passage requirements by the state) is a valid excuse for poor bar passage rates is a flawed one. It might be valid in the short term, before law schools can adapt, but it misses the point of the bar exam. The bar exists is not a test to determine how good someone will perform as a lawyer. I don't know any lawyer who would even say that with a straight face. It is an exam of minimum competency. The ABA doesn't determine what that minimum standard is, and neither does the law school. Each state has the right to determine what the requirements should be to practice law, and that requirement shouldn't change for the convenience of the law schools.

    Even if the requirements are higher now than they used to be,which I haven't seen any evidence of, bar exams are still there to determine minimum competence. If schools are producing significant numbers of students who can't pass, they're producing significant numbers of students who aren't minimally competent. If law schools are going to take high tuition from government loan programs and saddle students with non-dischargable debt, they have an ethical responsibility to ensure that all of their students are capable of passing the exam to prove they are minimally competent to be lawyers. It doesn't matter if those students are theoretically as good as they were 10 years ago. If they can't pass the bar now with a harder exam, they don't need to be in law school.

    ReplyDelete
  12. —— Bar passage rates have a numerator (student ability) AND a denominator (test difficulty).

    No: the numerator is the number of candidates that pass; the denominator is the number of candidates.

    Yesterday at Third Tier Reality I posted this:

    —— The reality is that if one ABA school gets a passage rate that is above the state average, another one will be below it.

    Idiots. Look at the following:

    12
    12
    12
    12
    2

    The average is 10. Four of the items are above the average, but only one is below.

    People who don't know what an average is shouldn't be teaching anything more intellectually taxing than underwater basketweaving.

    I'm beginning to think that a course or two in the integral calculus should be required for admission to law school, just to screen out the sorts of pretentious "learned academics" and Cooleyite morons who don't understand ratios and averages.

    ReplyDelete
    Replies
    1. Clearly, if you take test difficulty down to 0, the bar passage rate is NaN. (infinity) How can that not make sense?

      Sheesh. I think you're too generous on the underwater basketweaving comment; I think the OP would drown.

      Delete
  13. —— The bar exam does not properly measure or reflect the aptitudes necessary to be a successful lawyer.

    I fully agree. There are far too many lousy lawyers, almost all of whom passed the bar exam. Until the exam keeps most of these people out, it won't validly measure what someone above called minimal competence.

    But this complaint has nothing to do with the shabby performance of so many law skules' graduates. The bar exam is what it is. With a few exceptions (such as people covered by exemptions in New Hampshire and Wisconsin), everyone seeking to become a lawyer will have to pass the exam. Law schools know that. They also know that almost all of their students are undertaking their studies for the purpose of becoming lawyers and therefore both expect and need to pass the bar exam. They also know, or should know, that applicants with LSAT scores below 150 run a high risk of never passing, and that those below 145 are very unlikely to pass. Yet they go on irresponsibly admitting thousands of such applicants, and inducing them to spend not only three years of their lives but also hundreds of thousands of borrowed dollars in the pursuit of an unattainable goal.

    ReplyDelete
  14. Does anyone really believe the bar exam is getting harder? Look, these people (the law school cartel) are professional con artists. They will say and do anything to deflect responsibility and keep the scam going. The bar exam isn't getting harder; the graduates are getting dumber.

    ReplyDelete
    Replies
    1. Captain Hruska Carswell, Continuance KingOctober 3, 2016 at 4:02 PM

      Most Prawfs and Deans are not "con artists." They are academics and many are our colleagues. What happened is that some con artist for profit and recently accredited law schools with the ABA's blessing came on line and dumped lawyers onto a glutted market. That forced incomes down and rippled to all levels of the market. That resulted in a vicious downward spiral....less income and work for everyone in the legal profession created a disincentive for the best and the brightest enrolling in law schools. The numbers went down and the standards were lowered. The delicate system is out of kilter.

      Delete
    2. Captain Hruska Carswell, Continuance KingOctober 3, 2016 at 7:49 PM

      The best analogy is the Walmart affect for the legal profession.

      Delete
    3. It is possible to be an academic and a con artist simultaneously. They've been profiting off of lies for years.

      Delete
    4. If you're selling law degrees to people you know will not be able to gain licensure, you're a con artist.

      Delete
  15. curious. . . not sure if I remember correctly, its been more than 35 years for me, but I seem to recall the LSAT was graded where the average score was say 500 and a good score something like 650 or 700? Am I right about that? So a 500 then is equavalent to a 150 now? If so, the average college grad has an IQ of say 110. Surely an IQ of 110 is smart enough to pass the bar exam?? So if 150 is average, why are you suggesting running a risk of never passing (note, I took three bar exams in three states and passed them all with flying colors . . . I'm pretty smart . . but as I recall, my LSAT was somewhere in the 600 range if I am remembering right)

    ReplyDelete
    Replies
    1. Up until 1982, the LSAT was scored on a scale of 200-800, which was the scale when you took the exam. Between 1982 and 1991, it was scored on a scale of 10-48. From 1991 to the present, it has been scored on a scale of 120-180.

      I took the LSAT in the fall of 1989 and got a 41 which, based on percentiles, is roughly equivalent to scoring a 165 today. For whatever it's worth, that was only good enough to get me into a second tier school in 1990 and the (2015) 75% LSAT at my old school is 158.

      Delete
    2. For whatever this contributes, I took the LSAT in December 1981 on the 200-800 scale. I scored a 675 and that was 89th percentile.

      Delete