On March 12, 2013, University of Pittsburgh Law School Dean William Carter gave a thirty-six minute long talk to students and others to address the dreadful news that his school had declined from 69th to 91st in the annual US News and World Report law school ranking. And Carter did not even bear any responsibility for the catastrophe, having just been appointed dean a few months earlier.
https://www.youtube.com/watch?v=VQxxNBCUqJ8
https://www.youtube.com/watch?v=VQxxNBCUqJ8
I figured that Dean Carter’s presentation would be a hilarious and exasperating fog of crisis management PR plus boosterism. But instead I found myself nodding in agreement at many of his points. Carter attacked the flawed nature of US News ranking metrics, which is hardly surprising under the circumstances, but he did so in an a manner that I found to be thorough and informative. To be clear, Carter’s presentation was not scam-free--I mean, he is a second tier law school dean. Still, when a second tier law dean is battling US News’ ranking guru Bob Morse, he may well be the lesser of two very evil evils, like Scylla or the Democratic Party.
Carter went through the components and sub-components that comprise the US News law school ranking, and argued that US News' metrics are highly unreliable-- indeed, that an effort to look good for US News by changing law school practices in certain ways could actually hurt students. Carter's presentation is a good springboard for a discussion of exactly what the US News law school ranking measures. Carter’s points are in roman, and my comments are in italics.
I. Reputational Survey of Law School Deans and Faculty (25%):
The US news reputational survey consists of a bubble sheet sent to law school deans and selected faculty. These survey-takers are asked to rate every law school on a 1-5 scale, though there is a "don't know" option. Academics who fill out the survey are usually only familiar with a handful of schools. As for the rest, their primary knowledge of a school is often its place in the previous year’s US News ranking, thus creating an "echo chamber" effect in the current year’s ranking.
The importance of this factor--25%!--sheds light on why law professors have such disgusting (I mean, collegial) personalities and why you will often find them traveling to, speaking at, or hosting schmoozefests, aka conferences. You see, a law school’s reputation may depend on its professors generating favorable chatter and gossip among colleagues, leaving a dim impression, reinforced by promotional material (aka "law porn") that significant and exciting scholarship is happening at those professors' home institutions. That way, later on, those colleagues may color in the "4" or "5" bubble on the US News reputation survey, even if they have no direct experience of the school.
II. Reputational Survey of Lawyers and Judges (15%):
US News does not disclose which lawyers and judges are asked to fill out the survey, though the lawyers are known to be Big Law partners. Only 9% of those asked to fill out the survey actually responded-- kudos to the practicing bar for so thoroughly disregarding this nonsense. The lower the response rate, the greater the opportunity for skew.
III. Selectivity (25%):
a. LSAT (12.5%);
b. Undergrad GPA (10.0%);a. LSAT (12.5%);
c. Percent Accepted (2.5%)
Carter complained that his school was punished by US News because his school’s median LSAT fell from 159 to 158. He said: "Do I think that there is a substantive difference between a class that had a median of 159 and a median of 158?. . .I simply refuse to accept that."
Carter is wrong here. Selectivity matters; it is the only thing that does, other than placement and cost. By punishing schools that lower admissions standards, the mischief-making US News ranking can actually have a virtuous effect. Schools are no doubt sorely tempted to lower, or even abandon, admissions standards, as law school applications dwindle due to growing awareness of the scam. The one thing that prevents them from doing so is the prospect of a tumble in the US News ranking. "Percent accepted," particularly, should be worth way more than 2.5%. I mean, the LSAT can be taken again and again, and a high GPA may only mean that a kid took easy classes in college, but there is no getting around a law school rejection. So I say: bonus US News points when a law school refuses to accept some young moron’s borrowed fortune in the interest of maintaining its selectivity.
IV. Placement (18%):
a. Jobs Nine Months After Graduation (14%);
b. Employment Rate on Date of Graduation (4%)
US News changed its methodology this year as to how it calculates employment rates. Formerly, employment was calculated in absolute terms, with full credit for all jobs, including short-term jobs. Now, however, placement success is calculated by assigning various weights to the numbers of graduates employed in 22 different categories of jobs and durations of jobs. US News provides full-credit for JD Advantage jobs lasting more than one year. As well, US News does not discount law school funded jobs, giving an enormous advantage to schools that subsidize fellowships for recent grads.
US News gives full-credit for JD Advantage jobs? If US News is going to make such a monumental blunder, who cares about its careful weighting of 22 different categories of jobs and job durations. Since the definition of a JD Advantage job is so imprecise, schools have every incentive to scam-up their employment success rates by characterizing any old kind of job as JD Advantage. Thus, a third tier scam school can characterize a graduate's job peddling insurance as JD Advantage and the school will receive the same US News credit in the placement category as Stanford receives when it places a grad in a SCOTUS clerkship.
US News gives full-credit for law school funded "jobs"? That means that a law school that charges above average tuition, and uses the extra tuition money to subsidize one year long "fellowships" (i.e. stipends for some of its unemployed recent grads to go volunteer full-time somewhere) gets rewarded by US News as it calculates the rankings. I would rather law school be compressed into two years, after which grads can volunteer their services to practitioners in exchange for training, with a substantial savings in time and money.
V. Faculty Resources: Expenditure Per Student (9.75%):
This measure, an alleged proxy for quality, is based only on a law school’s budget divided by number of students enrolled. It doesn’t matter what the school actually did with money and, of course, the money need not be spent on students per se. A school can immediately boost its score in this category by either: (a) reducing the number of students or (b) increasing its budget. Carter stated that he could boost his score in this category by increasing tuition, but would rather take a US News hit than impose additional tuition burdens on his students. Carter asserted that he could make Pitt a US News top 50 school within two years by boosting tuition by $5,000 a year, but he refuses to do that.
Carter is correct. This factor is crude and monstrous. Any school can get a US News rating bump by increasing tuition. A prospective law student who attaches importance to a school’s US News ranking may want to think that one through. He or she may choose a school based on its being ranked slightly higher than some other school, without realizing that that school is ranked slightly higher solely because it charges higher tuition (producing a higher average expenditure per student). So that edition of US News may end up costing a kid thousands of dollars, not $9.95 (or $29.95 for the expanded online edition).
To show how much US News really cares about actual quality education, as opposed to its phony-baloney proxies, consider: it doesn’t matter to US News what the money is spent on, only the average expenditure per student. The money can be spent on renovating the Dean’s Office or on free booze for the faculty. Hell, a school can double tuition and spend the whole increase on a 40-foot tall solid gold statue of a naked Bob Morse with an extended middle finger. The boost in this category would be no different than if the additional money was spent on some pedagogical miracle capable of turning every dim-bulb Cooley student into Clarence Darrow.
VI. Faculty Resources: Student-Faculty Ratio (3%):
In this category, schools are ranked based on how many full-time faculty they have relative to the size of their student enrollment. Carter notes that U.S. News doesn’t count adjuncts, "who can often be some of the most valuable teachers in the building."
A law school with the maximum permissible number of adjuncts would strike out with US News in this category, and yet probably provide a better and more cost-effective professional education than one filled to the brim with tenured six figure salaried windbags, many of whom would not know the difference between a courtroom and a faculty lounge. Here, again, Morse’s formula has a deleterious effect on legal education as it exists in actual reality, in favor of his imaginary world where prestige and quality can be precisely measured and assigned a numerical rank.
VII. Library (0.75%):
This category does not seek to measure how much support a school provides to its library. It measures only one thing: number of bound volumes in the school’s library. It does not measure the quality of information services, only the number of books. Even law librarians think this is nuts in the digital age. Carter quotes the chief librarian at his school as saying. "I don’t need more books...this is the year 2013."
Okay. 3/4 of 1% of a law school’s US News rank is based on the number of bound volumes gathering mold in the stacks. Bound volumes--in the high noon of the digital age. Has Bob Morse heard of Westlaw? Kindle? I truly hope that Bob Morse is having an affair with some legal books dealer. Because the only other explanation is that he is raving mad.
VIII. Bar Passage Rate (2%) (based on plus of minus deviation from average pass rate in the state):
IX. Financial Aid (1.5%):
X. Conclusion:
Until only two years ago, US News was basically all that prospective law students had to go by. Now, however, there is much more data available because, starting with the Class of 2011, the ABA has required law schools to survey their graduates, nine months out, as to employment status. There are some fine sites to help prospective students and their influencers access and interpret the data (below), and the US News ranking is not one of them. If you must have a ranking, check out Above the Law's-- it is superior to that of US News in many ways, chief among them that it does not count alleged JD Advantage jobs.
Nobody should care about US News’ idiotic survey of academic gossips, or its obnoxious "expenditures per student" proxy for quality, or its concern for the vastness of each law school's collection of bound volumes.
Nobody should care about US News’ idiotic survey of academic gossips, or its obnoxious "expenditures per student" proxy for quality, or its concern for the vastness of each law school's collection of bound volumes.
http://www.lstscorereports.com/?r=other&show=jobs
http://abovethelaw.com/careers/law-school-rankings/
http://educatingtomorrowslawyers.du.edu/law-jobs/
http://thirdtierreality.blogspot.com/
http://www.lawschoolcafe.org/
Just the fact that a law school can take such a fall (or rise) in a single year tells me the rankings are horse shit. For better or worse, Pitt is the same school it was 12 months ago. That said, if Pitt had shot up 22 slots instead of falling 22 slots, something tells me the Dean would have had nothing but good things to say about the rankings. You can't have it both ways man.
ReplyDeleteThis is a great entry. I think you should do a series debunking each of the U.S. News ranking factors.
ReplyDeleteThat reputation accounts for 40% of the ranking score is a travesty. At best, "reputation" is an echo chamber for selectivity (25%). At worst, it is a vehicle for corruption. The 9.75% allocated to money wasted is an absolutely perverse moral hazard. Yuck, yuck yuck!
Any proper "ranking system" would consider inputs (i.e. LSAT) and outputs (i.e. employment as a lawyer, weighing quality of employment outcome). What else matters? I would put more emphasis on outputs than inputs.
Decide on whatever measuring metric you believe is fair but please remember the stark fact remains that the academy has long been overproducing graduates --now quite probably by double. The market no longer sustains anywhere near the output. In fact, the oversupply greatly harms and coarsens an already brutal practice. One way or another, the output will decline.
ReplyDeleteWhat role can the rating system(s) play in this?
The New York Times, huff post or WSJ should pick up this post. Excellent work.
ReplyDeleteDo you know how to get these posts more widely circulated?
DeleteThere is a lot of truth to what Dean Carter said. However, this doesn't prevent law school deans from bragging about their US News rank when it is good. At Hofstra, the administration bragged about being 11th on some unimportant public interest law school ranking. Then when Hofstra dropped from 89th from 113th in US News this year the dean downplayed it and blamed it on the previous dean.
ReplyDeleteDespite what Carter said there is a big problem for those of us who graduate from Pitt or Hofstra- employers read the rankings. With this terrible job market it is even worse when you've gone to a law school that just dropped a lot. I know a lot of people are worried at my law school.
All this is noise. It's a distraction. Nobody cares about the rankings. People/employers/the world cares about money. "How much money can you make for me?" If you went to Harvard and can't generate revenues, you're getting cut loose by end of the year. If you went to Cooley and can generate $2MM in bills, you're a damn rockstar and partner.
ReplyDeleteRespectfully disagree. We know the rankings are trash. But I think that 75% of 21 year old law applicants who are inexperienced and dumb and lazy and whose parents are excited to have a lawyer in the family "at last!", they still scour the US News rankings as their main source of info, and think that attending a school ranked at 47 is better than the one at 49. And the dumb boomer parents, who are prestige-crazy, will be able to brag that their kid is going to a top 25 or top 50 law school.
DeleteThe rankings are still the Bible. That will never change. We need to dismantle this religion.
This is a great article. I didn't realize how idiotic those USNWR rankings were. # of bound volumes in a library? WTF does that matter? Who does legal research with books anyway? This makes no more sense than Cooley's emphasis (in their own "rankings" that have them 2nd) on square footage of the library.
ReplyDeleteNot to be rude, but ^^^this^^^ is why the rankings still hold sway. The breakdowns are in there for anyone who actually buys them, but most people just look at the free BS version and never bother looking at what the rankings are made up of.
DeleteThat 1 point LSAT differential makes a difference. If the curriculum is the same as dictated by ABA or whatever regulatory body LSs follow, then the real value of attending a LS is its signalling power to employers. Thus, LS ranking should place heavier emphasis on the quality of its incoming class, using the LSAT as a proxy.
ReplyDeleteThe "Reputational Survey of Law School Deans and Faculty" (except for the bizzare sitaution involving the University of Illinois)is the most stable and reliable portion of the US News rankings. It is also the least susceptible to manipulation by the law schools.
ReplyDeleteHere Pitt was 55th: http://taxprof.typepad.com/taxprof_blog/2013/03/2014-us-news.html
Here Pitt was 50th: http://taxprof.typepad.com/taxprof_blog/2008/03/2009-us-news-pe.html
That is not much change over the last five years.
Stable, reliable, and utterly irrelevant. Who cares what a bunch of law professors think of each other? Why should that matter to prospective law students?
DeleteHeck, most of the time they aren't even looking at law schools.
DeleteLook at Drexel. Unranked in 2009 because it just graduated its first crop of students after being founded in 2006. It's in a saturated market and currently has 0 graduates that have been out for more than 5 years.
Yet Drexel wound up with 2.0, ahead of schools like Texas Tech, Cleveland State, North Dakota, South Dakota, Toledo, Washburn, John Marshall (Chi), Golden Gate, etc., places that are staples of the legal community in their areas and have generations of graduates.
Northern Kentucky has been around 120 years, has roughly equal employment numbers to Drexel, and is a fraction of the cost. Why should it suffer because a bunch of jackwads think the Philadelphia private school with the prestigious name is a more preferable place?
Look at Campbell and Elon. Campbell has been around 40 years and an employment score in the 60s. Elon just started up in 2006 at has an employment score in the 40s. They're down the road from each other and graduates are likely competing head to head for many opportunities. Campbell draws better students.
Oh, but Campbell is a middling Baptist school, while Elon is recognized nationally-known lib arts school.
Who cares?
The deans made this problem and they could stop it if they wanted to. They could just refuse to participate. Instead they focus their strategy on succeeding in the rankings. They spend their money on moving up in us news and it doesn't matter if they're honest while they do it. Lie a little; everybody does it.
ReplyDeleteMeanwhile their victims are jobless and in debt. And its even worse for graduates of schools that fall in the rankings like Pitt and Hofstra who suddenly find even worse job prospects. Where was dean Carter last year before he had to save his own behind? What a hypocrit!
Nailed it. This could all be stopped next year. Just don't play the game. Problem is, those at the top will always play because they always win. The same schools every year take the gold. They have no incentive to stop playing. And every other school has a choice then of not playing (and being ranked shamefully at the bottom and made to look like they have something to hide) or spending cash on the slim chance that they might move up the rankings.
DeleteBut every school is spending cash to move up in the rankings, so it's something that will never happen.
US business models gone nuts again, where the only motive is making boatloads of cash by gaming the system.
I disagree that the US News rankings can be stopped if the schools do not participate. US News just ranks with whatever they have available. If they get only 5% of their surveys back, they will rank based on that. What does it matter to them if there is any validity? US News is just trying to sell their magazine. US News will get data from the ABA or some other source. Does Above The Law require schools to participate for them to rank the schools? Does the National Jurist?
DeleteI agree. Carter is just probably trying to save his job. Where was he on US News before Pitt fell in the rankings? Did the Hofstra dean get fired because Hofstra fell so much?
ReplyDeleteWhat surprised me when I looked at the ratings was a lot of schools are given the same ranking, as a tie sort of. So there might be three schools ranked 20th, for example. Why does USNR do this? Not that these rankings mean much, but this does make it hard to assess a school's true ranking.
ReplyDeleteHuh? As if they were placed 20, 21 and 22, it would all be so clear and meaningful?
DeleteIt wouldn't be clear and meaningful, but it would be much less pussy of them.
DeleteIf you're going to have a ranking where you split hairs over 1 and 2 every year, might as well rank everyone. If not, then just publish general tiers (which is how people view LS anyway).
I had a close look at the rankings. Its not quite as bad as I thought. They do have a lot of tied rankings, but for example they have two at #19, but they skip over 20 and go straight to 21. So you know that #21 is actually at the 21st place.
DeleteBut with all the variables they use, I don't know why they have to have any tied placings at all, let alone the huge numbers they do have.
"Where was dean Carter last year before he had to save his own behind? What a hypocrite!"
ReplyDeleteHe wasn't at Pitt (or the dean) before this year, so it would have been difficult for him to have a position on Pitt's rankings before this year....