Over at the Monkey Cage, Erik Voeten analyzes which scholars mis-predicted Russia’s Crimean intervention. Most people got it wrong of course, and of various international relations subdisciplines, the high water mark was 20 percent getting it right. Of particular interest to 2C readers:
Self-identified Liberals and Constructivists did poorly, with Liberals both very unlikely to predict intervention and very likely to offer a definitive “no” rather than the “don’t know” answer that was very popular among Constructivists (who sometimes look dimly on the predictive ambitions of social science).
Perhaps a misplaced faith in the power of international law and institutions was at the root of this. After all, the Russian intervention violates a system of laws and norms that these paradigms hold dearly. Yet, non-realist scholars who study international law or international organizations as their primary or secondary field were more likely to foresee the military action (see graph).
Delving deeper into the data, I found that only 7 percent of the 150 self-identified Liberals and Constructivists who do not study international organizations and law foresaw the Russian military intervention. By contrast 15 percent of the 87 Liberals and Constructivists who study international law and organization got it right. This is admittedly speculative but it may be that paradigms impose blinders especially outside of ones field of study. Only 5 percent (4) of the 87 Liberals and Constructivists who do not study international security, Russia or international organizations and law correctly predicted a military intervention.
To follow up Erik’s speculation with more of my own, I would wager that legal scholars from non-social science backgrounds would probably fare even worse, if we had data on this, for similar reasons of misplaced faith in the normative powers of law. Those who are willing to treat legal processes empirically rather than normatively would probably score better.
These disciplinary distinctions are very much on my own mind right now, as I am writing up a paper on social science replication of empirical legal work. In my literature review, I came across some gems that really capture the difference between legal scholarship and social science.
Here are political scientists Lee Epstein and Gary King:
While a Ph.D. is taught to subject his or her favored hypothesis to every conceivable test and data source, seeking out all possible evidence against his or her theory, an attorney is taught to amass all the evidence for his or her hypothesis and distract attention from anything that might be seen as contradictory information. An attorney who treats a client like a hypothesis would be disbarred; a Ph.D. who advocates a hypothesis like a client would be ignored.
Later on, they quote Lawrence Friedman, who writes:
In legal scholarship, “theory” is king. But people who talk about legal “theory” have a strange idea of what “theory” means. In most fields, a theory has to be testable; it is a hypothesis, a prediction, and therefore subject to proof. When legal scholars use the word “theory,” they seem to mean (most of the time) something they consider deep, original, and completely untestable.
Hall and Wright, in a piece on content analysis of texts within legal scholarship, hold out a bit more hope for legal scholars’ added value, but with some words of caution:
We argue that the uses of content analysis that best combine the strengths of the legal scholar and the social scientist ask about the internal interaction of facts and arguments in an opinion. Content analysis also works best when the judicial opinions in a collection hold essentially equal value, such as where patterns across cases matter more than a deeply reflective understanding of a single pivotal case. While conventional legal scholarship analyzes issues presented in one case or a small group of exceptional or weighty cases, content analysis works by analyzing a larger group of similarly weighted cases to find overall patterns. Content analysis also assumes an equality among readers of judicial opinions. The traditional legal scholarly enterprise relies, like literary interpretation, on the interpreter’s authoritative expertise to select important cases and to draw out noteworthy themes and potential social effects of decisions. Content analysis requires the researcher to explain the selection of cases and themes in enough objective detail to allow others to replicate the steps. This method’s persuasiveness depends on the community’s ability to reproduce the findings rather than the author’s rhetorical power to proclaim them. Content analysis, however, does not displace traditional interpretive legal scholarship. Instead, it offers distinctive insights that complement the types of understanding that only traditional analysis can generate…
Legal scholars study cases not simply because they reflect or respond to the law, but mainly because they are the law. To know the law of a case classically requires subjective interpretation, sometimes of a deeply reflective nature. Legal readers ponder the meaning of a decision for future cases by asking how the outcome in the current case relates to its facts, procedural posture, and the court’s reasoning.98 It is doubtful that content analysis standing alone will be able to replace this core mode of interpretive case law analysis.
Todd N. Tucker, PhD 