On the ICJ, prediction markets, and the limits of expertise

The experts are wrong about Gaza

The Blind Leading the Blind (1568) by Pieter Bruegel the Elder

The nature, origin, and limits of knowledge are notoriously difficult to assess. Some argue that truth is best approached through reason alone; others that only experience can tell us what is real; still others that knowledge is less discovered than constructed, by communities, institutions, and the slow accretion of consensus.

In the last few decades yet another tradition has emerged, one that advocates the use of markets to help resolve our difficult questions.

Delphi Markets has very kindly offered me their rostrum to speak to what happens when two such traditions disagree—in our case, about whether Israel is committing genocide in Gaza.

---

If you hear the term “genocide”, what comes to mind?

Well—if you’re a normal, worldly person these days—the answer is likely “Gaza.” Peer into the global consciousness and see—on much of the modern internet, “genocide” is chiefly used in the context of Israel.

A representative example of such discourse is this short-form video from Mehdi Hasan. You can watch it or read the transcript below, but I'll save you the trouble of either: like 75% is Hasan listing the names of genocide experts who support the charge, never with more than a sentence of introduction. Quoth:

Genocide experts are almost all in agreement that what’s happening is a genocide, per the legal definition under the Genocide Convention, acts committed with the intent to destroy in whole or in part a national, ethnic, racial or religious group.

So here are the names, the expert names you need to know, including Jewish and Israeli experts, on the subject of genocide to avoid being gaslit by apologists for Israel. First, there are the academic experts who say Gaza is a genocide.

Martin Shaw, who literally wrote the book, What is Genocide?. Melanie O’Brien, who is the literal president of the International Association of Genocide Scholars. Dirk Moses, who is the senior editor of the Journal of Genocide Research. Iva Vukusic, who helped prosecute the Srebrenica genocide.

William Schabas, who even defended Myanmar against charges of genocide, against the Rohingya, but says what Israel is doing in Gaza now is a genocide. There are also the Israeli genocide experts, the Israeli historians of the Holocaust, who say Gaza is a genocide. Professors Omer Bartov, Amos Goldberg, Daniel Blatman, Raz Segal, Lee Mordechai, Dr. Shmuel Lederman, who was originally opposed to calling Gaza a genocide, but now says, quote, what Israel is doing in Gaza is genocidal in every sense.

Then there are the human rights groups who say Gaza is a genocide. Amnesty International, Human Rights Watch, the legal clinics at top U.S. universities who say it’s a genocide, at Boston University School of Law, at Cornell Law School, at Yale Law School, the United Nations organizations and experts who say it’s a genocide, the U.N. Independent International Commission of Inquiry, the U.N. Special Committee to Investigate Israeli Practices, multiple U.N. special rapporteurs. In fact, speaking the other week to the Dutch newspaper NRC, the Israeli expert Raz Segal said, can I name someone whose work I respect who doesn’t consider it genocide? His answer, no.

The Israeli expert Shmuel Lederman says it’s now the consensus view among genocide researchers. So apologists for Israel and lazy and cowardly both sides media outlets might want to reject the charge of genocide in Gaza. They might want to pretend it’s hyperbole or even anti-Semitic.

But the experts, the actual experts on genocide, they know a genocide when they see one, and they see one in Gaza.

Now, I’m afraid I can’t say I’ve read Mr. Hasan’s book How to Win Every Argument. But from this video, I’d hazard a guess that he places significant emphasis on the appeal to authority—Hasan in this video finds it sufficient to make no other argument for his case beyond beyond a spread of nearly contextless names and institutions endorsing his conclusion.

But though the impression of this video is a bit exaggerated (there are some scholars who take other views, such as John Spencer), it’s certainly not too far off. It is indeed the relative consensus of professional “genocide experts”, in academia and NGOs, that the Israeli government intends to “destroy in whole or in part a national, ethnic, racial or religious group”—a la the Geneva Convention definition of genocide.

No less than ever-present Wikipedia—bound by a unique quasi-legal process to the consensus of “reliable sources”—currently reflects this view.

So the evidence here seems pretty strong; nigh unassailable. Could you, dear author, really make the argument that 12 experts, Yale Law, a U.N. Special Committee, even Wikipedia itself, is wrong! Clearly, you must be a Zionist¹!

Well, dear reader, I could write an essay noting that Gaza's civilian casualty ratio—disputed, but likely not far from the 75% civilian implied by the Gazan Ministry of Health’s numbers—is within the normal range of modern urban warfare. After all—I could imagine writing—the US-backed campaign to retake Mosul and Raqqa from ISIS produced ratios of 2.5:1 or worse, and charges of genocide weren’t even contemplated. And though Israel has probably committed war crimes in several instances over the course of the conflict, basically every military action by Hamas—every single one!—violates the Geneva Convention: i.e. Hamas soldiers don’t wear uniforms, and basically only operate in civilian infrastructure which are both extremely serious war crimes.²³

But I’m not going to do that. Instead, I will analyze a rather different sort of evidence.

The Striking Asymmetry Between Experts and Markets

Now, while the assessments of experts and institutions on questions genocidal are important for deciding, for instance, what it says on Wikipedia, they have little legal import. But there is an institution that does: The International Court of Justice, or ICJ, in The Hague.

The ICJ is the judicial organ of the United Nations, and the body with jurisdiction over the Genocide Convention itself. This is the institution internationally recognized to decide what is genocide, and what isn’t.

In December 2023, South Africa invoked that jurisdiction, filing a case against Israel under the Convention over its conduct in Gaza: The internationally recognized process by which genocide is officially determined is underway.

So what does the world think the ICJ will find? Mehdi Hasan—presumably—thinks the vaunted court will find Israel guilty, and if you were only watching his show, you’d get the impression that it’s a foregone conclusion.

But what if there were a different kind of knowledge production? Not the consensus of vibes, papers, essays—but a process that aggregates everyone with an opinion, and asks them not merely to assert it, but to stake something on it?

Well, you’re in luck! Previously banned by the CFTC, over the last five years such a process has become tenuously legal in the US: That process is the prediction market. Under this mechanism, one buys a share in an outcome at some price between zero and a dollar. If the outcome occurs, it pays out a dollar. If it doesn’t, it pays nothing.

I think it’s easy to underestimate the power and promise of this mechanism. The price at any given moment reflects the aggregate judgment of everyone willing to bet—and because being wrong is costly, the incentive is to be correct rather than merely to sound correct. A prediction market is thus—in Bryan Caplan’s terms—a tax on bullshit: It introduces a tangible cost for believing false things about the world.

So what do our prediction markets think about the genocide question? Here are two on the subject:

Surprised? Perhaps Hasan would be.

---

How should we think about this?

One anticipates the objection, “who cares what the ICJ thinks?” but this objection is harder to maintain than it might appear.

First of all: The ICJ is the only institution with the authority to rule that a state is committing genocide under the Genocide Convention.

What’s more, there is only one legal definition of genocide, and it’s the one Mehdi Hasan quotes in his video—“acts committed with the intent to destroy in whole or in part a national, ethnic, racial or religious group.” The ICJ uses that definition too; there isn’t an actual asymmetry. At the very least, the markets demonstrate the conventional definition of genocide, assessed by a rigorous process—ie, a legal one—pretty clearly does not fit.

More importantly, the UN’s court is obviously not biased in Israel’s favor. The UN General Assembly has adopted 173 resolutions against Israel since 2015, versus 80 against all other countries combined! The UN Human Rights Council has condemned Israel 112 times since 2006 — more than Syria, Iran, Russia, and North Korea combined. That the UN’s court is still very unlikely to find genocide should be understood as some sort of admission against interest.

Now—another objection. Manifold is a play-money prediction market; there are no real money markets on whether Israel will be found guilty of genocide by the ICJ (America is not yet ready, it seems, for markets on genocide).

It’s natural to be a bit skeptical of play-money markets, but the literature on the question is unanimous—empirically, it’s crystal clear that play-money markets like Manifold’s are as efficient as real money ones, which—as markets have a way of being—are very efficient indeed. One in five Manifold markets priced at 20% resolve yes (i.e., the prices are correct).

---

So this is the situation we are left with: experts on one side and markets, coupled to a legal process, on the other.

“Well, sure,” you may say, “you have ‘markets’ which you claim are ‘efficient’, and some acronym ICJ, but the other side has all those experts with names—they probably know something I don’t!”

How much you should trust experts relative to markets and courts sounds like a thorny epistemological question—and perhaps in some sense it is. Perhaps someone could write an essay extolling the many virtues of prediction markets: Professional pundits may have some incentive to get things right, but this incentive is dominated by their desire for experiences like appearing on TV, fitting in with their colleagues, and ideological self-expression. It’s only with markets—I could imagine someone writing—where there is a tangible financial incentive to get things right, and legal processes crafted by millennia of cultural evolution to acceptably resolve contentious disputes, that we have a real shot at making sense of a complex world.

But the ambitions of the intellectual tradition behind prediction markets are to transcend that. No, I do not intend to argue that markets are better at ascertaining the truth than experts. I intend to show you that this is the case.

Tetlock’s Razor

The anti-expert position, ironically, has an expert of its own: the eminent professor at the University of Pennsylvania, Philip Tetlock.

A much higher h-index than Martin Shaw, “who literally wrote the book, What is Genocide?”.

Tetlock, along with Robin Hanson, founded the field of forecasting. And he is most famous for his longitudinal surveys on the accuracy of expert judgment.

Between 1984 and 2003, Tetlock recruited 284 experts—academics and pundits—and asked them to make roughly 28,000 predictions about geopolitical and economic outcomes. Questions were drawn from the expert’s own areas of competence, so a Soviet specialist would be asked about Soviet affairs, and a trade economist about trade.

The results, published in Expert Political Judgment in 2005 after nearly two decades of questionnairing, were unambiguous. The average expert didn’t outperform random guessing, and performed significantly worse than simple statistical heuristics such as “extrapolate the current trend” or “assume no change.” They failed even to measurably outperform educated laymen.

---

This was, and remains, a very powerful critique. Quoth Richard Hanania’s magisterial 2021 essay “Tetlock and the Taliban”:

Phil Tetlock’s work on experts is one of those things that gets a lot of attention, but still manages to be underrated. In his 2005 Expert Political Judgment: How Good Is It? How Can We Know?, he found that the forecasting abilities of subject-matter experts were no better than educated laymen when it came to predicting geopolitical events and economic outcomes. As Bryan Caplan points out, we shouldn’t exaggerate the results here and provide too much fodder for populists; the questions asked were chosen for their difficulty, and the experts were being compared to laymen who nonetheless had met some threshold of education and competence.

At the same time, we shouldn’t put too little emphasis on the results either. They show that “expertise” as we understand it is largely fake. Should you listen to epidemiologists or economists when it comes to COVID-19? Conventional wisdom says “trust the experts.” The lesson of Tetlock (and the Afghanistan War), is that while you certainly shouldn’t be getting all your information from your uncle’s Facebook Wall, there is no reason to start with a strong prior that people with medical degrees know more than any intelligent person who honestly looks at the available data.

On contentious geopolitical questions, the weight you should give to the opinions of experts is—measurably⁴—around 0. The forecasters on Manifold are aware of Amnesty International, Yale Law School, and Martin Shaw. They have priced all of that in, and they have still arrived at 14%.

The world is full of such incongruities if you know where to look. The incentives of pundits are systematically bad: Lacking antibodies against ideological capture, they unrestrainedly proclaim whatever positions are socially advantageous. As Steven Pinker notes in Language, Cognition, and Human Nature: “People are embraced or condemned according to their beliefs, so one function of the mind may be to hold beliefs that bring the belief-holder the greatest number of allies, protectors, or disciples, rather than beliefs that are most likely to be true.” When rationalists say politics is the mind-killer, it’s because in this area of life the social rewards universally dominate the pragmatic rewards of more accurate world modeling; thus pontificators about politics have almost no incentive to get at the truth.

If you want to actually understand the world, don’t just read the New York Times—and certainly don’t just watch reels. The best exercise for your epistemic habits is to play the prediction markets: They give you structured practice, grade you, and incentivize you to understand why you were wrong.

I’ll leave you with one final such market. If you disagree with its assessment of the issue, don’t just let us know in the comments—go change the price yourself!

1. I’m not!

2. The concept of “rocket attacks”, which we’ve become desensitized to is also worth considering: These rockets are always aimed at civilians, which is a very serious war crime.

3. Q. Why does Hamas not wear uniforms in Gaza, and embed itself super deeply in civilian areas (such as bringing hostages to Gaza’s largest hospital), whereas ISIS fought in the open around Mosul and Raqqa?

4. And it has been measured quite a bit subsequently, with similar results.