Prediction Market Calibration: What the PNAS Study Shows—and Its Limits

Prediction market calibration asks whether events given a 70% chance happen about 70% of the time across many forecasts. A new study in PNAS reports broad agreement between probabilities and outcomes, with systematic biases in some domains.

What the PNAS prediction markets study examined

The paper evaluates calibration and accuracy across horizons ranging from hours to years. This explainer focuses on calibration.

The authors’ earlier manuscript identifies Kalshi, Polymarket, Manifold and Metaculus. Calling all four prediction markets would blur an important distinction: Metaculus aggregates submitted probabilities rather than trading outcome contracts.

Kalshi and Polymarket use financial contracts or outcome shares. Manifold uses mana, a virtual currency that cannot be cashed out. Metaculus combines forecasters’ probability estimates into a community forecast. Those systems give participants different ways to express beliefs; their inclusion does not establish which incentive structure causes better predictions.

That distinction matters as probability estimates reach audiences beyond traders, including through prediction-market odds in Google Finance. A displayed percentage needs an explanation of what it measures.

What prediction market calibration means

Take 100 hypothetical forecasts, each assigning an event a 70% probability. A picture showing 70 events occurring and 30 not occurring illustrates the long-run relationship calibration describes. The 70% figure here is an arbitrary teaching example, not a measured result extracted from the paper.

A real batch can depart from that neat split. Calibration is assessed across collections of forecasts, and random variation does not disappear merely because the number of forecasts reaches 100. The pattern across probabilities and outcomes is what matters.

This also changes how an upset should be read. A 70% forecast leaves room for the other outcome. Treating its occurrence as automatic proof that the forecast was bad confuses probability with a promise. Repeatedly calling events 70% likely when they occur much less often would be a different problem.

Calibration is one part of prediction market accuracy

Calibration alone does not tell the whole story about forecasting skill. Imagine assigning 50% to every event in a collection where half occur. That set can be calibrated while doing nothing to distinguish more-likely events from less-likely ones. This is an illustrative deduction, not a description of the PNAS sample.

A useful forecasting assessment therefore asks more than whether percentages line up with frequencies. It also asks whether forecasts separate outcomes effectively. Allan Murphy’s decomposition of the Brier score, a measure of probabilistic forecast error, distinguishes calibration, resolution and underlying event uncertainty. A Brier score is not a pure calibration score.

Readers should consequently be wary of translating a calibration finding into a claim that a platform always selects the winning side. Nor does an aggregate result establish that every category, every probability range or every individual contract performs equally well.

Why calibrated odds do not establish a profitable trade

A useful estimate of uncertainty is not automatically a bargain. Trading costs raise the break-even probability above the purchase price expressed as a share of the payout. That is why a forecast can be informative without offering the buyer a profitable opportunity.

Work by Bürgi, Deng and Whelan on a separate Kalshi dataset found informative prices alongside favorite–longshot bias. That bias describes a tendency to overestimate low-probability outcomes and underestimate high-probability ones. Their working paper also considers trading costs; those findings are context, not additional results from the PNAS study.

The practical distinction is between interpreting a probability and deciding whether to pay for exposure to it. Dyutam’s explanation of how Polymarket fees affect returns covers that second question. Its historical fee figures should not be treated as today’s schedule.

Where the evidence needs qualification

The PNAS abstract reports systematic over- or underestimation in some consequential domains. An overall calibration result should therefore not be read as uniform performance.

Question selection matters to interpreting any such comparison. Platforms can host different types of questions, and related contracts should not be casually counted as independent tests. Those are considerations for assessing the evidence, not established flaws in this paper.

Source note: This account uses the published abstract, significance statement and disclosures. The final detailed methods and appendix were not available for inspection, so the exact scoring-time protocol remains unverified here. The earlier manuscript does not substitute for final methods, and forecast horizons should not be confused with the times at which probabilities were scored.

The final disclosure records Manifold-related support for Justin Dickerson’s Project Themis data pipeline and unrelated Manifold consulting that began after initial submission. The authors state that no platform participated in the study’s design, analysis, interpretation, manuscript preparation or publication decision.

How earlier research helps interpret the result

Page and Clemen’s 2013 study found stronger calibration near expiry and favorite–longshot bias at longer horizons in its historical sample. Its relevance is practical: the time at which a probability is observed can matter. A statement about calibration without a clear time reference has limited value for someone examining a forecast today.

Atanasov and colleagues compared markets with prediction polls, in which people submit forecasts. Markets beat simple averages, but statistically aggregated team forecasts outperformed markets. These were forecasting polls, not ordinary voter-opinion surveys. The result offers no basis for a universal claim that markets beat every alternative.

These studies use different samples and benchmarks. They help explain why the question has several parts; they should not be pooled into a single accuracy percentage. Dyutam’s earlier coverage of macroeconomic prediction markets addresses a separate part of that research landscape.

How to read a displayed probability

Start with the exact question and its resolution conditions. Then check the horizon and remember that a displayed price belongs to that contract, rather than to every interpretation of the underlying event. Dyutam’s recession-odds explainer shows why contract wording matters.

Use the probability as information about uncertainty. Assess trading costs separately. Broad calibration gives readers a reason to take forecast probabilities seriously, while leaving the work of interpreting the particular question intact.

FAQs

Sources

• Calibration, accuracy, and domain variation across one million collective forecasts from prediction platforms — Dickerson et al., PNAS (2026); published abstract and disclosures
• The knowability of the future: Evidence from one million collective forecasts — Authors’ March 2026 preprint; platform identification only
• What are prediction markets? — Kalshi Help Center; contract mechanics
• What is Polymarket? — Polymarket Help Center; outcome-share mechanics
• Manifold FAQ — Manifold; virtual-currency mechanics
• Metaculus FAQ — Metaculus; community forecast aggregation
• A New Vector Partition of the Probability Score — Allan H. Murphy, Journal of Applied Meteorology (1973)
• Do Prediction Markets Produce Well-Calibrated Probability Forecasts? — Lionel Page and Robert T. Clemen, The Economic Journal (2013)
• Distilling the Wisdom of Crowds: Prediction Markets vs. Prediction Polls — Atanasov et al., Management Science
• Makers and Takers: The Economics of the Kalshi Prediction Market — Bürgi, Deng and Whelan, UCD (2025); Kalshi prices, bias and trading costs