The world’s first prediction market for private, unlisted companies.

NOTSTOCKS
Sign in
Back to blog

For Researchers: Report Brier REL, RES, UNC and BSS, Avoid Bin Bias

Decorative Brier score research title card

The Brier score is the mean squared error between predicted probabilities and binary outcomes: lower is better, with 0 as the perfect score. A constant 0.5 forecaster scores 0.25, which makes a useful mental benchmark. As a strictly proper scoring rule, it rewards forecasts that are both well calibrated and sharply discriminating, which is why it shows up everywhere from weather forecasting to clinical risk models.

TL;DR:

  • The Brier score’s effectiveness depends heavily on assessing its decomposition into reliability, resolution, and uncertainty to identify calibration issues or discrimination problems.
  • Comparing models requires calculating the Brier skill score against a known baseline, especially when datasets differ in event prevalence, to avoid misleading conclusions.
  • Using bias-corrected estimators for the Brier score components improves the reliability of model comparisons, particularly in finite samples or when binning forecasts.
  • The Brier score alone is insufficient; combining it with calibration plots and other metrics like net benefit or expected calibration error yields a fuller evaluation.
  • In multiclass problems, recognize that the maximum Brier score can reach around 2, and always specify which baseline and binning methods are used when reporting results.
Notstocks
Explore Private Market Signals
NotStocks brings prediction markets for private companies to investors seeking insights beyond traditional public stock markets.

Table of Contents

Binary and multiclass formulas explained

In the binary case, the Brier score is defined as BS equals one over N times the sum over t of (f_t minus o_t) squared, where N is the number of forecast instances, f_t is the predicted probability for instance t, and o_t is the actual outcome coded as 0 or 1. This is literally mean squared error applied to probability forecasts, and the common binary formulation is bounded between 0 and 1, as verified by established sources.

The original multi-category version, as proposed by Brier, sums squared differences across all outcome classes rather than just the positive class. Because that formulation sums over every category instead of a single binary comparison, its scale differs and can run up to 2 rather than 1. Anyone comparing scores across papers should check which formulation is in use before assuming the numbers are on the same scale. Despite the computation resembling ordinary squared error, the Brier score’s status as a proper scoring rule means a forecaster cannot improve their expected score by reporting anything other than their true believed probability.

Binary and multiclass formulas explained — overview diagram

A worked example comparing two forecasters

Say two models forecast the probability of a binary event (such as a startup reaching a funding milestone) across six instances with these outcomes: 1, 0, 1, 1, 0, 1.

  1. Model A forecasts 0.9, 0.2, 0.8, 0.6, 0.3, 0.7. Squared errors: 0.01, 0.04, 0.04, 0.16, 0.09, 0.09.
  2. Model A’s Brier score: sum equals 0.43, divided by 6 equals 0.072.
  3. Model B forecasts 0.6, 0.5, 0.6, 0.5, 0.5, 0.6. Squared errors: 0.16, 0.25, 0.16, 0.25, 0.25, 0.16.
  4. Model B’s Brier score: sum equals 1.23, divided by 6 equals 0.205.
  5. A constant 0.5 forecaster on this same sample would score close to the general 0.25 baseline, confirming Model B is barely better than guessing while Model A is clearly sharper.

The gap between 0.072 and 0.205 is the kind of difference that should show up in a reliability diagram, not just a single summary number.

Murphy decomposition: reliability, resolution, uncertainty

Murphy’s decomposition splits the Brier score into three components that explain why a score is high or low rather than just reporting that it is. Reliability measures how well forecast probabilities track observed frequencies, resolution measures how much a forecaster’s predictions vary with actual outcomes, and uncertainty is the variance of the outcome itself, equal to the Brier score of a constant climatological forecast. The identity is BS equals REL minus RES plus UNC.

In practice, reliability and resolution are estimated by grouping forecasts into probability bins and comparing the average forecast in each bin to the observed outcome rate. That binning step is not free: two additional within-bin terms, often written WBV and WBC, appear once you bin continuous forecasts, and a generalized resolution measure (RES plus WBV plus WBC) reduces sensitivity to the chosen bin width.

  • When reliability dominates the decomposition, the model is miscalibrated and needs recalibration rather than retraining.
  • When resolution is low, the forecasts are not discriminating between outcomes, which points to a feature or model class problem.
  • Uncertainty is fixed by the sample’s base rate and cannot be improved by any model.

One sourced estimator refinement: bias-corrected decomposition estimators exist specifically because naive binned REL and RES are biased in finite samples, and reporting them alongside raw decomposition values strengthens published comparisons.

Brier skill score for comparing across base rates

The Brier skill score normalizes a model’s Brier score against a reference forecaster, usually the sample’s climatological base rate: BSS equals 1 minus BS_model divided by BS_reference. A BSS of 1 means a perfect forecaster, 0 means no better than the reference, and a negative value means the model is worse than simply guessing the base rate every time.

This matters because raw Brier scores are not directly comparable across datasets with different prevalence. A scaled version of this idea, 1 minus Brier divided by Brier_max, has been recommended specifically for comparing model performance across datasets with different prevalence, since it preserves comparability across base rates. Use BSS or a scaled Brier whenever you are comparing models trained or tested on populations with different event rates.

Where the Brier score misleads and what to use instead

The Brier score’s biggest practical flaw is prevalence dependence: its value shifts with the base rate of the outcome, independent of how good the forecasts actually are. In clinical settings this can produce an inappropriate rank ordering of tests or models, favoring a high-specificity test over a high-sensitivity one even when the high-sensitivity test is clearly preferable in a low-prevalence but high-consequence scenario. The same critique argues that decision-analytic measures like net benefit better capture clinical utility than an aggregate squared-error score ever can.

Complementary metrics fill specific gaps the Brier score leaves open.

  • Log loss is also a proper scoring rule but penalizes confident, wrong predictions far more heavily than the Brier score does.
  • AUC measures discrimination only and says nothing about whether probabilities are calibrated.
  • Expected calibration error and reliability plots visualize calibration directly, which a single Brier number hides.
  • Net benefit and other decision-analytic measures translate probability accuracy into actual clinical or business consequences.

Multiclass scoring and implementation notes

For more than two classes, the Brier score generalizes to a sum of squared differences across every class for each instance, then averaged over instances. That multi-category sum changes the scale compared to the binary case, commonly reaching a maximum around 2 rather than 1, so a multiclass score should never be read on the binary 0 to 1 scale.

For implementation, scikit-learn’s brier_score_loss function is the standard reference for the binary case, accepting predicted probabilities, true labels, and optional sample weights. For multiclass problems, practitioners typically compute a one-vs-rest sum of squared errors manually, since there is no single universally adopted multiclass function name across libraries. For reproducibility, vectorize the squared-error computation, fix any random seeds used in train or test splits, and always report which baseline (constant prevalence or otherwise) was used for any skill score.

A reporting checklist for papers and model documentation

A Brier score reported alone, without context, tells a reader little about whether a model is actually useful; tools like Betlog can help track and evaluate forecasts over time to improve predictive performance.

  1. Report the raw Brier score, the Brier skill score with its baseline clearly defined, and the REL, RES, and UNC decomposition with the binning method stated.
  2. Include a reliability diagram and a sharpness histogram so readers can see calibration and discrimination separately from the summary number.
  3. If reliability dominates the decomposition, apply a recalibration method such as Platt scaling or isotonic regression, then recompute BS and its components to confirm improvement.
  4. Always state the sample prevalence, since any Brier-based comparison across datasets is only meaningful once base rates are accounted for.

Pro Tip: Report confidence intervals or variance estimates for your Brier score and its decomposition whenever you compare two models, since a small aggregate difference can fall well inside the noise.

Applying Brier decomposition to prediction market prices

NotStocks lists market-implied probabilities as mid-prices for whether a private company hits a future milestone. Treating a market-implied mid-price as the forecast probability f_t and the eventual resolved outcome as o_t lets you compute a Brier score exactly as in the worked example above, then decompose it into reliability and resolution. A market with low reliability is systematically over- or underpricing certain outcome ranges, while low resolution means prices cluster too close to the base rate to discriminate winners from losers. Readers can reproduce this calculation using NotStocks’ own market and pricing data as a forecasting dataset.

What the decomposition debate gets wrong

Most discussions of the Brier score treat it as a single number to minimize, when the more useful habit is to never report it without its decomposition. A model with an excellent aggregate Brier score can still have a reliability problem hiding inside it, masked by strong resolution, and that distinction changes what you do next: recalibrate versus rebuild.

What the decomposition debate gets wrong — overview diagram

The conventional advice to just “use a proper scoring rule” undersells how much binning choices and finite-sample bias can distort the REL and RES terms you report. If you are publishing a comparison between two models, the bias-corrected estimators are not optional extras. They are the difference between a defensible claim and a coin flip dressed up as a finding.

Prioritize the decomposition and the skill score over the raw number, every time. A raw Brier score tells you almost nothing in isolation, since it conflates calibration, discrimination, and the base rate into one figure that two very different models can share by coincidence.

— Max

Sources

FAQ

What does the Brier score tell you?

The Brier score tells you how close a set of predicted probabilities came to the actual binary outcomes, combining calibration and discrimination into one number. A lower score means the forecasts were closer to reality, with 0 being a perfect score and 0.25 the benchmark for a constant 0.5 forecaster.

What is an acceptable Brier score?

There is no single universal threshold, since an acceptable score depends heavily on the outcome’s base rate and the comparison baseline used. A model is generally doing well if its score sits meaningfully below the constant 0.5 benchmark of 0.25 and its Brier skill score against sample prevalence is positive.

What is the Brier score used for?

The Brier score is used to evaluate how well a set of predicted probabilities matches actual binary outcomes, common in weather forecasting, machine learning calibration checks, and risk modeling. It is particularly useful when you need a single proper scoring rule that penalizes both poor calibration and poor discrimination.

What counts as a good Brier score?

A good Brier score is one that is both low in absolute terms and positive when converted into a Brier skill score against a relevant baseline, since the raw number alone can mislead across different base rates. A scaled Brier score of 1 indicates perfect agreement, 0 indicates no better than random, and negative values indicate worse than random.

How does the Brier score compare to log loss?

Both are proper scoring rules, but log loss penalizes confident and wrong predictions far more severely than the Brier score’s squared-error penalty does. The Brier score is often preferred when extreme miscalibration should not dominate the metric, while log loss is preferred when you specifically want to punish overconfidence.