The next phase of FrankenSME: The latest developments in combining expert judgements

Last year Hubbard Decision Research (HDR) presented the potential value of new methods for aggregating expert estimates.

Now the HDR tool has been completed.

Decades of experiments by researchers have shown that subjective assessment of probabilities is both a teachable skill and that combining the estimates of multiple subject matter experts (SMEs) can further improve the accuracy of estimates. In the last 22 years, over 2000 individuals have gone through calibrated probability assessment training by Hubbard Decision Research and HDR has used data from hundreds of individuals and tens of thousands of estimates to create an improved method for aggregating multiple experts. Data analytics and Bayesian methods are used to determine how to combine multiple experts in a way that not only produces well-calibrated estimates (i.e., 90% of actuals fall within the estimated 90% confidence intervals) but also produces estimates with a higher information value (e.g., narrower ranges) than the best individual SME on average.

The HDR methods also outperform simple averages and linear weighting methods developed by earlier researchers. Powerful expert aggregation methods should be the cornerstone of all expert estimation methods in risk analysis.