e diagrams for the averaged ranking results of the metric Log-loss.](figures/results/crit_diff_loss_v2 "fig:"){width="\linewidth"}

The same process is applied to each of the $11$ classifiers for every metric. Table \[table:loss\] shows the final average results of all classifiers. Notice that the row corresponding to naive Bayes has the rounded average rankings from Figure \[fig:cd:nbayes:loss\].

Results {#sec:res}
=======

In this Section we present all the final results, including ranking tables for every metric, critical difference diagrams, the best hyperparameters selected for Dirichlet calibration with L2 regularisation, Frequency binning and Width binning; a comparison of how calibrated the $11$ classifiers are, and additional results on deep neural networks.

Final ranking tables for all metrics {#sec:res:rank}
------------------------------------

We present here all the final ranking tables for all metrics (Tables \[table:acc\], \[table:loss\], \[table:brier\], \[table:mce\], \[table:conf-ece\], \[table:cw-ece\], \[table:p-conf-ece\], and \[table:p-cw-ece\]). For each ranking, a lower value is indicative of a better metric value (eg. a higher accuracy corresponds to a lower ranking, while a lower log-loss corresponds to a lower ranking as well). Additional details on how to interpret the tables can be found in Section \[sec:exp:example\].

Final critical difference diagrams for every metric
---------------------------------------------------

In order to perform a final comparison between calibration methods, we considered every combination of dataset and classifier as a group $n = \#datasets \times \#classifiers$, and ranked the results of the $k$ calibration methods. With this setting, we have performed the Friedman statistical test followed by the one-tailed Bonferroni-Dunn test to obtain critical differences (CDs) for every metric (See Figure \[fig:multi:cd:all\]). The results showed Dirichlet L2 as the best calibration method for the measures accuracy, log-loss and p-cw-ece with statistical signi