Investigating Calibrated Classification Scores through the Lens of Interpretability

Simple item page
dc.contributor.advisorUrner, Ruth
dc.contributor.authorTorabian, Alireza
dc.date.accessioned2023-12-08T14:36:46Z
dc.date.available2023-12-08T14:36:46Z
dc.date.issued2023-12-08
dc.date.updated2023-12-08T14:36:46Z
dc.degree.disciplineComputer Science
dc.degree.levelMaster's
dc.degree.nameMSc - Master of Science
dc.description.abstractCalibration is a frequently invoked concept when useful label probability estimates are required on top of classification accuracy. A calibrated model is a scoring function whose scores correctly reflect underlying label probabilities. Calibration in itself however does not imply classification accuracy, nor human interpretable estimates, nor is it straightforward to verify calibration from finite data. There is a plethora of evaluation metrics (and loss functions) that each assesses a specific aspect of a calibration model. In this work, we initiate an axiomatic study of the notion of calibration and evaluation measures for calibration. We catalogue desirable properties of calibration models as well as evaluation metrics and analyze their feasibility and correspondences. We complement this analysis with an empirical evaluation, comparing two metrics and comparing common calibration methods to employing a simple, interpretable decision tree.
dc.identifier.urihttps://hdl.handle.net/10315/41698
dc.languageen
dc.rightsAuthor owns copyright, except where explicitly noted. Please contact the author directly with licensing requests.
dc.subjectComputer science
dc.subject.keywordsMachine learning
dc.subject.keywordsCalibration
dc.subject.keywordsClassification
dc.subject.keywordsReliability
dc.subject.keywordsInterpretability
dc.subject.keywordsTheoretical machine learning
dc.titleInvestigating Calibrated Classification Scores through the Lens of Interpretability
dc.typeElectronic Thesis or Dissertation

Files

Original bundle
Now showing 1 - 1 of 1
Thumbnail Image
Name:
Torabian_Alireza_2023_Masters.pdf
Size:
2.95 MB
Format:
Adobe Portable Document Format
Download
License bundle
Now showing 1 - 2 of 2
No Thumbnail Available
Name:
license.txt
Size:
1.87 KB
Format:
Plain Text
Description:
Download
No Thumbnail Available
Name:
YorkU_ETDlicense.txt
Size:
3.39 KB
Format:
Plain Text
Description:
Download

Collections

Computer Science

All items in the YorkSpace institutional repository are protected by copyright, with all rights reserved except where explicitly noted.