Analyzer
The Analyzer component provides multiple analyses and metrics in order to evaluate the performances of the model.
It is possible to evaluate the model on the whole data set or only on a subset, splitting the evaluation for each category or for each meta-annotation value.
AnalyzerClassification
The AnalyzerClassification class can be used to perform diagnostics for classification models.
Parameters
- classifier_name
str- Name of the classifier. It is used as folder to save results.
- dataset
DatasetClassification- Dataset used to perform the analysis.
- result_saving_path
str, optional- Path used to save results.
(default is './results/') - use_normalization
bool, optional- Indicates whether normalisation should be used.
(default is False) - norm_factor_categories
float, optional- Normalisation factor for the categories. If not specified, the default value is 1/number_of_categories.
(default is None) - norm_factors_properties
list of pair, optional- Normalization factor for the properties. Each pair (property_name, value) specifies the normalisation factor to be applied to a specific property. If not specified, for each property the default value is 1/number_of_property_values.
(default is None) - conf_thresh
float, optional- Confidence threshold. All the predictions with a confidence value less than the threshold are ignored.
If not specified, for single-label classification problems, the default value is 0.
If not specified, for binary and multi-label classification problems the default value is 0.5.
(default is None) - metric
Metrics, optional- The evaluation metric that will be used as default.
(default is Metrics.F1_SCORE) - save_graphs_as_png
bool, optional- Indicates whether plots should be saved as .png images.
(default is True)
Example
from odin.classes import AnalyzerClassification
# N.B. If different models proposals have been loaded to the dataset, the
# classifier name must be the same of one of the models names provided
# with the predictions when my_classification_dataset has been created
my_analyzer = AnalyzerClassification("my_classifier_name", my_classification_dataset)
AnalyzerLocalization
The AnalyzerLocalization class can be used to perform diagnostics for localization models, such as object detection and instance segmentation.
Parameters
- detector_name
str- Name of the detector. It is used as folder to save results.
- dataset
DatasetLocalization- Dataset used to perform the analysis.
- result_saving_path
str, optional- Path used to save results.
(default is './results/') - use_normalization
bool, optional- Indicates whether normalisation should be used.
(default is False) - norm_factor_categories
float, optional- Normalisation factor for the categories. If not specified, the default value is 1/number_of_categories.
(default is None) - norm_factors_properties
list of pair, optional- Normalization factor for the properties. Each pair (property_name, value) specifies the normalisation factor to be applied to a specific property. If not specified, for each property the default value is 1/number_of_property_values.
(default is None) - iou
float, optional- Intersection Over Union threshold. All the predictions with a iou value less than the threshold are considered False Positives. If not specified, the default value is 0.5.
(default is None) - iou_weak
float, optional- Intersection Over Union weak threshold. Used for the identification of the localization errors. If not specified, the default value is 0.1.
(default is None) - conf_thresh
float, optional- Confidence threshold. All the predictions with a confidence value less than the threshold are ignored.
If not specified, the default value is 0.5.
(default is None) - metric
Metrics, optional- The evaluation metric that will be used as default.
(default is Metrics.Metrics.AVERAGE_PRECISION_SCORE) - save_graphs_as_png
bool, optional- Indicates whether plots should be saved as .png images.
(default is True)
Example
from odin.classes import AnalyzerLocalization
# N.B. If different models proposals have been loaded to the dataset, the
# detector name must be the same of one of the models names provided
# with the predictions when my_localization_dataset has been created
my_analyzer = AnalyzerLocalization("my_detector_name", my_localization_dataset)
AnalyzerCAMs
The AnalyzerCAMs class can be used to diagnose Class Activation Maps generated by classification models.
Parameters
- classifier_name
str- Name of the classifier. It is used as folder to save results.
- dataset
DatasetCAMs- Dataset used to perform the analysis.
- cam_thresh
float, optional- CAM confidence threshold. All the pixels with a confidence value less than the threshold are ignored.
(default is 0.5) - cam_coverage_thresh
float, optional- CAM coverage threshold. All the CAMs with a iou less than the threshold do not cover the annotation.
(default is 0.5) - result_saving_path
str, optional- Path used to save results.
(default is './results/') - use_normalization
bool, optional- Indicates whether normalisation should be used.
(default is False) - norm_factor_categories
float, optional- Normalisation factor for the categories. If not specified, the default value is 1/number_of_categories.
(default is None) - norm_factors_properties
list of pair, optional- Normalization factor for the properties. Each pair (property_name, value) specifies the normalisation factor to be applied to a specific property. If not specified, for each property the default value is 1/number_of_property_values.
(default is None) - conf_thresh
float, optional- Confidence threshold. All the predictions with a confidence value less than the threshold are ignored.
If not specified, the default value is 0.5.
(default is None) - metric
Metrics, optional- The evaluation metric that will be used as default.
(default is Metrics.Metrics.CAM_GLOBAL_IOU) - save_graphs_as_png
bool, optional- Indicates whether plots should be saved as .png images.
(default is True)
Example
from odin.classes import AnalyzerCAMs
# N.B. If different models CAMs have been loaded to the dataset, the
# classifier name must be the same of one of the models names provided
# with the CAMs when my_cams_dataset has been created
my_analyzer = AnalyzerCAMs("my_classifier_name", my_cams_dataset)