Comparator

The Comparator component provides the possibility to compare the performances of different models on the same data set.
All the comparison analyses provided are based on the Analyzer analyses.

ComparatorClassification

The ComparatorClassification class can be used to perform comparison of classification models.

Parameters

dataset_gt_param

str

Path of the ground truth .json file.

task_type

TaskType

Problem task type. It can be:
TaskType.CLASSIFICATION_BINARY, TaskType.CLASSIFICATION_SINGLE_LABEL, TaskType.CLASSIFICATION_MULTI_LABEL.

multiple_proposals_path

list of list

List of the models proposals path. For each model, it must be specified the model name and the model proposals path directory; optionally can be specified also the default confidence threshold.
Example:
multiple_proposals_path = [['model_a', 'model_a_path', 0.65], ['model_b', 'model_b_path']]

result_saving_path

str, optional

Path used to save results.
(default is './results/')

properties_file

str, optional

The name of the file used to store the names of and values of the properties and the names of the categories.
(default is 'properties.json')

use_normalization

bool, optional

Indicates whether normalisation should be used.
(default is False)

norm_factor_categories

float, optional

Normalisation factor for the categories. If not specified, the default value is 1/number_of_categories.
(default is None)

norm_factors_properties

list of pair, optional

Normalization factor for the properties. Each pair (property_name, value) specifies the normalisation factor to be applied to a specific property. If not specified, for each property the default value is 1/number_of_property_values.
(default is None)

conf_thresh

float, optional

Confidence threshold. All the predictions with a confidence value less than the threshold are ignored.
If not specified, for single-label classification problems, the default value is 0.
If not specified, for binary and multi-label classification problems the default value is 0.5.
(default is None)

metric

Metrics, optional

The evaluation metric that will be used as default.
(default is Metrics.F1_SCORE)

similar_classes

list of list, optional

List of groups of ids of categories which are similar to each other.
(default is None)

load_properties

bool, optional

Indicates whether the properties should be loaded.
(default is True)

match_on_filename

bool, optional

Indicates whether the predictions refer to the ground truth by file_name (set to True) or by id (set to False).
(default is False)

save_graphs_as_png

bool, optional

Indicates whether plots should be saved as .png images.
(default is True)

Example

from odin.classes import ComparatorClassification

dataset_gt_param = "path/to/gt/file.json"

classification_type = TaskType.CLASSIFICATION_MULTI_LABEL

# Models for the analyses ('model_name', 'predictions_path', [confidence_threshold])
models_proposals = [('model_A', "path/to/model_A/predictions", 0.35),
                    ('model_B', "path/to/model_B/predictions", 0.65),
                    ('model_C', "path/to/model_C/predictions")]

my_comparator = ComparatorClassification(dataset_gt_param, classification_type, models_proposals)

ComparatorLocalization

The ComparatorLocalization class can be used to perform comparison of localization models, such as object detection and instance segmentation.

Parameters

dataset_gt_param

str

Path of the ground truth .json file.

task_type

TaskType

Problem task type. It can be:
TaskType.OBJECT_DETECTION, TaskType.INSTANCE_SEGMENTATION.

multiple_proposals_path

list of list

List of the models proposals path. For each model, it must be specified the model name and the model proposals path directory; optionally can be specified also the default confidence threshold.
Example:
multiple_proposals_path = [['model_a', 'model_a_path', 0.65], ['model_b', 'model_b_path']]

result_saving_path

str, optional

Path used to save results.
(default is './results/')

properties_file

str, optional

The name of the file used to store the names of and values of the properties and the names of the categories.
(default is 'properties.json')

use_normalization

bool, optional

Indicates whether normalisation should be used.
(default is False)

norm_factor_categories

float, optional

Normalisation factor for the categories. If not specified, the default value is 1/number_of_categories.
(default is None)

norm_factors_properties

list of pair, optional

Normalization factor for the properties. Each pair (property_name, value) specifies the normalisation factor to be applied to a specific property. If not specified, for each property the default value is 1/number_of_property_values.
(default is None)

iou

float, optional

Intersection Over Union threshold. All the predictions with a iou value less than the threshold are considered False Positives. If not specified, the default value is 0.5.
(default is None)

iou_weak

float, optional

Intersection Over Union weak threshold. Used for the identification of the localization errors. If not specified, the default value is 0.1.
(default is None)

conf_thresh

float, optional

Confidence threshold. All the predictions with a confidence value less than the threshold are ignored.
If not specified the default value is 0.5.
(default is None)

metric

Metrics, optional

The evaluation metric that will be used as default.
(default is Metrics.AVERAGE_PRECISION_SCORE)

similar_classes

list of list, optional

List of groups of ids of categories which are similar to each other.
(default is None)

load_properties

bool, optional

Indicates whether the properties should be loaded.
(default is True)

match_on_filename

bool, optional

Indicates whether the predictions refer to the ground truth by file_name (set to True) or by id (set to False).
(default is False)

save_graphs_as_png

bool, optional

Indicates whether plots should be saved as .png images.
(default is True)

Example

from odin.classes import ComparatorLocalization

dataset_gt_param = "path/to/gt/file.json"

task_type = TaskType.INSTANCE_SEGMENTATION

# Models for the analyses ('model_name', 'predictions_path', [confidence_threshold])
models_proposals = [('model_A', "path/to/model_A/predictions", 0.35),
                    ('model_B', "path/to/model_B/predictions", 0.65),
                    ('model_C', "path/to/model_C/predictions")]

my_comparator = ComparatorLocalization(dataset_gt_param, task_type, models_proposals)

ComparatorCAMs

The ComparatorCAMs class can be used to compare Class Activation Maps generated by different classification models.

Parameters

dataset_gt_param

str

Path of the ground truth .json file.

task_type

TaskType

Problem task type. It can be:
TaskType.CLASSIFICATION_BINARY, TaskType.CLASSIFICATION_SINGLE_LABEL, TaskType.CLASSIFICATION_MULTI_LABEL.

multiple_proposals_path

list of list

List of the models proposals path. For each model, it must be specified the model name and the model cams path directory; optionally can be specified also the predictions_path and the default confidence threshold.
Example:
multiple_proposals_path = [['model_a', ['model_a_cam_path', 'model_a_path'], 0.65], ['model_b', ['model_b_cam_path', 'model_b_path'], 0.45]]

cam_thresh

float, optional

CAM confidence threshold. All the pixels with a confidence value less than the threshold are ignored.
(default is 0.5)

cam_coverage_thresh

float, optional

CAM coverage threshold. All the CAMs with a iou less than the threshold do not cover the annotation.
(default is 0.5)

annotation_type

AnnotationType, optional

Indicates whether the annotation is a bounding box (AnnotationType.BBOX) or a segmentation mask (AnnotationType.SEGMENTATION)
(default is AnnotationType.BBOX)

result_saving_path

str, optional

Path used to save results.
(default is './results/')

properties_file

str, optional

The name of the file used to store the names of and values of the properties and the names of the categories.
(default is 'properties.json')

use_normalization

bool, optional

Indicates whether normalisation should be used.
(default is False)

norm_factor_categories

float, optional

Normalisation factor for the categories. If not specified, the default value is 1/number_of_categories.
(default is None)

norm_factors_properties

list of pair, optional

Normalization factor for the properties. Each pair (property_name, value) specifies the normalisation factor to be applied to a specific property. If not specified, for each property the default value is 1/number_of_property_values.
(default is None)

conf_thresh

float, optional

Confidence threshold. All the predictions with a confidence value less than the threshold are ignored.
If not specified, for single-label classification problems, the default value is 0.
If not specified, for binary and multi-label classification problems the default value is 0.5.
(default is None)

metric

Metrics, optional

The evaluation metric that will be used as default.
(default is Metrics.CAM_GLOBAL_IOU)

similar_classes

list of list, optional

List of groups of ids of categories which are similar to each other.
(default is None)

load_properties

bool, optional

Indicates whether the properties should be loaded.
(default is True)

match_on_filename

bool, optional

Indicates whether the predictions refer to the ground truth by file_name (set to True) or by id (set to False).
(default is False)

save_graphs_as_png

bool, optional

Indicates whether plots should be saved as .png images.
(default is True)

Example

from odin.classes import ComparatorCAMs

dataset_gt_param = "path/to/gt/file.json"

classification_type = TaskType.CLASSIFICATION_MULTI_LABEL

# Models for the analyses ('model_name', ['cams_path', ['predictions_path']], [confidence_threshold])
models_proposals = [('model_A', ["path/to/model_A/predictions"]),
                    ('model_B', ["path/to/model_A/predictions"])]


my_comparator = ComparatorCAMs(dataset_gt_param, classification_type, models_proposals)