Meters

class classy_vision.meters.AccuracyMeter(topk)

Meter to calculate top-k accuracy for single label/ multi label image classification task.

__init__(topk)
Parameters

topk – list of int k values.

classmethod from_config(config: Dict[str, Any]) → classy_vision.meters.accuracy_meter.AccuracyMeter

Instantiates a AccuracyMeter from a configuration.

Parameters

config – A configuration for a AccuracyMeter. See __init__() for parameters expected in the config.

Returns

A AccuracyMeter instance.

get_classy_state()

Contains the states of the meter.

property name

The name of the meter.

reset()

Resets any internal meter state.

Should normally be called at the end of a phase.

set_classy_state(state)

Set the state of the ClassyMeter.

Parameters

state_dict – The state dictionary. Must be the output of a call to get_classy_state().

This is used to load the state of the meter from a checkpoint.

sync_state()

Syncs state with all other meters in distributed training.

If not provided by child class this does nothing by default and meter only provides the local process stats. If implemented then the meter provides the global stats at last sync + any local updates since the last sync.

Warning

Calls to sync_state could involve communications via torch.distributed which can result in a loss of performance or deadlocks if not coordinated among threads.

update(model_output, target, **kwargs)
Parameters
  • model_output – tensor of shape (B, C) where each value is either logit or class probability.

  • target – tensor of shape (B, C), which is one-hot / multi-label encoded, or tensor of shape (B) / (B, 1), integer encoded

validate(model_output_shape, target_shape)

Validate the meter.

Checks if the meter can be calculated on the given model_output_shape and target_shape.

property value

Value of meter based on local state, can be any python object.

Note

If there are multiple training processes then this represents the local state of the meter. If sync_state() is implemented, then value will return the global state since the last sync PLUS any local unsynced updates that have occurred in the local process.

class classy_vision.meters.ClassyMeter

Base class to measure various metrics during training and testing phases.

This can include meters like Accuracy, Precision and Recall, etc.

classmethod from_config(config: Dict[str, Any]) → classy_vision.meters.classy_meter.ClassyMeter

Instantiates a ClassyMeter using a configuration.

Parameters

config – A configuration for a ClassyMeter.

Returns

A ClassyMeter instance.

get_classy_state() → Dict[str, Any]

Get the state of the ClassyMeter.

The returned state is used for checkpointing.

Returns

A state dictionary containing the state of the meter.

property name

The name of the meter.

reset()

Resets any internal meter state.

Should normally be called at the end of a phase.

set_classy_state(state: Dict[str, Any]) → None

Set the state of the ClassyMeter.

Parameters

state_dict – The state dictionary. Must be the output of a call to get_classy_state().

This is used to load the state of the meter from a checkpoint.

sync_state() → None

Syncs state with all other meters in distributed training.

If not provided by child class this does nothing by default and meter only provides the local process stats. If implemented then the meter provides the global stats at last sync + any local updates since the last sync.

Warning

Calls to sync_state could involve communications via torch.distributed which can result in a loss of performance or deadlocks if not coordinated among threads.

update(model_output: torch.Tensor, target: torch.Tensor, **kwargs) → None

Updates any internal state of meter.

Should be called after each batch processing of each phase.

Parameters
  • model_output – Output of a ClassyModel.

  • target – Target provided by a dataloader from ClassyDataset.

validate(model_output_shape: Tuple, target_shape: Tuple) → None

Validate the meter.

Checks if the meter can be calculated on the given model_output_shape and target_shape.

property value

Value of meter based on local state, can be any python object.

Note

If there are multiple training processes then this represents the local state of the meter. If sync_state() is implemented, then value will return the global state since the last sync PLUS any local unsynced updates that have occurred in the local process.

class classy_vision.meters.PrecisionAtKMeter(topk)

Meter to calculate top-k precision for single-label or multi-label image classification task. Note, ties are resolved randomly.

__init__(topk)
Parameters

topk – list of int k values.

classmethod from_config(config: Dict[str, Any]) → classy_vision.meters.precision_meter.PrecisionAtKMeter

Instantiates a PrecisionAtKMeter from a configuration.

Parameters

config – A configuration for a PrecisionAtKMeter. See __init__() for parameters expected in the config.

Returns

A PrecisionAtKMeter instance.

get_classy_state()

Contains the states of the meter.

property name

The name of the meter.

reset()

Resets any internal meter state.

Should normally be called at the end of a phase.

set_classy_state(state)

Set the state of the ClassyMeter.

Parameters

state_dict – The state dictionary. Must be the output of a call to get_classy_state().

This is used to load the state of the meter from a checkpoint.

sync_state()

Syncs state with all other meters in distributed training.

If not provided by child class this does nothing by default and meter only provides the local process stats. If implemented then the meter provides the global stats at last sync + any local updates since the last sync.

Warning

Calls to sync_state could involve communications via torch.distributed which can result in a loss of performance or deadlocks if not coordinated among threads.

update(model_output, target, **kwargs)
Parameters
  • model_output – tensor of shape (B, C) where each value is either logit or class probability.

  • target – tensor of shape (B, C), which is one-hot / multi-label encoded, or tensor of shape (B) / (B, 1), integer encoded

validate(model_output_shape, target_shape)

Validate the meter.

Checks if the meter can be calculated on the given model_output_shape and target_shape.

property value

Value of meter based on local state, can be any python object.

Note

If there are multiple training processes then this represents the local state of the meter. If sync_state() is implemented, then value will return the global state since the last sync PLUS any local unsynced updates that have occurred in the local process.

class classy_vision.meters.RecallAtKMeter(topk, target_is_one_hot=True, num_classes=None)

Meter to calculate top-k recall for single-label or multi-label image classification task.

__init__(topk, target_is_one_hot=True, num_classes=None)
Parameters

topk – list of int k values.

classmethod from_config(config: Dict[str, Any]) → classy_vision.meters.recall_meter.RecallAtKMeter

Instantiates a RecallAtKMeter from a configuration.

Parameters

config – A configuration for a RecallAtKMeter. See __init__() for parameters expected in the config.

Returns

A RecallAtKMeter instance.

get_classy_state()

Contains the states of the meter.

property name

The name of the meter.

reset()

Resets any internal meter state.

Should normally be called at the end of a phase.

set_classy_state(state)

Set the state of the ClassyMeter.

Parameters

state_dict – The state dictionary. Must be the output of a call to get_classy_state().

This is used to load the state of the meter from a checkpoint.

sync_state()

Syncs state with all other meters in distributed training.

If not provided by child class this does nothing by default and meter only provides the local process stats. If implemented then the meter provides the global stats at last sync + any local updates since the last sync.

Warning

Calls to sync_state could involve communications via torch.distributed which can result in a loss of performance or deadlocks if not coordinated among threads.

update(model_output, target, **kwargs)
Parameters
  • model_output – tensor of shape (B, C) where each value is either logit or class probability.

  • target – tensor of shape (B, C), which is one-hot / multi-label encoded, or tensor of shape (B) / (B, 1), integer encoded

validate(model_output_shape, target_shape)

Validate the meter.

Checks if the meter can be calculated on the given model_output_shape and target_shape.

property value

Value of meter based on local state, can be any python object.

Note

If there are multiple training processes then this represents the local state of the meter. If sync_state() is implemented, then value will return the global state since the last sync PLUS any local unsynced updates that have occurred in the local process.

class classy_vision.meters.VideoAccuracyMeter(topk, clips_per_video_train, clips_per_video_test)
Meter to calculate top-k video-level accuracy for single/multi label

video classification task.

Video-level accuarcy is computed by averaging clip-level predictions and compare the reslt with video-level groundtruth label.

__init__(topk, clips_per_video_train, clips_per_video_test)
Parameters
  • topk – list of int k values.

  • clips_per_video_train – No. of clips sampled per video at train time

  • clips_per_video_test – No. of clips sampled per video at test time

classmethod from_config(config: Dict[str, Any]) → classy_vision.meters.video_accuracy_meter.VideoAccuracyMeter

Instantiates a VideoAccuracyMeter from a configuration.

Parameters

config – A configuration for a VideoAccuracyMeter. See __init__() for parameters expected in the config.

Returns

A VideoAccuracyMeter instance.

property meter

Every video meter should implement to have its own internal meter.

It consumes the video level predictions and ground truth label, and compute the actual metrics.

Returns

An instance of ClassyMeter.

property name

The name of the meter.

classy_vision.meters.build_meter(config)

Builds a ClassyMeter from a config.

This assumes a ‘name’ key in the config which is used to determine what meter class to instantiate. For instance, a config {“name”: “my_meter”, “foo”: “bar”} will find a class that was registered as “my_meter” (see register_meter()) and call .from_config on it.

classy_vision.meters.build_meters(config)
classy_vision.meters.register_meter(name)

Registers a ClassyMeter subclass.

This decorator allows Classy Vision to instantiate a subclass of ClassyMeter from a configuration file, even if the class itself is not part of the Classy Vision framework. To use it, apply this decorator to a ClassyMeter subclass, like this:

@register_meter('accuracy')
class AccuracyMeter(ClassyMeter):
    ...

To instantiate a meter from a configuration file, see build_meter().