ADLStream.evaluation.PrequentialEvaluator

Prequential Evaluator.

This evaluator implements the idea that more recent examples are more important. It uses a decaying factor. It is based on the inteleaved chunks evaluator, which incrementally updates the accuracy by evaluating chunks of data sequentially.

The fading factor is implemented as follow:

```
S = loss + fading_factor * S_prev
N = 1 + fading_factor * N_prev
preq_loss = S/N
```

Parameters:

Name	Type	Description	Default
chunck_size	int	Number of instances per chunk. the particular case of chunk_size = 1, represents prequential the interleaved train-then-test approach.	required
metric	str	loss function. Possible options can be found in ADLStream.evaluation.metrics.	required
fadding_factor	float	Fadding factor. Defaults to 0.98.	0.98
results_file	str	Name of the csv file where to write results. If None, no csv file is created. Defaults to "ADLStream.csv".	'ADLStream.csv'
dataset_name	str	Name of the data to validate. Defaults to None.	None
show_plot	bool	Whether to plot the evolution of the metric. Defaults to True.	True
plot_file	str	Name of the plot image file. If None, no image is saved. Defaults to None.	None

Source code in ADLStream/evaluation/prequential.py

class PrequentialEvaluator(InterleavedChunkEvaluator):
    """Prequential Evaluator.

    This evaluator implements the idea that more recent examples are more important. It
    uses a decaying factor. It is based on the inteleaved chunks evaluator, which
    incrementally updates the accuracy by evaluating chunks of data sequentially.

    The fading factor is implemented as follow:

        ```
        S = loss + fading_factor * S_prev
        N = 1 + fading_factor * N_prev
        preq_loss = S/N
        ```

    Arguments:
        chunck_size (int): Number of instances per chunk.
            the particular case of chunk_size = 1, represents prequential the
            interleaved train-then-test approach.
        metric (str): loss function.
            Possible options can be found in ADLStream.evaluation.metrics.
        fadding_factor (float, optional): Fadding factor.
            Defaults to 0.98.
        results_file (str, optional): Name of the csv file where to write results.
            If None, no csv file is created.
            Defaults to "ADLStream.csv".
        dataset_name (str, optional): Name of the data to validate.
            Defaults to None.
        show_plot (bool, optional): Whether to plot the evolution of the metric.
            Defaults to True.
        plot_file (str, optional): Name of the plot image file.
            If None, no image is saved.
            Defaults to None.
    """

    def __init__(
        self,
        chunk_size,
        metric,
        fadding_factor=0.98,
        results_file="ADLStream.csv",
        dataset_name=None,
        show_plot=True,
        plot_file=None,
        **kwargs
    ):
        self.fadding_factor = fadding_factor
        self.prev_metric = None
        self.prev_n = None
        super().__init__(
            chunk_size,
            metric,
            results_file=results_file,
            dataset_name=dataset_name,
            show_plot=show_plot,
            plot_file=plot_file,
            **kwargs
        )

    def compute_metric(self):
        new_metric = super().compute_metric()

        if self.prev_metric is None:
            self.prev_metric = new_metric
            self.prev_n = 1
            return new_metric

        new_metric = new_metric + self.fadding_factor * self.prev_metric
        n = 1 + self.fadding_factor * self.prev_n

        self.prev_metric = new_metric
        self.prev_n = n

        return new_metric / n