Metrics
This is the documentation for Metrics functions.
Incident recall
A new metric as an extension of recall, which for now we’ll call ‘traditional recall’. It can be most easily explained from an example:
Note
A prediction has to be made if a value will go over a threshold in the near future. We call such an exceedance an incident. These incidents have to be predicted before they happen, so someone can act on it. They need some time to act so we want to know six to four hours in advance. A prediction will be made every hour on the hour, so at t-6, t-5 and t-4 we expect a positive prediction. In our dataset we have 10 incident. Say we predict, for all those 10 incidents, correctly at t-5 (true postive) and incorrectly at t-6 and t-4. In this case we have a ‘traditional recall’ of only 33%.
This 33% is however not representative of how many incidents the model actually saw coming in the timespan of t-6 to t-4. Every incident was predicted to happen correctly at some point in time. To fix this we define incident recall. We aggregate the positive predictionos per incident to say if at any point during the lead up the incident was correctly predicted. In our previous (contrived) example the incident recall metric would be a 100%, as all incidents were predicted correctly at t-5.
Note that incident recall >= recall in every case, traditional recall is a lower bound. Also note that this does not translate to precision. For precision you care how many of your positive predicitions are correct, regardless if the incident was also correctly predict a moment earlier or later. We can still use ‘traditional’ precision.