Understanding intersection-over-union
Published:
Intersection-over-union (IoU), also known as the Jaccard index, is a commonly used measure for determining how accurate a proposed image segmentation is, compared to a known/ground-truth segmentation. In segmentation tasks the IoU is prefered over accuracy as it is not as affected by the class imblances that are inherent in foreground/background segmentation tasks. As an example, if a ground truth image is made up of 90% background pixels, a proposed segmentation that classifies all pixels as “background” will have an accuracy of 90% whereas it would have an IoU of 0%. The definition of IoU between a known segmentation of \(n\) pixels, \(Y\), and a similar set of predicted segmentation, \(\hat{Y}\) (in the binary case, i.e. where \(Y_i, \hat{Y}_i \in \{0,1\}, \forall i \in [1,n]\)) is as follows:
\[IoU(Y, \hat{Y}) = \frac{Y \cap \hat{Y}}{Y \cup \hat{Y}} = \frac{\sum_{i=1}^n \min(Y_i, \hat{Y}_i)}{\sum_{i=1}^n \max(Y_i, \hat{Y}_i)}\]Written as a function of the confusion matrix between \(Y\) and \(\hat{Y}\):
\[\begin{array}{|c|c|c|c|} \hline & \hat{Y}=1 & \hat{Y}=0\\ \hline Y=1 & \text{TP} & \text{FN}\\ \hline Y=0 & \text{FP} & \text{TN}\\ \hline \end{array}\]where (\(\text{TP} = \text{True positives}, \text{FP} = \text{False positives}\), etc.), IoU is:
\[IoU(Y, \hat{Y}) = \frac{TP}{TP + FN + FP}\]As the IoU can range from 0 to 1, it is usually expressed as a percent, however the intuition behind what an IoU score means in terms of visual error is not intuitive (to me atleast). How much better is an IoU score of 0.9 than an IoU score of 0.8 (in terms of accuracy, this difference would mean 10% more observations were correctly classified)? This problem is excacerbated as there can be many segmentations that correspond to a particular IoU score.
To get a better feel for how IoU changes with different segmentations I created the interactive tool below. You can move and resize the “ground truth” and “predicted” segmentations to see how the IoU changes; the intersection between the two will be highlighted in green. As a challenge, try to make several segmentation pairs that all match up to 0.8. While attempting this you should get a better feel for the meaning of IoU!