Mismatch of AP as compared to pycocotools due to mismatch number of categories.

[chunchet-ng]

Hi @dbolya,

I modified this awesome library for my own use case and test it on a new dataset. However, I found out that the AP @ IOU 0.5 is different from what I get when using pycocotools. The root cause of this issue is the mismatched number of categories between groundtruth and prediction. For example, I defined 10 categories in the categories dictionary (in JSON). But in fact, the groundtruth annotations only involve 8 categories. At this point, both TIDE and pycocotools will output the same AP by calculating (sum(APs) / 8), only when the predictions cover 8 or less categories.

However, the AP will be different if the prediction involves all 10 (or more than 8) categories. Let's assume that the total AP @ IOU 0.5 that I will get is 100.

What I get from cocoeval:
AP: 100 /8 = 12.5

What I get from TIDE:
AP: 100/10 = 10

The main reason that leads to this result is pycocotools only considers the number of classes available in groundtruth, which is 8. On the other hand, TIDE will consider all 10 classes, as the 2 respective ClassedAPDataObject are not empty (len(self.data_points) > 0).

This use case happens when the training set has 10 classes, but the validation set only has 8 of them. I m training my model with 10 classes, and sometimes it will output 10 classes while inferring on the validation set.

What do you think of this mismatch of results?

Thank you in advance.

[qraleq]

Hi @chunchet-ng, I have observed the same thing in my experiments. Did you ever figure out how to fix this?

[chunchet-ng]

Hi @qraleq, the most straightforward fix to this is to make sure that the number of classes in the annotations is exactly the same as in the categories dictionary defined in your JSON file.

From my past experiences, the mAP difference is negligible if you have a lot of classes. I hope this helps, do let me know if you want to extend TIDE, because I m working in a similar direction.