Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Metric value may slightly randomly change #303

Closed
AlekseySh opened this issue Feb 20, 2023 · 4 comments
Closed

Metric value may slightly randomly change #303

AlekseySh opened this issue Feb 20, 2023 · 4 comments
Assignees
@dapladoc
Labels
check
Projects
OML-planning

Comments

@AlekseySh
Copy link
Contributor

AlekseySh commented Feb 20, 2023 •
edited

We observed that the metric value might slightly change depending on the list of k for which we want to calculate our metric. Thus, if we calculate CMC for k_vals=(1,10) and k_vals=(1,20) the CMC@1 may be slightly different.

Most likely the reason for instability is in torch.topk and in a way how we calculate the metric:

open-metric-learning/oml/functional/metrics.py

Line 89 in c6004e4

_, ii_top_k = torch.topk(distances, k=max_k, largest=False)

Let's consider an example and calculate precision@2:

is_gt = [1, 1, 0, 1]
dist  = [1, 2, 2, 3]

Two elements have the same distance to the query, so, if top_k picks the first of them, than precision@2 = 1, otherwise precision@2 = 1/2.
@AlekseySh AlekseySh added this to To do in OML-planning via automation Feb 20, 2023
@AlekseySh
Copy link
Contributor Author

@dapladoc in case you are interested ^

@AlekseySh
Copy link
Contributor Author

AlekseySh commented Feb 28, 2023 •
edited

One of the possible solutions is to allow elements with the same distance to the query to share the same k-th place. This approach is used in Hyp-ViT: https://github.com/htdt/hyp_metric/blob/be2b829b21c279ab874f113c648c0296be89134d/helpers.py#L70

@dapladoc
Copy link
Collaborator

Hi, I'm interested in this issue.

@AlekseySh AlekseySh changed the title Metric value may slightly change depending on required @k Metric value may slightly randomly change Feb 28, 2023
@AlekseySh AlekseySh moved this from To do to In progress in OML-planning Mar 13, 2023
@AlekseySh AlekseySh linked a pull request Jun 4, 2023 that will close this issue
Feature/metrics/stable metrics #382
Closed
@AlekseySh AlekseySh moved this from In progress to Review in progress in OML-planning Jun 23, 2023
@AlekseySh
Copy link
Contributor Author

AlekseySh commented Jun 7, 2024 •
edited

Closed because of: #382 (comment)

OML-planning automation moved this from Review in progress to Done Jun 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dapladoc dapladoc

Labels
check
Projects
OML-planning
Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Feature/metrics/stable metrics dapladoc/open-metric-learning
2 participants
@dapladoc @AlekseySh