TY  - CONF
AU  - Anguelos Nicolaou
AU  - Sounak Dey
AU  - V.Christlein
AU  - A.Maier
AU  - Dimosthenis Karatzas
PY  - 2018//
TI  - Non-deterministic Behavior of Ranking-based Metrics when Evaluating Embeddings
T2  - LNCS
BT  - International Workshop on Reproducible Research in Pattern Recognition
SP  - 71
EP  - 82
VL  - 11455
N2  - Embedding data into vector spaces is a very popular strategy of pattern recognition methods. When distances between embeddings are quantized, performance metrics become ambiguous. In this paper, we present an analysis of the ambiguity quantized distances introduce and provide bounds on the effect. We demonstrate that it can have a measurable effect in empirical data in state-of-the-art systems. We also approach the phenomenon from a computer security perspective and demonstrate how someone being evaluated by a third party can exploit this ambiguity and greatly outperform a random predictor without even access to the input data. We also suggest a simple solution making the performance metrics, which rely on ranking, totally deterministic and impervious to such exploits.
UR  - https://link.springer.com/chapter/10.1007/978-3-030-23987-9_5
L1  - http://refbase.cvc.uab.es/files/NDC2018.pdf
N1  - DAG; 600.121; 600.129
ID  - Anguelos Nicolaou2018
ER  -