TY  - CONF
AU  - Baiyu Chen
AU  - Sergio Escalera
AU  - Isabelle Guyon
AU  - Victor Ponce
AU  - N. Shah
AU  - Marc Oliu
A2  - ECCVW
PY  - 2016//
TI  - Overcoming Calibration Problems in Pattern Labeling with Pairwise Ratings: Application to Personality Traits
BT  - 14th European Conference on Computer Vision Workshops
KW  - Calibration of labels
KW  - Label bias
KW  - Ordinal labeling
KW  - Variance Models
KW  - Bradley-Terry-Luce model
KW  - Continuous labels
KW  - Regression
KW  - Personality traits
KW  - Crowd-sourced labels
N2  - We address the problem of calibration of workers whose task is to label patterns with continuous variables, which arises for instance in labeling images of videos of humans with continuous traits. Worker bias is particularly dicult to evaluate and correct when many workers contribute just a few labels, a situation arising typically when labeling is crowd-sourced. In the scenario of labeling short videos of people facing a camera with personality traits, we evaluate the feasibility of the pairwise ranking method to alleviate bias problems. Workers are exposed to pairs of  videos  at  a  time  and  must  order  by  preference.  The  variable  levels are reconstructed by  fitting a Bradley-Terry-Luce model with maximum likelihood.  This  method  may  at   first  sight,  seem  prohibitively  expensive  because  for N videos, p = N (N-1)/2  pairs  must  be  potentially processed by workers rather that N videos. However, by performing extensive simulations, we determine an empirical law for the scaling of the number of pairs needed as a function of the number of videos in order to achieve a given accuracy of score reconstruction and show that the pairwise method is a ordable. We apply the method to the labeling of a  large  scale  dataset  of  10,000  videos  used  in  the  ChaLearn  Apparent Personality Trait challenge.
L1  - http://refbase.cvc.uab.es/files/CEG2016.pdf
N1  - HuPBA;MILAB;
ID  - Baiyu Chen2016
ER  -