%0 Conference Proceedings
%T Overcoming Calibration Problems in Pattern Labeling with Pairwise Ratings: Application to Personality Traits
%A Baiyu Chen
%A Sergio Escalera
%A Isabelle Guyon
%A Victor Ponce
%A N. Shah
%A Marc Oliu
%B 14th European Conference on Computer Vision Workshops
%D 2016
%F Baiyu Chen2016
%O HuPBA;MILAB;
%O exported from refbase (http://refbase.cvc.uab.es/show.php?record=2829), last updated on Fri, 26 Feb 2021 14:22:05 +0100
%X We address the problem of calibration of workers whose task is to label patterns with continuous variables, which arises for instance in labeling images of videos of humans with continuous traits. Worker bias is particularly dicult to evaluate and correct when many workers contribute just a few labels, a situation arising typically when labeling is crowd-sourced. In the scenario of labeling short videos of people facing a camera with personality traits, we evaluate the feasibility of the pairwise ranking method to alleviate bias problems. Workers are exposed to pairs of  videos  at  a  time  and  must  order  by  preference.  The  variable  levels are reconstructed by  fitting a Bradley-Terry-Luce model with maximum likelihood.  This  method  may  at   first  sight,  seem  prohibitively  expensive  because  for N videos, p = N (N-1)/2  pairs  must  be  potentially processed by workers rather that N videos. However, by performing extensive simulations, we determine an empirical law for the scaling of the number of pairs needed as a function of the number of videos in order to achieve a given accuracy of score reconstruction and show that the pairwise method is a ordable. We apply the method to the labeling of a  large  scale  dataset  of  10,000  videos  used  in  the  ChaLearn  Apparent Personality Trait challenge.
%K Calibration of labels
%K Label bias
%K Ordinal labeling
%K Variance Models
%K Bradley-Terry-Luce model
%K Continuous labels
%K Regression
%K Personality traits
%K Crowd-sourced labels
%U http://refbase.cvc.uab.es/files/CEG2016.pdf