TY  - JOUR
AU  - Yaxing Wang
AU  - Luis Herranz
AU  - Joost Van de Weijer
PY  - 2020//
TI  - Mix and match networks: multi-domain alignment for unpaired image-to-image translation
T2  - IJCV
JO  - International Journal of Computer Vision
SP  - 2849–2872
VL  - 128
N2  - This paper addresses the problem of inferring unseen cross-modal image-to-image translations between multiple modalities. We assume that only some of the pairwise translations have been seen (i.e. trained) and infer the remaining unseen translations (where training pairs are not available). We propose mix and match networks, an approach where multiple encoders and decoders are aligned in such a way that the desired translation can be obtained by simply cascading the source encoder and the target decoder, even when they have not interacted during the training stage (i.e. unseen). The main challenge lies in the alignment of the latent representations at the bottlenecks of encoder-decoder pairs. We propose an architecture with several tools to encourage alignment, including autoencoders and robust side information and latent consistency losses. We show the benefits of our approach in terms of effectiveness and scalability compared with other pairwise image-to-image translation approaches. We also propose zero-pair cross-modal image translation, a challenging setting where the objective is inferring semantic segmentation from depth (and vice-versa) without explicit segmentation-depth pairs, and only from two (disjoint) segmentation-RGB and depth-RGB training sets. We observe that a certain part of the shared information between unseen modalities might not be reachable, so we further propose a variant that leverages pseudo-pairs which allows us to exploit this shared information between the unseen modalities
UR  - https://link.springer.com/article/10.1007/s11263-020-01340-z
L1  - http://refbase.cvc.uab.es/files/.pdf
UR  - http://dx.doi.org/10.1007/s11263-020-01340-z
N1  - LAMP; 600.109; 600.106; 600.141; 600.120
ID  - Yaxing Wang2020
ER  -