|
|
Francesco Pelosin, Saurav Jha, Andrea Torsello, Bogdan Raducanu, & Joost Van de Weijer. (2022). Towards exemplar-free continual learning in vision transformers: an account of attention, functional and weight regularization. In IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).
Abstract: In this paper, we investigate the continual learning of Vision Transformers (ViT) for the challenging exemplar-free scenario, with special focus on how to efficiently distill the knowledge of its crucial self-attention mechanism (SAM). Our work takes an initial step towards a surgical investigation of SAM for designing coherent continual learning methods in ViTs. We first carry out an evaluation of established continual learning regularization techniques. We then examine the effect of regularization when applied to two key enablers of SAM: (a) the contextualized embedding layers, for their ability to capture well-scaled representations with respect to the values, and (b) the prescaled attention maps, for carrying value-independent global contextual information. We depict the perks of each distilling strategy on two image recognition benchmarks (CIFAR100 and ImageNet-32) – while (a) leads to a better overall accuracy, (b) helps enhance the rigidity by maintaining competitive performances. Furthermore, we identify the limitation imposed by the symmetric nature of regularization losses. To alleviate this, we propose an asymmetric variant and apply it to the pooled output distillation (POD) loss adapted for ViTs. Our experiments confirm that introducing asymmetry to POD boosts its plasticity while retaining stability across (a) and (b). Moreover, we acknowledge low forgetting measures for all the compared methods, indicating that ViTs might be naturally inclined continual learners. 1
Keywords: Learning systems; Weight measurement; Image recognition; Surgery; Benchmark testing; Transformers; Stability analysis
|
|
|
|
Hector Laria Mantecon, Yaxing Wang, Joost Van de Weijer, & Bogdan Raducanu. (2022). Transferring Unconditional to Conditional GANs With Hyper-Modulation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).
Abstract: GANs have matured in recent years and are able to generate high-resolution, realistic images. However, the computational resources and the data required for the training of high-quality GANs are enormous, and the study of transfer learning of these models is therefore an urgent topic. Many of the available high-quality pretrained GANs are unconditional (like StyleGAN). For many applications, however, conditional GANs are preferable, because they provide more control over the generation process, despite often suffering more training difficulties. Therefore, in this paper, we focus on transferring from high-quality pretrained unconditional GANs to conditional GANs. This requires architectural adaptation of the pretrained GAN to perform the conditioning. To this end, we propose hyper-modulated generative networks that allow for shared and complementary supervision. To prevent the additional weights of the hypernetwork to overfit, with subsequent mode collapse on small target domains, we introduce a self-initialization procedure that does not require any real data to initialize the hypernetwork parameters. To further improve the sample efficiency of the transfer, we apply contrastive learning in the discriminator, which effectively works on very limited batch sizes. In extensive experiments, we validate the efficiency of the hypernetworks, self-initialization and contrastive loss for knowledge transfer on standard benchmarks.
|
|
|
|
Dipam Goswami, Yuyang Liu, Bartlomiej Twardowski, & Joost Van de Weijer. (2023). FeCAM: Exploiting the Heterogeneity of Class Distributions in Exemplar-Free Continual Learning. In 37th Annual Conference on Neural Information Processing Systems.
|
|
|
|
Kai Wang, Fei Yang, Shiqi Yang, Muhammad Atif Butt, & Joost Van de Weijer. (2023). Dynamic Prompt Learning: Addressing Cross-Attention Leakage for Text-Based Image Editing. In 37th Annual Conference on Neural Information Processing Systems.
|
|
|
|
ChuanMing Fang, Kai Wang, & Joost Van de Weijer. (2023). IterInv: Iterative Inversion for Pixel-Level T2I Models. In 37th Annual Conference on Neural Information Processing Systems.
Abstract: Large-scale text-to-image diffusion models have been a ground-breaking development in generating convincing images following an input text prompt. The goal of image editing research is to give users control over the generated images by modifying the text prompt. Current image editing techniques are relying on DDIM inversion as a common practice based on the Latent Diffusion Models (LDM). However, the large pretrained T2I models working on the latent space as LDM suffer from losing details due to the first compression stage with an autoencoder mechanism. Instead, another mainstream T2I pipeline working on the pixel level, such as Imagen and DeepFloyd-IF, avoids this problem. They are commonly composed of several stages, normally with a text-to-image stage followed by several super-resolution stages. In this case, the DDIM inversion is unable to find the initial noise to generate the original image given that the super-resolution diffusion models are not compatible with the DDIM technique. According to our experimental findings, iteratively concatenating the noisy image as the condition is the root of this problem. Based on this observation, we develop an iterative inversion (IterInv) technique for this stream of T2I models and verify IterInv with the open-source DeepFloyd-IF model. By combining our method IterInv with a popular image editing method, we prove the application prospects of IterInv. The code will be released at \url{this https URL}.
|
|
|
|
Kai Wang, Xialei Liu, Andrew Bagdanov, Luis Herranz, Shangling Jui, & Joost Van de Weijer. (2022). Incremental Meta-Learning via Episodic Replay Distillation for Few-Shot Image Recognition. In CVPR 2022 Workshop on Continual Learning (CLVision, 3rd Edition) (pp. 3728–3738).
Abstract: In this paper we consider the problem of incremental meta-learning in which classes are presented incrementally in discrete tasks. We propose Episodic Replay Distillation (ERD), that mixes classes from the current task with exemplars from previous tasks when sampling episodes for meta-learning. To allow the training to benefit from a large as possible variety of classes, which leads to more gener-
alizable feature representations, we propose the cross-task meta loss. Furthermore, we propose episodic replay distillation that also exploits exemplars for improved knowledge distillation. Experiments on four datasets demonstrate that ERD surpasses the state-of-the-art. In particular, on the more challenging one-shot, long task sequence scenarios, we reduce the gap between Incremental Meta-Learning and
the joint-training upper bound from 3.5% / 10.1% / 13.4% / 11.7% with the current state-of-the-art to 2.6% / 2.9% / 5.0% / 0.2% with our method on Tiered-ImageNet / Mini-ImageNet / CIFAR100 / CUB, respectively.
Keywords: Training; Computer vision; Image recognition; Upper bound; Conferences; Pattern recognition; Task analysis
|
|
|
|
Bojana Gajic, Ariel Amato, Ramon Baldrich, Joost Van de Weijer, & Carlo Gatta. (2022). Area Under the ROC Curve Maximization for Metric Learning. In CVPR 2022 Workshop on Efficien Deep Learning for Computer Vision (ECV 2022, 5th Edition).
Abstract: Most popular metric learning losses have no direct relation with the evaluation metrics that are subsequently applied to evaluate their performance. We hypothesize that training a metric learning model by maximizing the area under the ROC curve (which is a typical performance measure of recognition systems) can induce an implicit ranking suitable for retrieval problems. This hypothesis is supported by previous work that proved that a curve dominates in ROC space if and only if it dominates in Precision-Recall space. To test this hypothesis, we design and maximize an approximated, derivable relaxation of the area under the ROC curve. The proposed AUC loss achieves state-of-the-art results on two large scale retrieval benchmark datasets (Stanford Online Products and DeepFashion In-Shop). Moreover, the AUC loss achieves comparable performance to more complex, domain specific, state-of-the-art methods for vehicle re-identification.
Keywords: Training; Computer vision; Conferences; Area measurement; Benchmark testing; Pattern recognition
|
|
|
|
Alex Gomez-Villa, Bartlomiej Twardowski, Lu Yu, Andrew Bagdanov, & Joost Van de Weijer. (2022). Continually Learning Self-Supervised Representations With Projected Functional Regularization. In CVPR 2022 Workshop on Continual Learning (CLVision, 3rd Edition) (pp. 3866–3876).
Abstract: Recent self-supervised learning methods are able to learn high-quality image representations and are closing the gap with supervised approaches. However, these methods are unable to acquire new knowledge incrementally – they are, in fact, mostly used only as a pre-training phase over IID data. In this work we investigate self-supervised methods in continual learning regimes without any replay
mechanism. We show that naive functional regularization,also known as feature distillation, leads to lower plasticity and limits continual learning performance. Instead, we propose Projected Functional Regularization in which a separate temporal projection network ensures that the newly learned feature space preserves information of the previous one, while at the same time allowing for the learning of new features. This prevents forgetting while maintaining the plasticity of the learner. Comparison with other incremental learning approaches applied to self-supervision demonstrates that our method obtains competitive performance in
different scenarios and on multiple datasets.
Keywords: Computer vision; Conferences; Self-supervised learning; Image representation; Pattern recognition
|
|
|
|
Mohamed Ramzy Ibrahim, Robert Benavente, Felipe Lumbreras, & Daniel Ponsa. (2022). 3DRRDB: Super Resolution of Multiple Remote Sensing Images using 3D Residual in Residual Dense Blocks. In CVPR 2022 Workshop on IEEE Perception Beyond the Visible Spectrum workshop series (PBVS, 18th Edition).
Abstract: The rapid advancement of Deep Convolutional Neural Networks helped in solving many remote sensing problems, especially the problems of super-resolution. However, most state-of-the-art methods focus more on Single Image Super-Resolution neglecting Multi-Image Super-Resolution. In this work, a new proposed 3D Residual in Residual Dense Blocks model (3DRRDB) focuses on remote sensing Multi-Image Super-Resolution for two different single spectral bands. The proposed 3DRRDB model explores the idea of 3D convolution layers in deeply connected Dense Blocks and the effect of local and global residual connections with residual scaling in Multi-Image Super-Resolution. The model tested on the Proba-V challenge dataset shows a significant improvement above the current state-of-the-art models scoring a Corrected Peak Signal to Noise Ratio (cPSNR) of 48.79 dB and 50.83 dB for Near Infrared (NIR) and RED Bands respectively. Moreover, the proposed 3DRRDB model scores a Corrected Structural Similarity Index Measure (cSSIM) of 0.9865 and 0.9909 for NIR and RED bands respectively.
Keywords: Training; Solid modeling; Three-dimensional displays; PSNR; Convolution; Superresolution; Pattern recognition
|
|
|
|
Angel Sappa, & M.A. Garcia. (2004). Hierarchical Clustering of 3D Objects and its Application to Minimum Distance Computation. In IEEE International Conference on Robotics & Automation, 5287–5292, New Orleans, LA (USA), ISBN: 0–7803–8232–3.
|
|
|
|
N. Zakaria, Jean-Marc Ogier, & Josep Llados. (2006). The Fuzzy-Spatial Descriptor for the Online Graphic Recognition: Overlapping Matrix Algorithm. In 7th International Workshop, Document Analysis Systems VII (DAS´06), LNCS 3872: 616–627.
|
|
|
|
T.O. Nguyen, Salvatore Tabbone, & Oriol Ramos Terrades. (2008). Symbol Descriptor Based on Shape Context and Vector Model of Information Retrieval. In Proceedings of the 8th IAPR International Workshop on Document Analysis Systems, (pp. 191–197).
|
|
|
|
Partha Pratim Roy, Umapada Pal, & Josep Llados. (2008). Multi-oriented English Text Line Extraction using Background and Foreground Information. In Proceedings of the 8th IAPR International Workshop on Document Analysis Systems, (315–322).
|
|
|
|
Marçal Rusiñol, & Josep Llados. (2008). Word and Symbol Spotting using Spatial Organization of Local Descriptors. In Proceedings of the 8th IAPR International Workshop on Document Analysis Systems, (489–496).
|
|
|
|
Mathieu Nicolas Delalandre, Ernest Valveny, & Josep Llados. (2008). Performance Evaluation of Symbol Recognition and Spotting Systems. In Proceedings of the 8th International Workshop on Document Analysis Systems, (497–505).
|
|
