| Home | << 1 2 3 4 5 6 7 8 9 10 >> |
|
| Records | |||||
|---|---|---|---|---|---|
| Author | Josep Brugues Pujolras; Lluis Gomez; Dimosthenis Karatzas | ||||
| Title | A Multilingual Approach to Scene Text Visual Question Answering | Type | Conference Article | ||
| Year | 2022 | Publication | Document Analysis Systems.15th IAPR International Workshop, (DAS2022) | Abbreviated Journal | |
| Volume | Issue | Pages | 65-79 | ||
Keywords  |
Scene text; Visual question answering; Multilingual word embeddings; Vision and language; Deep learning | ||||
| Abstract | Scene Text Visual Question Answering (ST-VQA) has recently emerged as a hot research topic in Computer Vision. Current ST-VQA models have a big potential for many types of applications but lack the ability to perform well on more than one language at a time due to the lack of multilingual data, as well as the use of monolingual word embeddings for training. In this work, we explore the possibility to obtain bilingual and multilingual VQA models. In that regard, we use an already established VQA model that uses monolingual word embeddings as part of its pipeline and substitute them by FastText and BPEmb multilingual word embeddings that have been aligned to English. Our experiments demonstrate that it is possible to obtain bilingual and multilingual VQA models with a minimal loss in performance in languages not used during training, as well as a multilingual model trained in multiple languages that match the performance of the respective monolingual baselines. | ||||
| Address | La Rochelle, France; May 22–25, 2022 | ||||
| Corporate Author | Thesis | ||||
| Publisher | Place of Publication | Editor | |||
| Language | Summary Language | Original Title | |||
| Series Editor | Series Title | Abbreviated Series Title | |||
| Series Volume | Series Issue | Edition | |||
| ISSN | ISBN | Medium | |||
| Area | Expedition | Conference | DAS | ||
| Notes | DAG; 611.004; 600.155; 601.002 | Approved | no | ||
| Call Number | Admin @ si @ BGK2022b | Serial | 3695 | ||
| Permanent link to this record | |||||
| Author | Diego Velazquez; Pau Rodriguez; Josep M. Gonfaus; Xavier Roca; Jordi Gonzalez | ||||
| Title | A Closer Look at Embedding Propagation for Manifold Smoothing | Type | Journal Article | ||
| Year | 2022 | Publication | Journal of Machine Learning Research | Abbreviated Journal | JMLR |
| Volume | 23 | Issue | 252 | Pages | 1-27 |
| Keywords |
Regularization; emi-supervised learning; self-supervised learning; adversarial robustness; few-shot classification | ||||
| Abstract | Supervised training of neural networks requires a large amount of manually annotated data and the resulting networks tend to be sensitive to out-of-distribution (OOD) data.
Self- and semi-supervised training schemes reduce the amount of annotated data required during the training process. However, OOD generalization remains a major challenge for most methods. Strategies that promote smoother decision boundaries play an important role in out-of-distribution generalization. For example, embedding propagation (EP) for manifold smoothing has recently shown to considerably improve the OOD performance for few-shot classification. EP achieves smoother class manifolds by building a graph from sample embeddings and propagating information through the nodes in an unsupervised manner. In this work, we extend the original EP paper providing additional evidence and experiments showing that it attains smoother class embedding manifolds and improves results in settings beyond few-shot classification. Concretely, we show that EP improves the robustness of neural networks against multiple adversarial attacks as well as semi- and self-supervised learning performance. |
||||
| Address | 9/2022 | ||||
| Corporate Author | Thesis | ||||
| Publisher | Place of Publication | Editor | |||
| Language | Summary Language | Original Title | |||
| Series Editor | Series Title | Abbreviated Series Title | |||
| Series Volume | Series Issue | Edition | |||
| ISSN | ISBN | Medium | |||
| Area | Expedition | Conference | |||
| Notes | Approved | no | |||
| Call Number | Admin @ si @ VRG2022 | Serial | 3762 | ||
| Permanent link to this record | |||||
| Author | Patricia Suarez; Angel Sappa; Dario Carpio; Henry Velesaca; Francisca Burgos; Patricia Urdiales | ||||
| Title | Deep Learning Based Shrimp Classification | Type | Conference Article | ||
| Year | 2022 | Publication | 17th International Symposium on Visual Computing | Abbreviated Journal | |
| Volume | 13598 | Issue | Pages | 36–45 | |
| Keywords |
Pigmentation; Color space; Light weight network | ||||
| Abstract | This work proposes a novel approach based on deep learning to address the classification of shrimp (Pennaeus vannamei) into two classes, according to their level of pigmentation accepted by shrimp commerce. The main goal of this actual study is to support the shrimp industry in terms of price and process. An efficient CNN architecture is proposed to perform image classification through a program that could be set other in mobile devices or in fixed support in the shrimp supply chain. The proposed approach is a lightweight model that uses HSV color space shrimp images. A simple pipeline shows the most important stages performed to determine a pattern that identifies the class to which they belong based on their pigmentation. For the experiments, a database acquired with mobile devices of various brands and models has been used to capture images of shrimp. The results obtained with the images in the RGB and HSV color space allow for testing the effectiveness of the proposed model. | ||||
| Address | |||||
| Corporate Author | Thesis | ||||
| Publisher | Place of Publication | Editor | |||
| Language | Summary Language | Original Title | |||
| Series Editor | Series Title | Abbreviated Series Title | |||
| Series Volume | Series Issue | Edition | |||
| ISSN | ISBN | Medium | |||
| Area | Expedition | Conference | ISVC | ||
| Notes | MSIAU; no proj | Approved | no | ||
| Call Number | Admin @ si @ SAC2022 | Serial | 3772 | ||
| Permanent link to this record | |||||
| Author | Julio C. S. Jacques Junior; Yagmur Gucluturk; Marc Perez; Umut Guçlu; Carlos Andujar; Xavier Baro; Hugo Jair Escalante; Isabelle Guyon; Marcel A. J. van Gerven; Rob van Lier; Sergio Escalera | ||||
| Title | First Impressions: A Survey on Vision-Based Apparent Personality Trait Analysis | Type | Journal Article | ||
| Year | 2022 | Publication | IEEE Transactions on Affective Computing | Abbreviated Journal | TAC |
| Volume | 13 | Issue | 1 | Pages | 75-95 |
| Keywords |
Personality computing; first impressions; person perception; big-five; subjective bias; computer vision; machine learning; nonverbal signals; facial expression; gesture; speech analysis; multi-modal recognition | ||||
| Abstract | Personality analysis has been widely studied in psychology, neuropsychology, and signal processing fields, among others. From the past few years, it also became an attractive research area in visual computing. From the computational point of view, by far speech and text have been the most considered cues of information for analyzing personality. However, recently there has been an increasing interest from the computer vision community in analyzing personality from visual data. Recent computer vision approaches are able to accurately analyze human faces, body postures and behaviors, and use these information to infer apparent personality traits. Because of the overwhelming research interest in this topic, and of the potential impact that this sort of methods could have in society, we present in this paper an up-to-date review of existing vision-based approaches for apparent personality trait recognition. We describe seminal and cutting edge works on the subject, discussing and comparing their distinctive features and limitations. Future venues of research in the field are identified and discussed. Furthermore, aspects on the subjectivity in data labeling/evaluation, as well as current datasets and challenges organized to push the research on the field are reviewed. | ||||
| Address | 1 Jan.-March 2022 | ||||
| Corporate Author | Thesis | ||||
| Publisher | Place of Publication | Editor | |||
| Language | Summary Language | Original Title | |||
| Series Editor | Series Title | Abbreviated Series Title | |||
| Series Volume | Series Issue | Edition | |||
| ISSN | ISBN | Medium | |||
| Area | Expedition | Conference | |||
| Notes | HuPBA | Approved | no | ||
| Call Number | Admin @ si @ JGP2022 | Serial | 3724 | ||
| Permanent link to this record | |||||
| Author | Pau Torras; Arnau Baro; Alicia Fornes; Lei Kang | ||||
| Title | Improving Handwritten Music Recognition through Language Model Integration | Type | Conference Article | ||
| Year | 2022 | Publication | 4th International Workshop on Reading Music Systems (WoRMS2022) | Abbreviated Journal | |
| Volume | Issue | Pages | 42-46 | ||
| Keywords |
optical music recognition; historical sources; diversity; music theory; digital humanities | ||||
| Abstract | Handwritten Music Recognition, especially in the historical domain, is an inherently challenging endeavour; paper degradation artefacts and the ambiguous nature of handwriting make recognising such scores an error-prone process, even for the current state-of-the-art Sequence to Sequence models. In this work we propose a way of reducing the production of statistically implausible output sequences by fusing a Language Model into a recognition Sequence to Sequence model. The idea is leveraging visually-conditioned and context-conditioned output distributions in order to automatically find and correct any mistakes that would otherwise break context significantly. We have found this approach to improve recognition results to 25.15 SER (%) from a previous best of 31.79 SER (%) in the literature. | ||||
| Address | November 18, 2022 | ||||
| Corporate Author | Thesis | ||||
| Publisher | Place of Publication | Editor | |||
| Language | Summary Language | Original Title | |||
| Series Editor | Series Title | Abbreviated Series Title | |||
| Series Volume | Series Issue | Edition | |||
| ISSN | ISBN | Medium | |||
| Area | Expedition | Conference | WoRMS | ||
| Notes | DAG; 600.121; 600.162; 602.230 | Approved | no | ||
| Call Number | Admin @ si @ TBF2022 | Serial | 3735 | ||
| Permanent link to this record | |||||
| Author | Arnau Baro; Carles Badal; Pau Torras; Alicia Fornes | ||||
| Title | Handwritten Historical Music Recognition through Sequence-to-Sequence with Attention Mechanism | Type | Conference Article | ||
| Year | 2022 | Publication | 3rd International Workshop on Reading Music Systems (WoRMS2021) | Abbreviated Journal | |
| Volume | Issue | Pages | 55-59 | ||
| Keywords |
Optical Music Recognition; Digits; Image Classification | ||||
| Abstract | Despite decades of research in Optical Music Recognition (OMR), the recognition of old handwritten music scores remains a challenge because of the variabilities in the handwriting styles, paper degradation, lack of standard notation, etc. Therefore, the research in OMR systems adapted to the particularities of old manuscripts is crucial to accelerate the conversion of music scores existing in archives into digital libraries, fostering the dissemination and preservation of our music heritage. In this paper we explore the adaptation of sequence-to-sequence models with attention mechanism (used in translation and handwritten text recognition) and the generation of specific synthetic data for recognizing old music scores. The experimental validation demonstrates that our approach is promising, especially when compared with long short-term memory neural networks. | ||||
| Address | July 23, 2021, Alicante (Spain) | ||||
| Corporate Author | Thesis | ||||
| Publisher | Place of Publication | Editor | |||
| Language | Summary Language | Original Title | |||
| Series Editor | Series Title | Abbreviated Series Title | |||
| Series Volume | Series Issue | Edition | |||
| ISSN | ISBN | Medium | |||
| Area | Expedition | Conference | WoRMS | ||
| Notes | DAG; 600.121; 600.162; 602.230; 600.140 | Approved | no | ||
| Call Number | Admin @ si @ BBT2022 | Serial | 3734 | ||
| Permanent link to this record | |||||
| Author | Arnau Baro; Pau Riba; Alicia Fornes | ||||
| Title | Musigraph: Optical Music Recognition Through Object Detection and Graph Neural Network | Type | Conference Article | ||
| Year | 2022 | Publication | Frontiers in Handwriting Recognition. International Conference on Frontiers in Handwriting Recognition (ICFHR2022) | Abbreviated Journal | |
| Volume | 13639 | Issue | Pages | 171-184 | |
| Keywords |
Object detection; Optical music recognition; Graph neural network | ||||
| Abstract | During the last decades, the performance of optical music recognition has been increasingly improving. However, and despite the 2-dimensional nature of music notation (e.g. notes have rhythm and pitch), most works treat musical scores as a sequence of symbols in one dimension, which make their recognition still a challenge. Thus, in this work we explore the use of graph neural networks for musical score recognition. First, because graphs are suited for n-dimensional representations, and second, because the combination of graphs with deep learning has shown a great performance in similar applications. Our methodology consists of: First, we will detect each isolated/atomic symbols (those that can not be decomposed in more graphical primitives) and the primitives that form a musical symbol. Then, we will build the graph taking as root node the notehead and as leaves those primitives or symbols that modify the note’s rhythm (stem, beam, flag) or pitch (flat, sharp, natural). Finally, the graph is translated into a human-readable character sequence for a final transcription and evaluation. Our method has been tested on more than five thousand measures, showing promising results. | ||||
| Address | December 04 – 07, 2022; Hyderabad, India | ||||
| Corporate Author | Thesis | ||||
| Publisher | Place of Publication | Editor | |||
| Language | Summary Language | Original Title | |||
| Series Editor | Series Title | Abbreviated Series Title | LNCS | ||
| Series Volume | Series Issue | Edition | |||
| ISSN | ISBN | Medium | |||
| Area | Expedition | Conference | ICFHR | ||
| Notes | DAG; 600.162; 600.140; 602.230 | Approved | no | ||
| Call Number | Admin @ si @ BRF2022b | Serial | 3740 | ||
| Permanent link to this record | |||||
| Author | Idoia Ruiz; Joan Serrat | ||||
| Title | Hierarchical Novelty Detection for Traffic Sign Recognition | Type | Journal Article | ||
| Year | 2022 | Publication | Sensors | Abbreviated Journal | SENS |
| Volume | 22 | Issue | 12 | Pages | 4389 |
| Keywords |
Novelty detection; hierarchical classification; deep learning; traffic sign recognition; autonomous driving; computer vision | ||||
| Abstract | Recent works have made significant progress in novelty detection, i.e., the problem of detecting samples of novel classes, never seen during training, while classifying those that belong to known classes. However, the only information this task provides about novel samples is that they are unknown. In this work, we leverage hierarchical taxonomies of classes to provide informative outputs for samples of novel classes. We predict their closest class in the taxonomy, i.e., its parent class. We address this problem, known as hierarchical novelty detection, by proposing a novel loss, namely Hierarchical Cosine Loss that is designed to learn class prototypes along with an embedding of discriminative features consistent with the taxonomy. We apply it to traffic sign recognition, where we predict the parent class semantics for new types of traffic signs. Our model beats state-of-the art approaches on two large scale traffic sign benchmarks, Mapillary Traffic Sign Dataset (MTSD) and Tsinghua-Tencent 100K (TT100K), and performs similarly on natural images benchmarks (AWA2, CUB). For TT100K and MTSD, our approach is able to detect novel samples at the correct nodes of the hierarchy with 81% and 36% of accuracy, respectively, at 80% known class accuracy. | ||||
| Address | |||||
| Corporate Author | Thesis | ||||
| Publisher | Place of Publication | Editor | |||
| Language | Summary Language | Original Title | |||
| Series Editor | Series Title | Abbreviated Series Title | |||
| Series Volume | Series Issue | Edition | |||
| ISSN | ISBN | Medium | |||
| Area | Expedition | Conference | |||
| Notes | ADAS; 600.154 | Approved | no | ||
| Call Number | Admin @ si @ RuS2022 | Serial | 3684 | ||
| Permanent link to this record | |||||
| Author | Bhalaji Nagarajan; Ricardo Marques; Marcos Mejia; Petia Radeva | ||||
| Title | Class-conditional Importance Weighting for Deep Learning with Noisy Labels | Type | Conference Article | ||
| Year | 2022 | Publication | 17th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications | Abbreviated Journal | |
| Volume | 5 | Issue | Pages | 679-686 | |
| Keywords |
Noisy Labeling; Loss Correction; Class-conditional Importance Weighting; Learning with Noisy Labels | ||||
| Abstract | Large-scale accurate labels are very important to the Deep Neural Networks to train them and assure high performance. However, it is very expensive to create a clean dataset since usually it relies on human interaction. To this purpose, the labelling process is made cheap with a trade-off of having noisy labels. Learning with Noisy Labels is an active area of research being at the same time very challenging. The recent advances in Self-supervised learning and robust loss functions have helped in advancing noisy label research. In this paper, we propose a loss correction method that relies on dynamic weights computed based on the model training. We extend the existing Contrast to Divide algorithm coupled with DivideMix using a new class-conditional weighted scheme. We validate the method using the standard noise experiments and achieved encouraging results. | ||||
| Address | Virtual; February 2022 | ||||
| Corporate Author | Thesis | ||||
| Publisher | Place of Publication | Editor | |||
| Language | Summary Language | Original Title | |||
| Series Editor | Series Title | Abbreviated Series Title | |||
| Series Volume | Series Issue | Edition | |||
| ISSN | ISBN | Medium | |||
| Area | Expedition | Conference | VISAPP | ||
| Notes | MILAB; no menciona | Approved | no | ||
| Call Number | Admin @ si @ NMM2022 | Serial | 3798 | ||
| Permanent link to this record | |||||
| Author | Asma Bensalah; Alicia Fornes; Cristina Carmona_Duarte; Josep Llados | ||||
| Title | Easing Automatic Neurorehabilitation via Classification and Smoothness Analysis | Type | Conference Article | ||
| Year | 2022 | Publication | Intertwining Graphonomics with Human Movements. 20th International Conference of the International Graphonomics Society, IGS 2022 | Abbreviated Journal | |
| Volume | 13424 | Issue | Pages | 336-348 | |
| Keywords |
Neurorehabilitation; Upper-lim; Movement classification; Movement smoothness; Deep learning; Jerk | ||||
| Abstract | Assessing the quality of movements for post-stroke patients during the rehabilitation phase is vital given that there is no standard stroke rehabilitation plan for all the patients. In fact, it depends basically on the patient’s functional independence and its progress along the rehabilitation sessions. To tackle this challenge and make neurorehabilitation more agile, we propose an automatic assessment pipeline that starts by recognising patients’ movements by means of a shallow deep learning architecture, then measuring the movement quality using jerk measure and related measures. A particularity of this work is that the dataset used is clinically relevant, since it represents movements inspired from Fugl-Meyer a well common upper-limb clinical stroke assessment scale for stroke patients. We show that it is possible to detect the contrast between healthy and patients movements in terms of smoothness, besides achieving conclusions about the patients’ progress during the rehabilitation sessions that correspond to the clinicians’ findings about each case. | ||||
| Address | June 7-9, 2022, Las Palmas de Gran Canaria, Spain | ||||
| Corporate Author | Thesis | ||||
| Publisher | Place of Publication | Editor | |||
| Language | Summary Language | Original Title | |||
| Series Editor | Series Title | Abbreviated Series Title | LNCS | ||
| Series Volume | Series Issue | Edition | |||
| ISSN | ISBN | Medium | |||
| Area | Expedition | Conference | IGS | ||
| Notes | DAG; 600.121; 600.162; 602.230; 600.140 | Approved | no | ||
| Call Number | Admin @ si @ BFC2022 | Serial | 3738 | ||
| Permanent link to this record | |||||
| Author | Giuseppe De Gregorio; Sanket Biswas; Mohamed Ali Souibgui; Asma Bensalah; Josep Llados; Alicia Fornes; Angelo Marcelli | ||||
| Title | A Few Shot Multi-representation Approach for N-Gram Spotting in Historical Manuscripts | Type | Conference Article | ||
| Year | 2022 | Publication | Frontiers in Handwriting Recognition. International Conference on Frontiers in Handwriting Recognition (ICFHR2022) | Abbreviated Journal | |
| Volume | 13639 | Issue | Pages | 3-12 | |
| Keywords |
N-gram spotting; Few-shot learning; Multimodal understanding; Historical handwritten collections | ||||
| Abstract | Despite recent advances in automatic text recognition, the performance remains moderate when it comes to historical manuscripts. This is mainly because of the scarcity of available labelled data to train the data-hungry Handwritten Text Recognition (HTR) models. The Keyword Spotting System (KWS) provides a valid alternative to HTR due to the reduction in error rate, but it is usually limited to a closed reference vocabulary. In this paper, we propose a few-shot learning paradigm for spotting sequences of a few characters (N-gram) that requires a small amount of labelled training data. We exhibit that recognition of important n-grams could reduce the system’s dependency on vocabulary. In this case, an out-of-vocabulary (OOV) word in an input handwritten line image could be a sequence of n-grams that belong to the lexicon. An extensive experimental evaluation of our proposed multi-representation approach was carried out on a subset of Bentham’s historical manuscript collections to obtain some really promising results in this direction. | ||||
| Address | December 04 – 07, 2022; Hyderabad, India | ||||
| Corporate Author | Thesis | ||||
| Publisher | Place of Publication | Editor | |||
| Language | Summary Language | Original Title | |||
| Series Editor | Series Title | Abbreviated Series Title | LNCS | ||
| Series Volume | Series Issue | Edition | |||
| ISSN | ISBN | Medium | |||
| Area | Expedition | Conference | ICFHR | ||
| Notes | DAG; 600.121; 600.162; 602.230; 600.140 | Approved | no | ||
| Call Number | Admin @ si @ GBS2022 | Serial | 3733 | ||
| Permanent link to this record | |||||
| Author | Aitor Alvarez-Gila; Joost Van de Weijer; Yaxing Wang; Estibaliz Garrote | ||||
| Title | MVMO: A Multi-Object Dataset for Wide Baseline Multi-View Semantic Segmentation | Type | Conference Article | ||
| Year | 2022 | Publication | 29th IEEE International Conference on Image Processing | Abbreviated Journal | |
| Volume | Issue | Pages | |||
| Keywords |
multi-view; cross-view; semantic segmentation; synthetic dataset | ||||
| Abstract | We present MVMO (Multi-View, Multi-Object dataset): a synthetic dataset of 116,000 scenes containing randomly placed objects of 10 distinct classes and captured from 25 camera locations in the upper hemisphere. MVMO comprises photorealistic, path-traced image renders, together with semantic segmentation ground truth for every view. Unlike existing multi-view datasets, MVMO features wide baselines between cameras and high density of objects, which lead to large disparities, heavy occlusions and view-dependent object appearance. Single view semantic segmentation is hindered by self and inter-object occlusions that could benefit from additional viewpoints. Therefore, we expect that MVMO will propel research in multi-view semantic segmentation and cross-view semantic transfer. We also provide baselines that show that new research is needed in such fields to exploit the complementary information of multi-view setups 1 . | ||||
| Address | Bordeaux; France; October2022 | ||||
| Corporate Author | Thesis | ||||
| Publisher | Place of Publication | Editor | |||
| Language | Summary Language | Original Title | |||
| Series Editor | Series Title | Abbreviated Series Title | |||
| Series Volume | Series Issue | Edition | |||
| ISSN | ISBN | Medium | |||
| Area | Expedition | Conference | ICIP | ||
| Notes | LAMP | Approved | no | ||
| Call Number | Admin @ si @ AWW2022 | Serial | 3781 | ||
| Permanent link to this record | |||||
| Author | Jorge Charco; Angel Sappa; Boris X. Vintimilla | ||||
| Title | Human Pose Estimation through a Novel Multi-view Scheme | Type | Conference Article | ||
| Year | 2022 | Publication | 17th International Conference on Computer Vision Theory and Applications (VISAPP 2022) | Abbreviated Journal | |
| Volume | 5 | Issue | Pages | 855-862 | |
| Keywords |
Multi-view Scheme; Human Pose Estimation; Relative Camera Pose; Monocular Approach | ||||
| Abstract | This paper presents a multi-view scheme to tackle the challenging problem of the self-occlusion in human pose estimation problem. The proposed approach first obtains the human body joints of a set of images, which are captured from different views at the same time. Then, it enhances the obtained joints by using a
multi-view scheme. Basically, the joints from a given view are used to enhance poorly estimated joints from another view, especially intended to tackle the self occlusions cases. A network architecture initially proposed for the monocular case is adapted to be used in the proposed multi-view scheme. Experimental results and comparisons with the state-of-the-art approaches on Human3.6m dataset are presented showing improvements in the accuracy of body joints estimations. |
||||
| Address | On line; Feb 6, 2022 – Feb 8, 2022 | ||||
| Corporate Author | Thesis | ||||
| Publisher | Place of Publication | Editor | |||
| Language | Summary Language | Original Title | |||
| Series Editor | Series Title | Abbreviated Series Title | |||
| Series Volume | Series Issue | Edition | |||
| ISSN | 2184-4321 | ISBN | 978-989-758-555-5 | Medium | |
| Area | Expedition | Conference | VISAPP | ||
| Notes | MSIAU; 600.160 | Approved | no | ||
| Call Number | Admin @ si @ CSV2022 | Serial | 3689 | ||
| Permanent link to this record | |||||
| Author | Ali Furkan Biten; Lluis Gomez; Dimosthenis Karatzas | ||||
| Title | Let there be a clock on the beach: Reducing Object Hallucination in Image Captioning | Type | Conference Article | ||
| Year | 2022 | Publication | Winter Conference on Applications of Computer Vision | Abbreviated Journal | |
| Volume | Issue | Pages | 1381-1390 | ||
| Keywords |
Measurement; Training; Visualization; Analytical models; Computer vision; Computational modeling; Training data | ||||
| Abstract | Explaining an image with missing or non-existent objects is known as object bias (hallucination) in image captioning. This behaviour is quite common in the state-of-the-art captioning models which is not desirable by humans. To decrease the object hallucination in captioning, we propose three simple yet efficient training augmentation method for sentences which requires no new training data or increase
in the model size. By extensive analysis, we show that the proposed methods can significantly diminish our models’ object bias on hallucination metrics. Moreover, we experimentally demonstrate that our methods decrease the dependency on the visual features. All of our code, configuration files and model weights are available online. |
||||
| Address | Virtual; Waikoloa; Hawai; USA; January 2022 | ||||
| Corporate Author | Thesis | ||||
| Publisher | Place of Publication | Editor | |||
| Language | Summary Language | Original Title | |||
| Series Editor | Series Title | Abbreviated Series Title | |||
| Series Volume | Series Issue | Edition | |||
| ISSN | ISBN | Medium | |||
| Area | Expedition | Conference | WACV | ||
| Notes | DAG; 600.155; 302.105 | Approved | no | ||
| Call Number | Admin @ si @ BGK2022 | Serial | 3662 | ||
| Permanent link to this record | |||||
| Author | Ali Furkan Biten; Andres Mafla; Lluis Gomez; Dimosthenis Karatzas | ||||
| Title | Is An Image Worth Five Sentences? A New Look into Semantics for Image-Text Matching | Type | Conference Article | ||
| Year | 2022 | Publication | Winter Conference on Applications of Computer Vision | Abbreviated Journal | |
| Volume | Issue | Pages | 1391-1400 | ||
| Keywords |
Measurement; Training; Integrated circuits; Annotations; Semantics; Training data; Semisupervised learning | ||||
| Abstract | The task of image-text matching aims to map representations from different modalities into a common joint visual-textual embedding. However, the most widely used datasets for this task, MSCOCO and Flickr30K, are actually image captioning datasets that offer a very limited set of relationships between images and sentences in their ground-truth annotations. This limited ground truth information forces us to use evaluation metrics based on binary relevance: given a sentence query we consider only one image as relevant. However, many other relevant images or captions may be present in the dataset. In this work, we propose two metrics that evaluate the degree of semantic relevance of retrieved items, independently of their annotated binary relevance. Additionally, we incorporate a novel strategy that uses an image captioning metric, CIDEr, to define a Semantic Adaptive Margin (SAM) to be optimized in a standard triplet loss. By incorporating our formulation to existing models, a large improvement is obtained in scenarios where available training data is limited. We also demonstrate that the performance on the annotated image-caption pairs is maintained while improving on other non-annotated relevant items when employing the full training set. The code for our new metric can be found at github. com/furkanbiten/ncsmetric and the model implementation at github. com/andrespmd/semanticadaptive_margin. | ||||
| Address | Virtual; Waikoloa; Hawai; USA; January 2022 | ||||
| Corporate Author | Thesis | ||||
| Publisher | Place of Publication | Editor | |||
| Language | Summary Language | Original Title | |||
| Series Editor | Series Title | Abbreviated Series Title | |||
| Series Volume | Series Issue | Edition | |||
| ISSN | ISBN | Medium | |||
| Area | Expedition | Conference | WACV | ||
| Notes | DAG; 600.155; 302.105; | Approved | no | ||
| Call Number | Admin @ si @ BMG2022 | Serial | 3663 | ||
| Permanent link to this record | |||||