| Records |
| Author |
Carlos Boned Riera; Oriol Ramos Terrades |
| Title |
Discriminative Neural Variational Model for Unbalanced Classification Tasks in Knowledge Graph |
Type |
Conference Article |
| Year |
2022 |
Publication |
26th International Conference on Pattern Recognition |
Abbreviated Journal |
|
| Volume |
|
Issue |
|
Pages |
2186-2191 |
| Keywords |
Measurement; Couplings; Semantics; Ear; Benchmark testing; Data models; Pattern recognition |
| Abstract |
Nowadays the paradigm of link discovery problems has shown significant improvements on Knowledge Graphs. However, method performances are harmed by the unbalanced nature of this classification problem, since many methods are easily biased to not find proper links. In this paper we present a discriminative neural variational auto-encoder model, called DNVAE from now on, in which we have introduced latent variables to serve as embedding vectors. As a result, the learnt generative model approximate better the underlying distribution and, at the same time, it better differentiate the type of relations in the knowledge graph. We have evaluated this approach on benchmark knowledge graph and Census records. Results in this last data set are quite impressive since we reach the highest possible score in the evaluation metrics. However, further experiments are still needed to deeper evaluate the performance of the method in more challenging tasks. |
| Address |
Montreal; Quebec; Canada; August 2022 |
| Corporate Author |
|
Thesis |
|
| Publisher |
|
Place of Publication |
|
Editor |
|
| Language |
|
Summary Language |
|
Original Title |
|
| Series Editor |
|
Series Title |
|
Abbreviated Series Title |
|
| Series Volume |
|
Series Issue |
|
Edition |
|
| ISSN |
|
ISBN |
|
Medium |
|
| Area |
|
Expedition |
|
Conference |
ICPR |
Notes  |
DAG; 600.121; 600.162 |
Approved |
no |
| Call Number |
Admin @ si @ BoR2022 |
Serial |
3741 |
| Permanent link to this record |
| |
|
| |
| Author |
Pau Torras; Arnau Baro; Alicia Fornes; Lei Kang |
| Title |
Improving Handwritten Music Recognition through Language Model Integration |
Type |
Conference Article |
| Year |
2022 |
Publication |
4th International Workshop on Reading Music Systems (WoRMS2022) |
Abbreviated Journal |
|
| Volume |
|
Issue |
|
Pages |
42-46 |
| Keywords |
optical music recognition; historical sources; diversity; music theory; digital humanities |
| Abstract |
Handwritten Music Recognition, especially in the historical domain, is an inherently challenging endeavour; paper degradation artefacts and the ambiguous nature of handwriting make recognising such scores an error-prone process, even for the current state-of-the-art Sequence to Sequence models. In this work we propose a way of reducing the production of statistically implausible output sequences by fusing a Language Model into a recognition Sequence to Sequence model. The idea is leveraging visually-conditioned and context-conditioned output distributions in order to automatically find and correct any mistakes that would otherwise break context significantly. We have found this approach to improve recognition results to 25.15 SER (%) from a previous best of 31.79 SER (%) in the literature. |
| Address |
November 18, 2022 |
| Corporate Author |
|
Thesis |
|
| Publisher |
|
Place of Publication |
|
Editor |
|
| Language |
|
Summary Language |
|
Original Title |
|
| Series Editor |
|
Series Title |
|
Abbreviated Series Title |
|
| Series Volume |
|
Series Issue |
|
Edition |
|
| ISSN |
|
ISBN |
|
Medium |
|
| Area |
|
Expedition |
|
Conference |
WoRMS |
| Notes |
DAG; 600.121; 600.162; 602.230 |
Approved |
no |
| Call Number |
Admin @ si @ TBF2022 |
Serial |
3735 |
| Permanent link to this record |
| |
|
| |
| Author |
Mohamed Ali Souibgui; Alicia Fornes; Yousri Kessentini; Beata Megyesi |
| Title |
Few shots are all you need: A progressive learning approach for low resource handwritten text recognition |
Type |
Journal Article |
| Year |
2022 |
Publication |
Pattern Recognition Letters |
Abbreviated Journal |
PRL |
| Volume |
160 |
Issue |
|
Pages |
43-49 |
| Keywords |
|
| Abstract |
Handwritten text recognition in low resource scenarios, such as manuscripts with rare alphabets, is a challenging problem. In this paper, we propose a few-shot learning-based handwriting recognition approach that significantly reduces the human annotation process, by requiring only a few images of each alphabet symbols. The method consists of detecting all the symbols of a given alphabet in a textline image and decoding the obtained similarity scores to the final sequence of transcribed symbols. Our model is first pretrained on synthetic line images generated from an alphabet, which could differ from the alphabet of the target domain. A second training step is then applied to reduce the gap between the source and the target data. Since this retraining would require annotation of thousands of handwritten symbols together with their bounding boxes, we propose to avoid such human effort through an unsupervised progressive learning approach that automatically assigns pseudo-labels to the unlabeled data. The evaluation on different datasets shows that our model can lead to competitive results with a significant reduction in human effort. The code will be publicly available in the following repository: https://github.com/dali92002/HTRbyMatching |
| Address |
|
| Corporate Author |
|
Thesis |
|
| Publisher |
Elsevier |
Place of Publication |
|
Editor |
|
| Language |
|
Summary Language |
|
Original Title |
|
| Series Editor |
|
Series Title |
|
Abbreviated Series Title |
|
| Series Volume |
|
Series Issue |
|
Edition |
|
| ISSN |
|
ISBN |
|
Medium |
|
| Area |
|
Expedition |
|
Conference |
|
| Notes |
DAG; 600.121; 600.162; 602.230 |
Approved |
no |
| Call Number |
Admin @ si @ SFK2022 |
Serial |
3736 |
| Permanent link to this record |
| |
|
| |
| Author |
Giacomo Magnifico; Beata Megyesi; Mohamed Ali Souibgui; Jialuo Chen; Alicia Fornes |
| Title |
Lost in Transcription of Graphic Signs in Ciphers |
Type |
Conference Article |
| Year |
2022 |
Publication |
International Conference on Historical Cryptology (HistoCrypt 2022) |
Abbreviated Journal |
|
| Volume |
|
Issue |
|
Pages |
153-158 |
| Keywords |
transcription of ciphers; hand-written text recognition of symbols; graphic signs |
| Abstract |
Hand-written Text Recognition techniques with the aim to automatically identify and transcribe hand-written text have been applied to historical sources including ciphers. In this paper, we compare the performance of two machine learning architectures, an unsupervised method based on clustering and a deep learning method with few-shot learning. Both models are tested on seen and unseen data from historical ciphers with different symbol sets consisting of various types of graphic signs. We compare the models and highlight their differences in performance, with their advantages and shortcomings. |
| Address |
Amsterdam, Netherlands, June 20-22, 2022 |
| Corporate Author |
|
Thesis |
|
| Publisher |
|
Place of Publication |
|
Editor |
|
| Language |
|
Summary Language |
|
Original Title |
|
| Series Editor |
|
Series Title |
|
Abbreviated Series Title |
|
| Series Volume |
|
Series Issue |
|
Edition |
|
| ISSN |
|
ISBN |
|
Medium |
|
| Area |
|
Expedition |
|
Conference |
HystoCrypt |
| Notes |
DAG; 600.121; 600.162; 602.230; 600.140 |
Approved |
no |
| Call Number |
Admin @ si @ MBS2022 |
Serial |
3731 |
| Permanent link to this record |
| |
|
| |
| Author |
Mohamed Ali Souibgui; Sanket Biswas; Sana Khamekhem Jemni; Yousri Kessentini; Alicia Fornes; Josep Llados; Umapada Pal |
| Title |
DocEnTr: An End-to-End Document Image Enhancement Transformer |
Type |
Conference Article |
| Year |
2022 |
Publication |
26th International Conference on Pattern Recognition |
Abbreviated Journal |
|
| Volume |
|
Issue |
|
Pages |
1699-1705 |
| Keywords |
Degradation; Head; Optical character recognition; Self-supervised learning; Benchmark testing; Transformers; Magnetic heads |
| Abstract |
Document images can be affected by many degradation scenarios, which cause recognition and processing difficulties. In this age of digitization, it is important to denoise them for proper usage. To address this challenge, we present a new encoder-decoder architecture based on vision transformers to enhance both machine-printed and handwritten document images, in an end-to-end fashion. The encoder operates directly on the pixel patches with their positional information without the use of any convolutional layers, while the decoder reconstructs a clean image from the encoded patches. Conducted experiments show a superiority of the proposed model compared to the state-of the-art methods on several DIBCO benchmarks. Code and models will be publicly available at: https://github.com/dali92002/DocEnTR |
| Address |
August 21-25, 2022 , Montréal Québec |
| Corporate Author |
|
Thesis |
|
| Publisher |
|
Place of Publication |
|
Editor |
|
| Language |
|
Summary Language |
|
Original Title |
|
| Series Editor |
|
Series Title |
|
Abbreviated Series Title |
|
| Series Volume |
|
Series Issue |
|
Edition |
|
| ISSN |
|
ISBN |
|
Medium |
|
| Area |
|
Expedition |
|
Conference |
ICPR |
| Notes |
DAG; 600.121; 600.162; 602.230; 600.140 |
Approved |
no |
| Call Number |
Admin @ si @ SBJ2022 |
Serial |
3730 |
| Permanent link to this record |
| |
|
| |
| Author |
Giuseppe De Gregorio; Sanket Biswas; Mohamed Ali Souibgui; Asma Bensalah; Josep Llados; Alicia Fornes; Angelo Marcelli |
| Title |
A Few Shot Multi-representation Approach for N-Gram Spotting in Historical Manuscripts |
Type |
Conference Article |
| Year |
2022 |
Publication |
Frontiers in Handwriting Recognition. International Conference on Frontiers in Handwriting Recognition (ICFHR2022) |
Abbreviated Journal |
|
| Volume |
13639 |
Issue |
|
Pages |
3-12 |
| Keywords |
N-gram spotting; Few-shot learning; Multimodal understanding; Historical handwritten collections |
| Abstract |
Despite recent advances in automatic text recognition, the performance remains moderate when it comes to historical manuscripts. This is mainly because of the scarcity of available labelled data to train the data-hungry Handwritten Text Recognition (HTR) models. The Keyword Spotting System (KWS) provides a valid alternative to HTR due to the reduction in error rate, but it is usually limited to a closed reference vocabulary. In this paper, we propose a few-shot learning paradigm for spotting sequences of a few characters (N-gram) that requires a small amount of labelled training data. We exhibit that recognition of important n-grams could reduce the system’s dependency on vocabulary. In this case, an out-of-vocabulary (OOV) word in an input handwritten line image could be a sequence of n-grams that belong to the lexicon. An extensive experimental evaluation of our proposed multi-representation approach was carried out on a subset of Bentham’s historical manuscript collections to obtain some really promising results in this direction. |
| Address |
December 04 – 07, 2022; Hyderabad, India |
| Corporate Author |
|
Thesis |
|
| Publisher |
|
Place of Publication |
|
Editor |
|
| Language |
|
Summary Language |
|
Original Title |
|
| Series Editor |
|
Series Title |
|
Abbreviated Series Title |
LNCS |
| Series Volume |
|
Series Issue |
|
Edition |
|
| ISSN |
|
ISBN |
|
Medium |
|
| Area |
|
Expedition |
|
Conference |
ICFHR |
| Notes |
DAG; 600.121; 600.162; 602.230; 600.140 |
Approved |
no |
| Call Number |
Admin @ si @ GBS2022 |
Serial |
3733 |
| Permanent link to this record |
| |
|
| |
| Author |
Arnau Baro; Carles Badal; Pau Torras; Alicia Fornes |
| Title |
Handwritten Historical Music Recognition through Sequence-to-Sequence with Attention Mechanism |
Type |
Conference Article |
| Year |
2022 |
Publication |
3rd International Workshop on Reading Music Systems (WoRMS2021) |
Abbreviated Journal |
|
| Volume |
|
Issue |
|
Pages |
55-59 |
| Keywords |
Optical Music Recognition; Digits; Image Classification |
| Abstract |
Despite decades of research in Optical Music Recognition (OMR), the recognition of old handwritten music scores remains a challenge because of the variabilities in the handwriting styles, paper degradation, lack of standard notation, etc. Therefore, the research in OMR systems adapted to the particularities of old manuscripts is crucial to accelerate the conversion of music scores existing in archives into digital libraries, fostering the dissemination and preservation of our music heritage. In this paper we explore the adaptation of sequence-to-sequence models with attention mechanism (used in translation and handwritten text recognition) and the generation of specific synthetic data for recognizing old music scores. The experimental validation demonstrates that our approach is promising, especially when compared with long short-term memory neural networks. |
| Address |
July 23, 2021, Alicante (Spain) |
| Corporate Author |
|
Thesis |
|
| Publisher |
|
Place of Publication |
|
Editor |
|
| Language |
|
Summary Language |
|
Original Title |
|
| Series Editor |
|
Series Title |
|
Abbreviated Series Title |
|
| Series Volume |
|
Series Issue |
|
Edition |
|
| ISSN |
|
ISBN |
|
Medium |
|
| Area |
|
Expedition |
|
Conference |
WoRMS |
| Notes |
DAG; 600.121; 600.162; 602.230; 600.140 |
Approved |
no |
| Call Number |
Admin @ si @ BBT2022 |
Serial |
3734 |
| Permanent link to this record |
| |
|
| |
| Author |
Joana Maria Pujadas-Mora; Alicia Fornes; Oriol Ramos Terrades; Josep Llados; Jialuo Chen; Miquel Valls-Figols; Anna Cabre |
| Title |
The Barcelona Historical Marriage Database and the Baix Llobregat Demographic Database. From Algorithms for Handwriting Recognition to Individual-Level Demographic and Socioeconomic Data |
Type |
Journal |
| Year |
2022 |
Publication |
Historical Life Course Studies |
Abbreviated Journal |
HLCS |
| Volume |
12 |
Issue |
|
Pages |
99-132 |
| Keywords |
Individual demographic databases; Computer vision, Record linkage; Social mobility; Inequality; Migration; Word spotting; Handwriting recognition; Local censuses; Marriage Licences |
| Abstract |
The Barcelona Historical Marriage Database (BHMD) gathers records of the more than 600,000 marriages celebrated in the Diocese of Barcelona and their taxation registered in Barcelona Cathedral's so-called Marriage Licenses Books for the long period 1451–1905 and the BALL Demographic Database brings together the individual information recorded in the population registers, censuses and fiscal censuses of the main municipalities of the county of Baix Llobregat (Barcelona). In this ongoing collection 263,786 individual observations have been assembled, dating from the period between 1828 and 1965 by December 2020. The two databases started as part of different interdisciplinary research projects at the crossroads of Historical Demography and Computer Vision. Their construction uses artificial intelligence and computer vision methods as Handwriting Recognition to reduce the time of execution. However, its current state still requires some human intervention which explains the implemented crowdsourcing and game sourcing experiences. Moreover, knowledge graph techniques have allowed the application of advanced record linkage to link the same individuals and families across time and space. Moreover, we will discuss the main research lines using both databases developed so far in historical demography. |
| Address |
June 23, 2022 |
| Corporate Author |
|
Thesis |
|
| Publisher |
|
Place of Publication |
|
Editor |
|
| Language |
|
Summary Language |
|
Original Title |
|
| Series Editor |
|
Series Title |
|
Abbreviated Series Title |
|
| Series Volume |
|
Series Issue |
|
Edition |
|
| ISSN |
|
ISBN |
|
Medium |
|
| Area |
|
Expedition |
|
Conference |
|
| Notes |
DAG; 600.121; 600.162; 602.230; 600.140 |
Approved |
no |
| Call Number |
Admin @ si @ PFR2022 |
Serial |
3737 |
| Permanent link to this record |
| |
|
| |
| Author |
Asma Bensalah; Alicia Fornes; Cristina Carmona_Duarte; Josep Llados |
| Title |
Easing Automatic Neurorehabilitation via Classification and Smoothness Analysis |
Type |
Conference Article |
| Year |
2022 |
Publication |
Intertwining Graphonomics with Human Movements. 20th International Conference of the International Graphonomics Society, IGS 2022 |
Abbreviated Journal |
|
| Volume |
13424 |
Issue |
|
Pages |
336-348 |
| Keywords |
Neurorehabilitation; Upper-lim; Movement classification; Movement smoothness; Deep learning; Jerk |
| Abstract |
Assessing the quality of movements for post-stroke patients during the rehabilitation phase is vital given that there is no standard stroke rehabilitation plan for all the patients. In fact, it depends basically on the patient’s functional independence and its progress along the rehabilitation sessions. To tackle this challenge and make neurorehabilitation more agile, we propose an automatic assessment pipeline that starts by recognising patients’ movements by means of a shallow deep learning architecture, then measuring the movement quality using jerk measure and related measures. A particularity of this work is that the dataset used is clinically relevant, since it represents movements inspired from Fugl-Meyer a well common upper-limb clinical stroke assessment scale for stroke patients. We show that it is possible to detect the contrast between healthy and patients movements in terms of smoothness, besides achieving conclusions about the patients’ progress during the rehabilitation sessions that correspond to the clinicians’ findings about each case. |
| Address |
June 7-9, 2022, Las Palmas de Gran Canaria, Spain |
| Corporate Author |
|
Thesis |
|
| Publisher |
|
Place of Publication |
|
Editor |
|
| Language |
|
Summary Language |
|
Original Title |
|
| Series Editor |
|
Series Title |
|
Abbreviated Series Title |
LNCS |
| Series Volume |
|
Series Issue |
|
Edition |
|
| ISSN |
|
ISBN |
|
Medium |
|
| Area |
|
Expedition |
|
Conference |
IGS |
| Notes |
DAG; 600.121; 600.162; 602.230; 600.140 |
Approved |
no |
| Call Number |
Admin @ si @ BFC2022 |
Serial |
3738 |
| Permanent link to this record |
| |
|
| |
| Author |
Alicia Fornes; Asma Bensalah; Cristina Carmona_Duarte; Jialuo Chen; Miguel A. Ferrer; Andreas Fischer; Josep Llados; Cristina Martin; Eloy Opisso; Rejean Plamondon; Anna Scius-Bertrand; Josep Maria Tormos |
| Title |
The RPM3D Project: 3D Kinematics for Remote Patient Monitoring |
Type |
Conference Article |
| Year |
2022 |
Publication |
Intertwining Graphonomics with Human Movements. 20th International Conference of the International Graphonomics Society, IGS 2022 |
Abbreviated Journal |
|
| Volume |
13424 |
Issue |
|
Pages |
217-226 |
| Keywords |
Healthcare applications; Kinematic; Theory of Rapid Human Movements; Human activity recognition; Stroke rehabilitation; 3D kinematics |
| Abstract |
This project explores the feasibility of remote patient monitoring based on the analysis of 3D movements captured with smartwatches. We base our analysis on the Kinematic Theory of Rapid Human Movement. We have validated our research in a real case scenario for stroke rehabilitation at the Guttmann Institute (https://www.guttmann.com/en/) (neurorehabilitation hospital), showing promising results. Our work could have a great impact in remote healthcare applications, improving the medical efficiency and reducing the healthcare costs. Future steps include more clinical validation, developing multi-modal analysis architectures (analysing data from sensors, images, audio, etc.), and exploring the application of our technology to monitor other neurodegenerative diseases. |
| Address |
June 7-9, 2022, Las Palmas de Gran Canaria, Spain |
| Corporate Author |
|
Thesis |
|
| Publisher |
|
Place of Publication |
|
Editor |
|
| Language |
|
Summary Language |
|
Original Title |
|
| Series Editor |
|
Series Title |
|
Abbreviated Series Title |
LNCS |
| Series Volume |
|
Series Issue |
|
Edition |
|
| ISSN |
|
ISBN |
|
Medium |
|
| Area |
|
Expedition |
|
Conference |
IGS |
| Notes |
DAG; 600.121; 600.162; 602.230; 600.140 |
Approved |
no |
| Call Number |
Admin @ si @ FBC2022 |
Serial |
3739 |
| Permanent link to this record |
| |
|
| |
| Author |
S.K. Jemni; Mohamed Ali Souibgui; Yousri Kessentini; Alicia Fornes |
| Title |
Enhance to Read Better: A Multi-Task Adversarial Network for Handwritten Document Image Enhancement |
Type |
Journal Article |
| Year |
2022 |
Publication |
Pattern Recognition |
Abbreviated Journal |
PR |
| Volume |
123 |
Issue |
|
Pages |
108370 |
| Keywords |
|
| Abstract |
Handwritten document images can be highly affected by degradation for different reasons: Paper ageing, daily-life scenarios (wrinkles, dust, etc.), bad scanning process and so on. These artifacts raise many readability issues for current Handwritten Text Recognition (HTR) algorithms and severely devalue their efficiency. In this paper, we propose an end to end architecture based on Generative Adversarial Networks (GANs) to recover the degraded documents into a and form. Unlike the most well-known document binarization methods, which try to improve the visual quality of the degraded document, the proposed architecture integrates a handwritten text recognizer that promotes the generated document image to be more readable. To the best of our knowledge, this is the first work to use the text information while binarizing handwritten documents. Extensive experiments conducted on degraded Arabic and Latin handwritten documents demonstrate the usefulness of integrating the recognizer within the GAN architecture, which improves both the visual quality and the readability of the degraded document images. Moreover, we outperform the state of the art in H-DIBCO challenges, after fine tuning our pre-trained model with synthetically degraded Latin handwritten images, on this task. |
| Address |
|
| Corporate Author |
|
Thesis |
|
| Publisher |
|
Place of Publication |
|
Editor |
|
| Language |
|
Summary Language |
|
Original Title |
|
| Series Editor |
|
Series Title |
|
Abbreviated Series Title |
|
| Series Volume |
|
Series Issue |
|
Edition |
|
| ISSN |
|
ISBN |
|
Medium |
|
| Area |
|
Expedition |
|
Conference |
|
| Notes |
DAG; 600.124; 600.121; 602.230 |
Approved |
no |
| Call Number |
Admin @ si @ JSK2022 |
Serial |
3613 |
| Permanent link to this record |
| |
|
| |
| Author |
Adria Molina; Lluis Gomez; Oriol Ramos Terrades; Josep Llados |
| Title |
A Generic Image Retrieval Method for Date Estimation of Historical Document Collections |
Type |
Conference Article |
| Year |
2022 |
Publication |
Document Analysis Systems.15th IAPR International Workshop, (DAS2022) |
Abbreviated Journal |
|
| Volume |
13237 |
Issue |
|
Pages |
583–597 |
| Keywords |
Date estimation; Document retrieval; Image retrieval; Ranking loss; Smooth-nDCG |
| Abstract |
Date estimation of historical document images is a challenging problem, with several contributions in the literature that lack of the ability to generalize from one dataset to others. This paper presents a robust date estimation system based in a retrieval approach that generalizes well in front of heterogeneous collections. We use a ranking loss function named smooth-nDCG to train a Convolutional Neural Network that learns an ordination of documents for each problem. One of the main usages of the presented approach is as a tool for historical contextual retrieval. It means that scholars could perform comparative analysis of historical images from big datasets in terms of the period where they were produced. We provide experimental evaluation on different types of documents from real datasets of manuscript and newspaper images. |
| Address |
La Rochelle, France; May 22–25, 2022 |
| Corporate Author |
|
Thesis |
|
| Publisher |
|
Place of Publication |
|
Editor |
|
| Language |
|
Summary Language |
|
Original Title |
|
| Series Editor |
|
Series Title |
|
Abbreviated Series Title |
|
| Series Volume |
|
Series Issue |
|
Edition |
|
| ISSN |
|
ISBN |
|
Medium |
|
| Area |
|
Expedition |
|
Conference |
DAS |
| Notes |
DAG; 600.140; 600.121 |
Approved |
no |
| Call Number |
Admin @ si @ MGR2022 |
Serial |
3694 |
| Permanent link to this record |
| |
|
| |
| Author |
Minesh Mathew; Viraj Bagal; Ruben Tito; Dimosthenis Karatzas; Ernest Valveny; C.V. Jawahar |
| Title |
InfographicVQA |
Type |
Conference Article |
| Year |
2022 |
Publication |
Winter Conference on Applications of Computer Vision |
Abbreviated Journal |
|
| Volume |
|
Issue |
|
Pages |
1697-1706 |
| Keywords |
Document Analysis Datasets; Evaluation and Comparison of Vision Algorithms; Vision and Languages |
| Abstract |
Infographics communicate information using a combination of textual, graphical and visual elements. This work explores the automatic understanding of infographic images by using a Visual Question Answering technique. To this end, we present InfographicVQA, a new dataset comprising a diverse collection of infographics and question-answer annotations. The questions require methods that jointly reason over the document layout, textual content, graphical elements, and data visualizations. We curate the dataset with an emphasis on questions that require elementary reasoning and basic arithmetic skills. For VQA on the dataset, we evaluate two Transformer-based strong baselines. Both the baselines yield unsatisfactory results compared to near perfect human performance on the dataset. The results suggest that VQA on infographics--images that are designed to communicate information quickly and clearly to human brain--is ideal for benchmarking machine understanding of complex document images. The dataset is available for download at docvqa. org |
| Address |
Virtual; Waikoloa; Hawai; USA; January 2022 |
| Corporate Author |
|
Thesis |
|
| Publisher |
|
Place of Publication |
|
Editor |
|
| Language |
|
Summary Language |
|
Original Title |
|
| Series Editor |
|
Series Title |
|
Abbreviated Series Title |
|
| Series Volume |
|
Series Issue |
|
Edition |
|
| ISSN |
|
ISBN |
|
Medium |
|
| Area |
|
Expedition |
|
Conference |
WACV |
| Notes |
DAG; 600.155 |
Approved |
no |
| Call Number |
MBT2022 |
Serial |
3625 |
| Permanent link to this record |
| |
|
| |
| Author |
Ali Furkan Biten; Lluis Gomez; Dimosthenis Karatzas |
| Title |
Let there be a clock on the beach: Reducing Object Hallucination in Image Captioning |
Type |
Conference Article |
| Year |
2022 |
Publication |
Winter Conference on Applications of Computer Vision |
Abbreviated Journal |
|
| Volume |
|
Issue |
|
Pages |
1381-1390 |
| Keywords |
Measurement; Training; Visualization; Analytical models; Computer vision; Computational modeling; Training data |
| Abstract |
Explaining an image with missing or non-existent objects is known as object bias (hallucination) in image captioning. This behaviour is quite common in the state-of-the-art captioning models which is not desirable by humans. To decrease the object hallucination in captioning, we propose three simple yet efficient training augmentation method for sentences which requires no new training data or increase
in the model size. By extensive analysis, we show that the proposed methods can significantly diminish our models’ object bias on hallucination metrics. Moreover, we experimentally demonstrate that our methods decrease the dependency on the visual features. All of our code, configuration files and model weights are available online. |
| Address |
Virtual; Waikoloa; Hawai; USA; January 2022 |
| Corporate Author |
|
Thesis |
|
| Publisher |
|
Place of Publication |
|
Editor |
|
| Language |
|
Summary Language |
|
Original Title |
|
| Series Editor |
|
Series Title |
|
Abbreviated Series Title |
|
| Series Volume |
|
Series Issue |
|
Edition |
|
| ISSN |
|
ISBN |
|
Medium |
|
| Area |
|
Expedition |
|
Conference |
WACV |
| Notes |
DAG; 600.155; 302.105 |
Approved |
no |
| Call Number |
Admin @ si @ BGK2022 |
Serial |
3662 |
| Permanent link to this record |
| |
|
| |
| Author |
Ali Furkan Biten; Andres Mafla; Lluis Gomez; Dimosthenis Karatzas |
| Title |
Is An Image Worth Five Sentences? A New Look into Semantics for Image-Text Matching |
Type |
Conference Article |
| Year |
2022 |
Publication |
Winter Conference on Applications of Computer Vision |
Abbreviated Journal |
|
| Volume |
|
Issue |
|
Pages |
1391-1400 |
| Keywords |
Measurement; Training; Integrated circuits; Annotations; Semantics; Training data; Semisupervised learning |
| Abstract |
The task of image-text matching aims to map representations from different modalities into a common joint visual-textual embedding. However, the most widely used datasets for this task, MSCOCO and Flickr30K, are actually image captioning datasets that offer a very limited set of relationships between images and sentences in their ground-truth annotations. This limited ground truth information forces us to use evaluation metrics based on binary relevance: given a sentence query we consider only one image as relevant. However, many other relevant images or captions may be present in the dataset. In this work, we propose two metrics that evaluate the degree of semantic relevance of retrieved items, independently of their annotated binary relevance. Additionally, we incorporate a novel strategy that uses an image captioning metric, CIDEr, to define a Semantic Adaptive Margin (SAM) to be optimized in a standard triplet loss. By incorporating our formulation to existing models, a large improvement is obtained in scenarios where available training data is limited. We also demonstrate that the performance on the annotated image-caption pairs is maintained while improving on other non-annotated relevant items when employing the full training set. The code for our new metric can be found at github. com/furkanbiten/ncsmetric and the model implementation at github. com/andrespmd/semanticadaptive_margin. |
| Address |
Virtual; Waikoloa; Hawai; USA; January 2022 |
| Corporate Author |
|
Thesis |
|
| Publisher |
|
Place of Publication |
|
Editor |
|
| Language |
|
Summary Language |
|
Original Title |
|
| Series Editor |
|
Series Title |
|
Abbreviated Series Title |
|
| Series Volume |
|
Series Issue |
|
Edition |
|
| ISSN |
|
ISBN |
|
Medium |
|
| Area |
|
Expedition |
|
Conference |
WACV |
| Notes |
DAG; 600.155; 302.105; |
Approved |
no |
| Call Number |
Admin @ si @ BMG2022 |
Serial |
3663 |
| Permanent link to this record |
