| Home | << 1 2 3 4 5 6 7 8 9 10 >> |
|
| Records | |||||
|---|---|---|---|---|---|
| Author | Ali Furkan Biten | ||||
| Title | A Bitter-Sweet Symphony on Vision and Language: Bias and World Knowledge | Type | Book Whole | ||
| Year | 2022 | Publication | PhD Thesis, Universitat Autonoma de Barcelona-CVC | Abbreviated Journal | |
| Volume | Issue | Pages | |||
| Keywords | |||||
| Abstract | Vision and Language are broadly regarded as cornerstones of intelligence. Even though language and vision have different aims – language having the purpose of communication, transmission of information and vision having the purpose of constructing mental representations around us to navigate and interact with objects – they cooperate and depend on one another in many tasks we perform effortlessly. This reliance is actively being studied in various Computer Vision tasks, e.g. image captioning, visual question answering, image-sentence retrieval, phrase grounding, just to name a few. All of these tasks share the inherent difficulty of the aligning the two modalities, while being robust to language
priors and various biases existing in the datasets. One of the ultimate goal for vision and language research is to be able to inject world knowledge while getting rid of the biases that come with the datasets. In this thesis, we mainly focus on two vision and language tasks, namely Image Captioning and Scene-Text Visual Question Answering (STVQA). In both domains, we start by defining a new task that requires the utilization of world knowledge and in both tasks, we find that the models commonly employed are prone to biases that exist in the data. Concretely, we introduce new tasks and discover several problems that impede performance at each level and provide remedies or possible solutions in each chapter: i) We define a new task to move beyond Image Captioning to Image Interpretation that can utilize Named Entities in the form of world knowledge. ii) We study the object hallucination problem in classic Image Captioning systems and develop an architecture-agnostic solution. iii) We define a sub-task of Visual Question Answering that requires reading the text in the image (STVQA), where we highlight the limitations of current models. iv) We propose an architecture for the STVQA task that can point to the answer in the image and show how to combine it with classic VQA models. v) We show how far language can get us in STVQA and discover yet another bias which causes the models to disregard the image while doing Visual Question Answering. |
||||
| Address | |||||
| Corporate Author | Thesis | Ph.D. thesis | |||
| Publisher | IMPRIMA | Place of Publication | Editor | Dimosthenis Karatzas;Lluis Gomez | |
| Language | Summary Language | Original Title | |||
| Series Editor | Series Title | Abbreviated Series Title | |||
| Series Volume | Series Issue | Edition | |||
| ISSN | ISBN  |
978-84-124793-5-5 | Medium | ||
| Area | Expedition | Conference | |||
| Notes | DAG | Approved | no | ||
| Call Number | Admin @ si @ Bit2022 | Serial | 3755 | ||
| Permanent link to this record | |||||
| Author | Andres Mafla | ||||
| Title | Leveraging Scene Text Information for Image Interpretation | Type | Book Whole | ||
| Year | 2022 | Publication | PhD Thesis, Universitat Autonoma de Barcelona-CVC | Abbreviated Journal | |
| Volume | Issue | Pages | |||
| Keywords | |||||
| Abstract | Until recently, most computer vision models remained illiterate, largely ignoring the semantically rich and explicit information contained in scene text. Recent progress in scene text detection and recognition has recently allowed exploring its role in a diverse set of open computer vision problems, e.g. image classification, image-text retrieval, image captioning, and visual question answering to name a few. The explicit semantics of scene text closely requires specific modeling similar to language. However, scene text is a particular signal that has to be interpreted according to a comprehensive perspective that encapsulates all the visual cues in an image. Incorporating this information is a straightforward task for humans, but if we are unfamiliar with a language or scripture, achieving a complete world understanding is impossible (e.a. visiting a foreign country with a different alphabet). Despite the importance of scene text, modeling it requires considering the several ways in which scene text interacts with an image, processing and fusing an additional modality. In this thesis, we mainly focus
on two tasks, scene text-based fine-grained image classification, and cross-modal retrieval. In both studied tasks we identify existing limitations in current approaches and propose plausible solutions. Concretely, in each chapter: i) We define a compact way to embed scene text that generalizes to unseen words at training time while performing in real-time. ii) We incorporate the previously learned scene text embedding to create an image-level descriptor that overcomes optical character recognition (OCR) errors which is well-suited to the fine-grained image classification task. iii) We design a region-level reasoning network that learns the interaction through semantics among salient visual regions and scene text instances. iv) We employ scene text information in image-text matching and introduce the Scene Text Aware Cross-Modal retrieval StacMR task. We gather a dataset that incorporates scene text and design a model suited for the newly studied modality. v) We identify the drawbacks of current retrieval metrics in cross-modal retrieval. An image captioning metric is proposed as a way of better evaluating semantics in retrieved results. Ample experimentation shows that incorporating such semantics into a model yields better semantic results while requiring significantly less data to converge. |
||||
| Address | |||||
| Corporate Author | Thesis | Ph.D. thesis | |||
| Publisher | IMPRIMA | Place of Publication | Editor | Dimosthenis Karatzas;Lluis Gomez | |
| Language | Summary Language | Original Title | |||
| Series Editor | Series Title | Abbreviated Series Title | |||
| Series Volume | Series Issue | Edition | |||
| ISSN | ISBN |
978-84-124793-6-2 | Medium | ||
| Area | Expedition | Conference | |||
| Notes | DAG | Approved | no | ||
| Call Number | Admin @ si @ Maf2022 | Serial | 3756 | ||
| Permanent link to this record | |||||
| Author | Arnau Baro | ||||
| Title | Reading Music Systems: From Deep Optical Music Recognition to Contextual Methods | Type | Book Whole | ||
| Year | 2022 | Publication | PhD Thesis, Universitat Autonoma de Barcelona-CVC | Abbreviated Journal | |
| Volume | Issue | Pages | |||
| Keywords | |||||
| Abstract | The transcription of sheet music into some machine-readable format can be carried out manually. However, the complexity of music notation inevitably leads to burdensome software for music score editing, which makes the whole process
very time-consuming and prone to errors. Consequently, automatic transcription systems for musical documents represent interesting tools. Document analysis is the subject that deals with the extraction and processing of documents through image and pattern recognition. It is a branch of computer vision. Taking music scores as source, the field devoted to address this task is known as Optical Music Recognition (OMR). Typically, an OMR system takes an image of a music score and automatically extracts its content into some symbolic structure such as MEI or MusicXML. In this dissertation, we have investigated different methods for recognizing a single staff section (e.g. scores for violin, flute, etc.), much in the same way as most text recognition research focuses on recognizing words appearing in a given line image. These methods are based in two different methodologies. On the one hand, we present two methods based on Recurrent Neural Networks, in particular, the Long Short-Term Memory Neural Network. On the other hand, a method based on Sequence to Sequence models is detailed. Music context is needed to improve the OMR results, just like language models and dictionaries help in handwriting recognition. For example, syntactical rules and grammars could be easily defined to cope with the ambiguities in the rhythm. In music theory, for example, the time signature defines the amount of beats per bar unit. Thus, in the second part of this dissertation, different methodologies have been investigated to improve the OMR recognition. We have explored three different methods: (a) a graphic tree-structure representation, Dendrograms, that joins, at each level, its primitives following a set of rules, (b) the incorporation of Language Models to model the probability of a sequence of tokens, and (c) graph neural networks to analyze the music scores to avoid meaningless relationships between music primitives. Finally, to train all these methodologies, and given the method-specificity of the datasets in the literature, we have created four different music datasets. Two of them are synthetic with a modern or old handwritten appearance, whereas the other two are real handwritten scores, being one of them modern and the other old. |
||||
| Address | |||||
| Corporate Author | Thesis | Ph.D. thesis | |||
| Publisher | IMPRIMA | Place of Publication | Editor | Alicia Fornes | |
| Language | Summary Language | Original Title | |||
| Series Editor | Series Title | Abbreviated Series Title | |||
| Series Volume | Series Issue | Edition | |||
| ISSN | ISBN |
978-84-124793-8-6 | Medium | ||
| Area | Expedition | Conference | |||
| Notes | DAG; | Approved | no | ||
| Call Number | Admin @ si @ Bar2022 | Serial | 3754 | ||
| Permanent link to this record | |||||
| Author | Mohamed Ali Souibgui | ||||
| Title | Document Image Enhancement and Recognition in Low Resource Scenarios: Application to Ciphers and Handwritten Text | Type | Book Whole | ||
| Year | 2022 | Publication | PhD Thesis, Universitat Autonoma de Barcelona-CVC | Abbreviated Journal | |
| Volume | Issue | Pages | |||
| Keywords | |||||
| Abstract | In this thesis, we propose different contributions with the goal of enhancing and recognizing historical handwritten document images, especially the ones with rare scripts, such as cipher documents.
In the first part, some effective end-to-end models for Document Image Enhancement (DIE) using deep learning models were presented. First, Generative Adversarial Networks (cGAN) for different tasks (document clean-up, binarization, deblurring, and watermark removal) were explored. Next, we further improve the results by recovering the degraded document images into a clean and readable form by integrating a text recognizer into the cGAN model to promote the generated document image to be more readable. Afterward, we present a new encoder-decoder architecture based on vision transformers to enhance both machine-printed and handwritten document images, in an end-to-end fashion. The second part of the thesis addresses Handwritten Text Recognition (HTR) in low resource scenarios, i.e. when only few labeled training data is available. We propose novel methods for recognizing ciphers with rare scripts. First, a few-shot object detection based method was proposed. Then, we incorporate a progressive learning strategy that automatically assignspseudo-labels to a set of unlabeled data to reduce the human labor of annotating few pages while maintaining the good performance of the model. Secondly, a data generation technique based on Bayesian Program Learning (BPL) is proposed to overcome the lack of data in such rare scripts. Thirdly, we propose a Text-Degradation Invariant Auto Encoder (Text-DIAE). This latter self-supervised model is designed to tackle two tasks, text recognition and document image enhancement. The proposed model does not exhibit limitations of previous state-of-the-art methods based on contrastive losses, while at the same time, it requires substantially fewer data samples to converge. In the third part of the thesis, we analyze, from the user perspective, the usage of HTR systems in low resource scenarios. This contrasts with the usual research on HTR, which often focuses on technical aspects only and rarely devotes efforts on implementing software tools for scholars in Humanities. |
||||
| Address | |||||
| Corporate Author | Thesis | Ph.D. thesis | |||
| Publisher | IMPRIMA | Place of Publication | Editor | Alicia Fornes;Yousri Kessentini | |
| Language | Summary Language | Original Title | |||
| Series Editor | Series Title | Abbreviated Series Title | |||
| Series Volume | Series Issue | Edition | |||
| ISSN | ISBN |
978-84-124793-8-6 | Medium | ||
| Area | Expedition | Conference | |||
| Notes | DAG | Approved | no | ||
| Call Number | Admin @ si @ Sou2022 | Serial | 3757 | ||
| Permanent link to this record | |||||
| Author | Jorge Charco; Angel Sappa; Boris X. Vintimilla | ||||
| Title | Human Pose Estimation through a Novel Multi-view Scheme | Type | Conference Article | ||
| Year | 2022 | Publication | 17th International Conference on Computer Vision Theory and Applications (VISAPP 2022) | Abbreviated Journal | |
| Volume | 5 | Issue | Pages | 855-862 | |
| Keywords | Multi-view Scheme; Human Pose Estimation; Relative Camera Pose; Monocular Approach | ||||
| Abstract | This paper presents a multi-view scheme to tackle the challenging problem of the self-occlusion in human pose estimation problem. The proposed approach first obtains the human body joints of a set of images, which are captured from different views at the same time. Then, it enhances the obtained joints by using a
multi-view scheme. Basically, the joints from a given view are used to enhance poorly estimated joints from another view, especially intended to tackle the self occlusions cases. A network architecture initially proposed for the monocular case is adapted to be used in the proposed multi-view scheme. Experimental results and comparisons with the state-of-the-art approaches on Human3.6m dataset are presented showing improvements in the accuracy of body joints estimations. |
||||
| Address | On line; Feb 6, 2022 – Feb 8, 2022 | ||||
| Corporate Author | Thesis | ||||
| Publisher | Place of Publication | Editor | |||
| Language | Summary Language | Original Title | |||
| Series Editor | Series Title | Abbreviated Series Title | |||
| Series Volume | Series Issue | Edition | |||
| ISSN | 2184-4321 | ISBN |
978-989-758-555-5 | Medium | |
| Area | Expedition | Conference | VISAPP | ||
| Notes | MSIAU; 600.160 | Approved | no | ||
| Call Number | Admin @ si @ CSV2022 | Serial | 3689 | ||
| Permanent link to this record | |||||