| |
Records |
Links |
|
Author |
Ruben Tito; Dimosthenis Karatzas; Ernest Valveny |
|
| |
Title  |
Hierarchical multimodal transformers for Multi-Page DocVQA |
Type |
Journal Article |
| |
Year |
2023 |
Publication |
Pattern Recognition |
Abbreviated Journal |
PR |
|
| |
Volume |
144 |
Issue |
|
Pages |
109834 |
|
| |
Keywords |
|
|
| |
Abstract |
Document Visual Question Answering (DocVQA) refers to the task of answering questions from document images. Existing work on DocVQA only considers single-page documents. However, in real scenarios documents are mostly composed of multiple pages that should be processed altogether. In this work we extend DocVQA to the multi-page scenario. For that, we first create a new dataset, MP-DocVQA, where questions are posed over multi-page documents instead of single pages. Second, we propose a new hierarchical method, Hi-VT5, based on the T5 architecture, that overcomes the limitations of current methods to process long multi-page documents. The proposed method is based on a hierarchical transformer architecture where the encoder summarizes the most relevant information of every page and then, the decoder takes this summarized information to generate the final answer. Through extensive experimentation, we demonstrate that our method is able, in a single stage, to answer the questions and provide the page that contains the relevant information to find the answer, which can be used as a kind of explainability measure. |
|
| |
Address |
|
|
| |
Corporate Author |
|
Thesis |
|
|
| |
Publisher |
|
Place of Publication |
|
Editor |
|
|
| |
Language |
|
Summary Language |
|
Original Title |
|
|
| |
Series Editor |
|
Series Title |
|
Abbreviated Series Title |
|
|
| |
Series Volume |
|
Series Issue |
|
Edition |
|
|
| |
ISSN |
ISSN 0031-3203 |
ISBN |
|
Medium |
|
|
| |
Area |
|
Expedition |
|
Conference |
|
|
| |
Notes |
DAG; 600.155; 600.121 |
Approved |
no |
|
| |
Call Number |
Admin @ si @ TKV2023 |
Serial |
3825 |
|
| Permanent link to this record |
| |
|
| |
|
Author |
Ruben Tito; Dimosthenis Karatzas; Ernest Valveny |
|
| |
Title |
Hierarchical multimodal transformers for Multipage DocVQA |
Type |
Journal Article |
| |
Year |
2023 |
Publication |
Pattern Recognition |
Abbreviated Journal |
PR |
|
| |
Volume |
144 |
Issue |
109834 |
Pages |
|
|
| |
Keywords |
|
|
| |
Abstract |
Existing work on DocVQA only considers single-page documents. However, in real applications documents are mostly composed of multiple pages that should be processed altogether. In this work, we propose a new multimodal hierarchical method Hi-VT5, that overcomes the limitations of current methods to process long multipage documents. In contrast to previous hierarchical methods that focus on different semantic granularity (He et al., 2021) or different subtasks (Zhou et al., 2022) used in image classification. Our method is a hierarchical transformer architecture where the encoder learns to summarize the most relevant information of every page and then, the decoder uses this summarized representation to generate the final answer, following a bottom-up approach. Moreover, due to the lack of multipage DocVQA datasets, we also introduce MP-DocVQA, an extension of SP-DocVQA where questions are posed over multipage documents instead of single pages. Through extensive experimentation, we demonstrate that Hi-VT5 is able, in a single stage, to answer the questions and provide the page that contains the answer, which can be used as a kind of explainability measure. |
|
| |
Address |
|
|
| |
Corporate Author |
|
Thesis |
|
|
| |
Publisher |
|
Place of Publication |
|
Editor |
|
|
| |
Language |
|
Summary Language |
|
Original Title |
|
|
| |
Series Editor |
|
Series Title |
|
Abbreviated Series Title |
|
|
| |
Series Volume |
|
Series Issue |
|
Edition |
|
|
| |
ISSN |
|
ISBN |
|
Medium |
|
|
| |
Area |
|
Expedition |
|
Conference |
|
|
| |
Notes |
DAG |
Approved |
no |
|
| |
Call Number |
Admin @ si @ TKV2023 |
Serial |
3836 |
|
| Permanent link to this record |
| |
|
| |
|
Author |
Anjan Dutta; Pau Riba; Josep Llados; Alicia Fornes |
|
| |
Title |
Hierarchical Stochastic Graphlet Embedding for Graph-based Pattern Recognition |
Type |
Journal Article |
| |
Year |
2020 |
Publication |
Neural Computing and Applications |
Abbreviated Journal |
NEUCOMA |
|
| |
Volume |
32 |
Issue |
|
Pages |
11579–11596 |
|
| |
Keywords |
|
|
| |
Abstract |
Despite being very successful within the pattern recognition and machine learning community, graph-based methods are often unusable because of the lack of mathematical operations defined in graph domain. Graph embedding, which maps graphs to a vectorial space, has been proposed as a way to tackle these difficulties enabling the use of standard machine learning techniques. However, it is well known that graph embedding functions usually suffer from the loss of structural information. In this paper, we consider the hierarchical structure of a graph as a way to mitigate this loss of information. The hierarchical structure is constructed by topologically clustering the graph nodes and considering each cluster as a node in the upper hierarchical level. Once this hierarchical structure is constructed, we consider several configurations to define the mapping into a vector space given a classical graph embedding, in particular, we propose to make use of the stochastic graphlet embedding (SGE). Broadly speaking, SGE produces a distribution of uniformly sampled low-to-high-order graphlets as a way to embed graphs into the vector space. In what follows, the coarse-to-fine structure of a graph hierarchy and the statistics fetched by the SGE complements each other and includes important structural information with varied contexts. Altogether, these two techniques substantially cope with the usual information loss involved in graph embedding techniques, obtaining a more robust graph representation. This fact has been corroborated through a detailed experimental evaluation on various benchmark graph datasets, where we outperform the state-of-the-art methods. |
|
| |
Address |
|
|
| |
Corporate Author |
|
Thesis |
|
|
| |
Publisher |
|
Place of Publication |
|
Editor |
|
|
| |
Language |
|
Summary Language |
|
Original Title |
|
|
| |
Series Editor |
|
Series Title |
|
Abbreviated Series Title |
|
|
| |
Series Volume |
|
Series Issue |
|
Edition |
|
|
| |
ISSN |
|
ISBN |
|
Medium |
|
|
| |
Area |
|
Expedition |
|
Conference |
|
|
| |
Notes |
DAG; 600.140; 600.121; 600.141 |
Approved |
no |
|
| |
Call Number |
Admin @ si @ DRL2020 |
Serial |
3348 |
|
| Permanent link to this record |
| |
|
| |
|
Author |
Lluis Gomez; Anguelos Nicolaou; Dimosthenis Karatzas |
|
| |
Title |
Improving patch‐based scene text script identification with ensembles of conjoined networks |
Type |
Journal Article |
| |
Year |
2017 |
Publication |
Pattern Recognition |
Abbreviated Journal |
PR |
|
| |
Volume |
67 |
Issue |
|
Pages |
85-96 |
|
| |
Keywords |
|
|
| |
Abstract |
|
|
| |
Address |
|
|
| |
Corporate Author |
|
Thesis |
|
|
| |
Publisher |
|
Place of Publication |
|
Editor |
|
|
| |
Language |
|
Summary Language |
|
Original Title |
|
|
| |
Series Editor |
|
Series Title |
|
Abbreviated Series Title |
|
|
| |
Series Volume |
|
Series Issue |
|
Edition |
|
|
| |
ISSN |
|
ISBN |
|
Medium |
|
|
| |
Area |
|
Expedition |
|
Conference |
|
|
| |
Notes |
DAG; 600.084; 600.121; 600.129 |
Approved |
no |
|
| |
Call Number |
Admin @ si @ GNK2017 |
Serial |
2887 |
|
| Permanent link to this record |
| |
|
| |
|
Author |
Lasse Martensson; Ekta Vats; Anders Hast; Alicia Fornes |
|
| |
Title |
In Search of the Scribe: Letter Spotting as a Tool for Identifying Scribes in Large Handwritten Text Corpora |
Type |
Journal |
| |
Year |
2019 |
Publication |
Journal for Information Technology Studies as a Human Science |
Abbreviated Journal |
HUMAN IT |
|
| |
Volume |
14 |
Issue |
2 |
Pages |
95-120 |
|
| |
Keywords |
Scribal attribution/ writer identification; digital palaeography; word spotting; mediaeval charters; mediaeval manuscripts |
|
| |
Abstract |
In this article, a form of the so-called word spotting-method is used on a large set of handwritten documents in order to identify those that contain script of similar execution. The point of departure for the investigation is the mediaeval Swedish manuscript Cod. Holm. D 3. The main scribe of this manuscript has yet not been identified in other documents. The current attempt aims at localising other documents that display a large degree of similarity in the characteristics of the script, these being possible candidates for being executed by the same hand. For this purpose, the method of word spotting has been employed, focusing on individual letters, and therefore the process is referred to as letter spotting in the article. In this process, a set of ‘g’:s, ‘h’:s and ‘k’:s have been selected as templates, and then a search has been made for close matches among the mediaeval Swedish charters. The search resulted in a number of charters that displayed great similarities with the manuscript D 3. The used letter spotting method thus proofed to be a very efficient sorting tool localising similar script samples. |
|
| |
Address |
|
|
| |
Corporate Author |
|
Thesis |
|
|
| |
Publisher |
|
Place of Publication |
|
Editor |
|
|
| |
Language |
|
Summary Language |
|
Original Title |
|
|
| |
Series Editor |
|
Series Title |
|
Abbreviated Series Title |
|
|
| |
Series Volume |
|
Series Issue |
|
Edition |
|
|
| |
ISSN |
|
ISBN |
|
Medium |
|
|
| |
Area |
|
Expedition |
|
Conference |
|
|
| |
Notes |
DAG; 600.097; 600.140; 600.121 |
Approved |
no |
|
| |
Call Number |
Admin @ si @ MVH2019 |
Serial |
3234 |
|
| Permanent link to this record |
