| Home | [181–190] << 191 192 193 194 195 196 197 198 199 200 >> [201–210] |
|
| Records | |||||
|---|---|---|---|---|---|
Author  |
Bogdan Raducanu; D. Gatica-Perez | ||||
| Title | Inferring competitive role patterns in reality TV show through nonverbal analysis | Type | Journal Article | ||
| Year | 2012 | Publication | Multimedia Tools and Applications | Abbreviated Journal | MTAP |
| Volume | 56 | Issue | 1 | Pages | 207-226 |
| Keywords | |||||
| Abstract | This paper introduces a new facet of social media, namely that depicting social interaction. More concretely, we address this problem from the perspective of nonverbal behavior-based analysis of competitive meetings. For our study, we made use of “The Apprentice” reality TV show, which features a competition for a real, highly paid corporate job. Our analysis is centered around two tasks regarding a person's role in a meeting: predicting the person with the highest status, and predicting the fired candidates. We address this problem by adopting both supervised and unsupervised strategies. The current study was carried out using nonverbal audio cues. Our approach is based only on the nonverbal interaction dynamics during the meeting without relying on the spoken words. The analysis is based on two types of data: individual and relational measures. Results obtained from the analysis of a full season of the show are promising (up to 85.7% of accuracy in the first case and up to 92.8% in the second case). Our approach has been conveniently compared with the Influence Model, demonstrating its superiority. | ||||
| Address | |||||
| Corporate Author | Thesis | ||||
| Publisher | Elsevier | Place of Publication | Editor | ||
| Language | Summary Language | Original Title | |||
| Series Editor | Series Title | Abbreviated Series Title | |||
| Series Volume | Series Issue | Edition | |||
| ISSN | 1380-7501 | ISBN | Medium | ||
| Area | Expedition | Conference | |||
| Notes | OR;MV | Approved | no | ||
| Call Number | BCNPCL @ bcnpcl @ RaG2012 | Serial | 1360 | ||
| Permanent link to this record | |||||
| Author |
Bogdan Raducanu; Alireza Bosaghzadeh; Fadi Dornaika | ||||
| Title | Facial Expression Recognition based on Multi-view Observations with Application to Social Robotics | Type | Conference Article | ||
| Year | 2014 | Publication | 1st Workshop on Computer Vision for Affective Computing | Abbreviated Journal | |
| Volume | Issue | Pages | 1-8 | ||
| Keywords | |||||
| Abstract | Human-robot interaction is a hot topic nowadays in the social robotics community. One crucial aspect is represented by the affective communication which comes encoded through the facial expressions. In this paper, we propose a novel approach for facial expression recognition, which exploits an efficient and adaptive graph-based label propagation (semi-supervised mode) in a multi-observation framework. The facial features are extracted using an appearance-based 3D face tracker, view- and texture independent. Our method has been extensively tested on the CMU dataset, and has been conveniently compared with other methods for graph construction. With the proposed approach, we developed an application for an AIBO robot, in which it mirrors the recognized facial
expression. |
||||
| Address | Singapore; November 2014 | ||||
| Corporate Author | Thesis | ||||
| Publisher | Place of Publication | Editor | |||
| Language | Summary Language | Original Title | |||
| Series Editor | Series Title | Abbreviated Series Title | |||
| Series Volume | Series Issue | Edition | |||
| ISSN | ISBN | Medium | |||
| Area | Expedition | Conference | ACCV | ||
| Notes | LAMP; | Approved | no | ||
| Call Number | Admin @ si @ RBD2014 | Serial | 2599 | ||
| Permanent link to this record | |||||
| Author |
Bogdan Raducanu; Alireza Bosaghzadeh; Fadi Dornaika | ||||
| Title | Multi-observation Face Recognition in Videos based on Label Propagation | Type | Conference Article | ||
| Year | 2015 | Publication | 6th Workshop on Analysis and Modeling of Faces and Gestures AMFG2015 | Abbreviated Journal | |
| Volume | Issue | Pages | 10-17 | ||
| Keywords | |||||
| Abstract | In order to deal with the huge amount of content generated by social media, especially for indexing and retrieval purposes, the focus shifted from single object recognition to multi-observation object recognition. Of particular interest is the problem of face recognition (used as primary cue for persons’ identity assessment), since it is highly required by popular social media search engines like Facebook and Youtube. Recently, several approaches for graph-based label propagation were proposed. However, the associated graphs were constructed in an ad-hoc manner (e.g., using the KNN graph) that cannot cope properly with the rapid and frequent changes in data appearance, a phenomenon intrinsically related with video sequences. In this paper, we
propose a novel approach for efficient and adaptive graph construction, based on a two-phase scheme: (i) the first phase is used to adaptively find the neighbors of a sample and also to find the adequate weights for the minimization function of the second phase; (ii) in the second phase, the selected neighbors along with their corresponding weights are used to locally and collaboratively estimate the sparse affinity matrix weights. Experimental results performed on Honda Video Database (HVDB) and a subset of video sequences extracted from the popular TV-series ’Friends’ show a distinct advantage of the proposed method over the existing standard graph construction methods. |
||||
| Address | Boston; USA; June 2015 | ||||
| Corporate Author | Thesis | ||||
| Publisher | Place of Publication | Editor | |||
| Language | Summary Language | Original Title | |||
| Series Editor | Series Title | Abbreviated Series Title | |||
| Series Volume | Series Issue | Edition | |||
| ISSN | ISBN | Medium | |||
| Area | Expedition | Conference | CVPRW | ||
| Notes | LAMP; 600.068; 600.072; | Approved | no | ||
| Call Number | Admin @ si @ RBD2015 | Serial | 2627 | ||
| Permanent link to this record | |||||
| Author |
Bhaskar Chakraborty; Ognjen Rudovic; Jordi Gonzalez | ||||
| Title | View-Invariant Human-Body Detection with Extension to Human Action Recognition using Component-Wise HMM of Body Parts | Type | Conference Article | ||
| Year | 2008 | Publication | 8th IEEE International Conference on Automatic Face and Gesture Recognition | Abbreviated Journal | |
| Volume | Issue | Pages | |||
| Keywords | |||||
| Abstract | |||||
| Address | Amsterdam; The Netherlands | ||||
| Corporate Author | Thesis | ||||
| Publisher | Place of Publication | Editor | |||
| Language | Summary Language | Original Title | |||
| Series Editor | Series Title | Abbreviated Series Title | |||
| Series Volume | Series Issue | Edition | |||
| ISSN | ISBN | Medium | |||
| Area | Expedition | Conference | FG | ||
| Notes | ISE | Approved | no | ||
| Call Number | ISE @ ise @ CRG2008 | Serial | 1113 | ||
| Permanent link to this record | |||||
| Author |
Bhaskar Chakraborty; Michael Holte; Thomas B. Moeslund; Jordi Gonzalez; Xavier Roca | ||||
| Title | A Selective Spatio-Temporal Interest Point Detector for Human Action Recognition in Complex Scenes | Type | Conference Article | ||
| Year | 2011 | Publication | 13th IEEE International Conference on Computer Vision | Abbreviated Journal | |
| Volume | Issue | Pages | 1776-1783 | ||
| Keywords | |||||
| Abstract | Recent progress in the field of human action recognition points towards the use of Spatio-Temporal Interest Points (STIPs) for local descriptor-based recognition strategies. In this paper we present a new approach for STIP detection by applying surround suppression combined with local and temporal constraints. Our method is significantly different from existing STIP detectors and improves the performance by detecting more repeatable, stable and distinctive STIPs for human actors, while suppressing unwanted background STIPs. For action representation we use a bag-of-visual words (BoV) model of local N-jet features to build a vocabulary of visual-words. To this end, we introduce a novel vocabulary building strategy by combining spatial pyramid and vocabulary compression techniques, resulting in improved performance and efficiency. Action class specific Support Vector Machine (SVM) classifiers are trained for categorization of human actions. A comprehensive set of experiments on existing benchmark datasets, and more challenging datasets of complex scenes, validate our approach and show state-of-the-art performance. | ||||
| Address | Barcelona | ||||
| Corporate Author | Thesis | ||||
| Publisher | Place of Publication | Editor | |||
| Language | Summary Language | Original Title | |||
| Series Editor | Series Title | Abbreviated Series Title | |||
| Series Volume | Series Issue | Edition | |||
| ISSN | 1550-5499 | ISBN | 978-1-4577-1101-5 | Medium | |
| Area | Expedition | Conference | ICCV | ||
| Notes | ISE | Approved | no | ||
| Call Number | Admin @ si @ CHM2011 | Serial | 1811 | ||
| Permanent link to this record | |||||
| Author |
Bhaskar Chakraborty; Michael Holte; Thomas B. Moeslund; Jordi Gonzalez | ||||
| Title | Selective Spatio-Temporal Interest Points | Type | Journal Article | ||
| Year | 2012 | Publication | Computer Vision and Image Understanding | Abbreviated Journal | CVIU |
| Volume | 116 | Issue | 3 | Pages | 396-410 |
| Keywords | |||||
| Abstract | Recent progress in the field of human action recognition points towards the use of Spatio-TemporalInterestPoints (STIPs) for local descriptor-based recognition strategies. In this paper, we present a novel approach for robust and selective STIP detection, by applying surround suppression combined with local and temporal constraints. This new method is significantly different from existing STIP detection techniques and improves the performance by detecting more repeatable, stable and distinctive STIPs for human actors, while suppressing unwanted background STIPs. For action representation we use a bag-of-video words (BoV) model of local N-jet features to build a vocabulary of visual-words. To this end, we introduce a novel vocabulary building strategy by combining spatial pyramid and vocabulary compression techniques, resulting in improved performance and efficiency. Action class specific Support Vector Machine (SVM) classifiers are trained for categorization of human actions. A comprehensive set of experiments on popular benchmark datasets (KTH and Weizmann), more challenging datasets of complex scenes with background clutter and camera motion (CVC and CMU), movie and YouTube video clips (Hollywood 2 and YouTube), and complex scenes with multiple actors (MSR I and Multi-KTH), validates our approach and show state-of-the-art performance. Due to the unavailability of ground truth action annotation data for the Multi-KTH dataset, we introduce an actor specific spatio-temporal clustering of STIPs to address the problem of automatic action annotation of multiple simultaneous actors. Additionally, we perform cross-data action recognition by training on source datasets (KTH and Weizmann) and testing on completely different and more challenging target datasets (CVC, CMU, MSR I and Multi-KTH). This documents the robustness of our proposed approach in the realistic scenario, using separate training and test datasets, which in general has been a shortcoming in the performance evaluation of human action recognition techniques. | ||||
| Address | |||||
| Corporate Author | Thesis | ||||
| Publisher | Elsevier | Place of Publication | Editor | ||
| Language | Summary Language | Original Title | |||
| Series Editor | Series Title | Abbreviated Series Title | |||
| Series Volume | Series Issue | Edition | |||
| ISSN | 1077-3142 | ISBN | Medium | ||
| Area | Expedition | Conference | |||
| Notes | ISE | Approved | no | ||
| Call Number | Admin @ si @ CHM2012 | Serial | 1806 | ||
| Permanent link to this record | |||||
| Author |
Bhaskar Chakraborty; Marco Pedersoli; Jordi Gonzalez | ||||
| Title | View-Invariant Human Action Detection using Component-Wise HMM of Body Parts | Type | Book Chapter | ||
| Year | 2008 | Publication | Articulated Motion and Deformable Objects, 5th International Conference | Abbreviated Journal | |
| Volume | 5098 | Issue | Pages | 208–217 | |
| Keywords | |||||
| Abstract | |||||
| Address | Port d'Andratx (Mallorca) | ||||
| Corporate Author | Thesis | ||||
| Publisher | Place of Publication | Editor | |||
| Language | Summary Language | Original Title | |||
| Series Editor | Series Title | Abbreviated Series Title | LNCS | ||
| Series Volume | Series Issue | Edition | |||
| ISSN | ISBN | Medium | |||
| Area | Expedition | Conference | AMDO | ||
| Notes | ISE | Approved | no | ||
| Call Number | ISE @ ise @ CPG2008 | Serial | 975 | ||
| Permanent link to this record | |||||
| Author |
Bhaskar Chakraborty; Jordi Gonzalez; Xavier Roca | ||||
| Title | Large scale continuous visual event recognition using max-margin Hough transformation framework | Type | Journal Article | ||
| Year | 2013 | Publication | Computer Vision and Image Understanding | Abbreviated Journal | CVIU |
| Volume | 117 | Issue | 10 | Pages | 1356–1368 |
| Keywords | |||||
| Abstract | In this paper we propose a novel method for continuous visual event recognition (CVER) on a large scale video dataset using max-margin Hough transformation framework. Due to high scalability, diverse real environmental state and wide scene variability direct application of action recognition/detection methods such as spatio-temporal interest point (STIP)-local feature based technique, on the whole dataset is practically infeasible. To address this problem, we apply a motion region extraction technique which is based on motion segmentation and region clustering to identify possible candidate “event of interest” as a preprocessing step. On these candidate regions a STIP detector is applied and local motion features are computed. For activity representation we use generalized Hough transform framework where each feature point casts a weighted vote for possible activity class centre. A max-margin frame work is applied to learn the feature codebook weight. For activity detection, peaks in the Hough voting space are taken into account and initial event hypothesis is generated using the spatio-temporal information of the participating STIPs. For event recognition a verification Support Vector Machine is used. An extensive evaluation on benchmark large scale video surveillance dataset (VIRAT) and as well on a small scale benchmark dataset (MSR) shows that the proposed method is applicable on a wide range of continuous visual event recognition applications having extremely challenging conditions. | ||||
| Address | |||||
| Corporate Author | Thesis | ||||
| Publisher | Place of Publication | Editor | |||
| Language | Summary Language | Original Title | |||
| Series Editor | Series Title | Abbreviated Series Title | |||
| Series Volume | Series Issue | Edition | |||
| ISSN | 1077-3142 | ISBN | Medium | ||
| Area | Expedition | Conference | |||
| Notes | ISE | Approved | no | ||
| Call Number | Admin @ si @ CGR2013 | Serial | 2413 | ||
| Permanent link to this record | |||||
| Author |
Bhaskar Chakraborty; Andrew Bagdanov; Jordi Gonzalez; Xavier Roca | ||||
| Title | Human Action Recognition Using an Ensemble of Body-Part Detectors | Type | Journal Article | ||
| Year | 2013 | Publication | Expert Systems | Abbreviated Journal | EXSY |
| Volume | 30 | Issue | 2 | Pages | 101-114 |
| Keywords | Human action recognition;body-part detection;hidden Markov model | ||||
| Abstract | This paper describes an approach to human action recognition based on a probabilistic optimization model of body parts using hidden Markov model (HMM). Our method is able to distinguish between similar actions by only considering the body parts having major contribution to the actions, for example, legs for walking, jogging and running; arms for boxing, waving and clapping. We apply HMMs to model the stochastic movement of the body parts for action recognition. The HMM construction uses an ensemble of body-part detectors, followed by grouping of part detections, to perform human identification. Three example-based body-part detectors are trained to detect three components of the human body: the head, legs and arms. These detectors cope with viewpoint changes and self-occlusions through the use of ten sub-classifiers that detect body parts over a specific range of viewpoints. Each sub-classifier is a support vector machine trained on features selected for the discriminative power for each particular part/viewpoint combination. Grouping of these detections is performed using a simple geometric constraint model that yields a viewpoint-invariant human detector. We test our approach on three publicly available action datasets: the KTH dataset, Weizmann dataset and HumanEva dataset. Our results illustrate that with a simple and compact representation we can achieve robust recognition of human actions comparable to the most complex, state-of-the-art methods. | ||||
| Address | |||||
| Corporate Author | Thesis | ||||
| Publisher | Place of Publication | Editor | |||
| Language | Summary Language | Original Title | |||
| Series Editor | Series Title | Abbreviated Series Title | |||
| Series Volume | Series Issue | Edition | |||
| ISSN | ISBN | Medium | |||
| Area | Expedition | Conference | |||
| Notes | ISE | Approved | no | ||
| Call Number | Admin @ si @ CBG2013 | Serial | 1809 | ||
| Permanent link to this record | |||||
| Author |
Bhaskar Chakraborty; Andrew Bagdanov; Jordi Gonzalez | ||||
| Title | Towards Real-Time Human Action Recognition | Type | Conference Article | ||
| Year | 2009 | Publication | 4th Iberian Conference on Pattern Recognition and Image Analysis | Abbreviated Journal | |
| Volume | 5524 | Issue | Pages | ||
| Keywords | |||||
| Abstract | This work presents a novel approach to human detection based action-recognition in real-time. To realize this goal our method first detects humans in different poses using a correlation-based approach. Recognition of actions is done afterward based on the change of the angular values subtended by various body parts. Real-time human detection and action recognition are very challenging, and most state-of-the-art approaches employ complex feature extraction and classification techniques, which ultimately becomes a handicap for real-time recognition. Our correlation-based method, on the other hand, is computationally efficient and uses very simple gradient-based features. For action recognition angular features of body parts are extracted using a skeleton technique. Results for action recognition are comparable with the present state-of-the-art. | ||||
| Address | Póvoa de Varzim, Portugal | ||||
| Corporate Author | Thesis | ||||
| Publisher | Springer Berlin Heidelberg | Place of Publication | Editor | ||
| Language | Summary Language | Original Title | |||
| Series Editor | Series Title | Abbreviated Series Title | LNCS | ||
| Series Volume | Series Issue | Edition | |||
| ISSN | 0302-9743 | ISBN | 978-3-642-02171-8 | Medium | |
| Area | Expedition | Conference | IbPRIA | ||
| Notes | ISE | Approved | no | ||
| Call Number | DAG @ dag @ CBG2009 | Serial | 1215 | ||
| Permanent link to this record | |||||
| Author |
Bhaskar Chakraborty | ||||
| Title | View-Invariant Human-Body Detection with Extension to Human Action Recognition using Component Wise HMM of Body Parts | Type | Miscellaneous | ||
| Year | 2008 | Publication | CVC Technical Report #123 | Abbreviated Journal | |
| Volume | Issue | Pages | |||
| Keywords | |||||
| Abstract | |||||
| Address | Barcelona, Spain | ||||
| Corporate Author | Thesis | ||||
| Publisher | Place of Publication | Editor | |||
| Language | Summary Language | Original Title | |||
| Series Editor | Series Title | Abbreviated Series Title | |||
| Series Volume | Series Issue | Edition | |||
| ISSN | ISBN | Medium | |||
| Area | Expedition | Conference | |||
| Notes | ISE | Approved | no | ||
| Call Number | Admin @ si @ Cha2008 | Serial | 1149 | ||
| Permanent link to this record | |||||
| Author |
Bhaskar Chakraborty | ||||
| Title | Model free approach to human action recognition | Type | Book Whole | ||
| Year | 2012 | Publication | PhD Thesis, Universitat Autonoma de Barcelona-CVC | Abbreviated Journal | |
| Volume | Issue | Pages | |||
| Keywords | |||||
| Abstract | Automatic understanding of human activity and action is very important and challenging research area of Computer Vision with wide applications in video surveillance, motion analysis, virtual reality interfaces, video indexing, content based video retrieval, HCI and health care. This thesis presents a series of techniques to solve the problem of human action recognition in video. First approach towards this goal is based on a probabilistic optimization model of body parts using Hidden Markov Model. This strong model based approach is able to distinguish between similar actions by only considering the body parts having major contributions to the actions. In next approach, we apply a weak model based human detector and actions are represented by Bag-of-key poses model to capture the human pose changes during the actions. To tackle the problem of human action recognition in complex scenes, a selective spatio-temporal interest point (STIP) detector is proposed by using a mechanism similar to that of the non-classical receptive field inhibition that is exhibited by most oriented selective neuron in the primary visual cortex. An extension of the selective STIP detector is applied to multi-view action recognition system by introducing a novel 4D STIPs (3D space + time). Finally, we use our STIP detector on large scale continuous visual event recognition problem and propose a novel generalized max-margin Hough transformation framework for activity detection | ||||
| Address | |||||
| Corporate Author | Thesis | Ph.D. thesis | |||
| Publisher | Ediciones Graficas Rey | Place of Publication | Editor | Jordi Gonzalez;Xavier Roca | |
| Language | Summary Language | Original Title | |||
| Series Editor | Series Title | Abbreviated Series Title | |||
| Series Volume | Series Issue | Edition | |||
| ISSN | ISBN | Medium | |||
| Area | Expedition | Conference | |||
| Notes | ISE | Approved | no | ||
| Call Number | Admin @ si @ Cha2012 | Serial | 2207 | ||
| Permanent link to this record | |||||
| Author |
Bhalaji Nagarajan; Ricardo Marques; Marcos Mejia; Petia Radeva | ||||
| Title | Class-conditional Importance Weighting for Deep Learning with Noisy Labels | Type | Conference Article | ||
| Year | 2022 | Publication | 17th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications | Abbreviated Journal | |
| Volume | 5 | Issue | Pages | 679-686 | |
| Keywords | Noisy Labeling; Loss Correction; Class-conditional Importance Weighting; Learning with Noisy Labels | ||||
| Abstract | Large-scale accurate labels are very important to the Deep Neural Networks to train them and assure high performance. However, it is very expensive to create a clean dataset since usually it relies on human interaction. To this purpose, the labelling process is made cheap with a trade-off of having noisy labels. Learning with Noisy Labels is an active area of research being at the same time very challenging. The recent advances in Self-supervised learning and robust loss functions have helped in advancing noisy label research. In this paper, we propose a loss correction method that relies on dynamic weights computed based on the model training. We extend the existing Contrast to Divide algorithm coupled with DivideMix using a new class-conditional weighted scheme. We validate the method using the standard noise experiments and achieved encouraging results. | ||||
| Address | Virtual; February 2022 | ||||
| Corporate Author | Thesis | ||||
| Publisher | Place of Publication | Editor | |||
| Language | Summary Language | Original Title | |||
| Series Editor | Series Title | Abbreviated Series Title | |||
| Series Volume | Series Issue | Edition | |||
| ISSN | ISBN | Medium | |||
| Area | Expedition | Conference | VISAPP | ||
| Notes | MILAB; no menciona | Approved | no | ||
| Call Number | Admin @ si @ NMM2022 | Serial | 3798 | ||
| Permanent link to this record | |||||
| Author |
Bhalaji Nagarajan; Marc Bolaños; Eduardo Aguilar; Petia Radeva | ||||
| Title | Deep ensemble-based hard sample mining for food recognition | Type | Journal Article | ||
| Year | 2023 | Publication | Journal of Visual Communication and Image Representation | Abbreviated Journal | JVCIR |
| Volume | 95 | Issue | Pages | 103905 | |
| Keywords | |||||
| Abstract | Deep neural networks represent a compelling technique to tackle complex real-world problems, but are over-parameterized and often suffer from over- or under-confident estimates. Deep ensembles have shown better parameter estimations and often provide reliable uncertainty estimates that contribute to the robustness of the results. In this work, we propose a new metric to identify samples that are hard to classify. Our metric is defined as coincidence score for deep ensembles which measures the agreement of its individual models. The main hypothesis we rely on is that deep learning algorithms learn the low-loss samples better compared to large-loss samples. In order to compensate for this, we use controlled over-sampling on the identified ”hard” samples using proper data augmentation schemes to enable the models to learn those samples better. We validate the proposed metric using two public food datasets on different backbone architectures and show the improvements compared to the conventional deep neural network training using different performance metrics. | ||||
| Address | |||||
| Corporate Author | Thesis | ||||
| Publisher | Place of Publication | Editor | |||
| Language | Summary Language | Original Title | |||
| Series Editor | Series Title | Abbreviated Series Title | |||
| Series Volume | Series Issue | Edition | |||
| ISSN | ISBN | Medium | |||
| Area | Expedition | Conference | |||
| Notes | MILAB | Approved | no | ||
| Call Number | Admin @ si @ NBA2023 | Serial | 3844 | ||
| Permanent link to this record | |||||
| Author |
Benjia Zhou; Zhigang Chen; Albert Clapes; Jun Wan; Yanyan Liang; Sergio Escalera; Zhen Lei; Du Zhang | ||||
| Title | Gloss-free Sign Language Translation: Improving from Visual-Language Pretraining | Type | Conference Article | ||
| Year | 2023 | Publication | IEEE/CVF International Conference on Computer Vision (ICCV) Workshops | Abbreviated Journal | |
| Volume | Issue | Pages | |||
| Keywords | |||||
| Abstract | Sign Language Translation (SLT) is a challenging task due to its cross-domain nature, involving the translation of visual-gestural language to text. Many previous methods employ an intermediate representation, i.e., gloss sequences, to facilitate SLT, thus transforming it into a two-stage task of sign language recognition (SLR) followed by sign language translation (SLT). However, the scarcity of gloss-annotated sign language data, combined with the information bottleneck in the mid-level gloss representation, has hindered the further development of the SLT task. To address this challenge, we propose a novel Gloss-Free SLT based on Visual-Language Pretraining (GFSLT-VLP), which improves SLT by inheriting language-oriented prior knowledge from pre-trained models, without any gloss annotation assistance. Our approach involves two stages: (i) integrating Contrastive Language-Image Pre-training (CLIP) with masked self-supervised learning to create pre-tasks that bridge the semantic gap between visual and textual representations and restore masked sentences, and (ii) constructing an end-to-end architecture with an encoder-decoder-like structure that inherits the parameters of the pre-trained Visual Encoder and Text Decoder from the first stage. The seamless combination of these novel designs forms a robust sign language representation and significantly improves gloss-free sign language translation. In particular, we have achieved unprecedented improvements in terms of BLEU-4 score on the PHOENIX14T dataset (>+5) and the CSL-Daily dataset (>+3) compared to state-of-the-art gloss-free SLT methods. Furthermore, our approach also achieves competitive results on the PHOENIX14T dataset when compared with most of the gloss-based methods. | ||||
| Address | Vancouver; Canada; June 2023 | ||||
| Corporate Author | Thesis | ||||
| Publisher | Place of Publication | Editor | |||
| Language | Summary Language | Original Title | |||
| Series Editor | Series Title | Abbreviated Series Title | |||
| Series Volume | Series Issue | Edition | |||
| ISSN | ISBN | Medium | |||
| Area | Expedition | Conference | ICCVW | ||
| Notes | HUPBA; | Approved | no | ||
| Call Number | Admin @ si @ ZCC2023 | Serial | 3839 | ||
| Permanent link to this record | |||||