|
Jose L. Gomez, Gabriel Villalonga and Antonio Lopez. 2021. Co-Training for Deep Object Detection: Comparing Single-Modal and Multi-Modal Approaches. SENS, 21(9), 3185.
Abstract: Top-performing computer vision models are powered by convolutional neural networks (CNNs). Training an accurate CNN highly depends on both the raw sensor data and their associated ground truth (GT). Collecting such GT is usually done through human labeling, which is time-consuming and does not scale as we wish. This data-labeling bottleneck may be intensified due to domain shifts among image sensors, which could force per-sensor data labeling. In this paper, we focus on the use of co-training, a semi-supervised learning (SSL) method, for obtaining self-labeled object bounding boxes (BBs), i.e., the GT to train deep object detectors. In particular, we assess the goodness of multi-modal co-training by relying on two different views of an image, namely, appearance (RGB) and estimated depth (D). Moreover, we compare appearance-based single-modal co-training with multi-modal. Our results suggest that in a standard SSL setting (no domain shift, a few human-labeled data) and under virtual-to-real domain shift (many virtual-world labeled data, no human-labeled data) multi-modal co-training outperforms single-modal. In the latter case, by performing GAN-based domain translation both co-training modalities are on par, at least when using an off-the-shelf depth estimation model not specifically trained on the translated images.
Keywords: co-training; multi-modality; vision-based object detection; ADAS; self-driving
|
|
|
Hannes Mueller, Andre Groeger, Jonathan Hersh, Andrea Matranga and Joan Serrat. 2021. Monitoring war destruction from space using machine learning. PNAS, 118(23), e2025400118.
Abstract: Existing data on building destruction in conflict zones rely on eyewitness reports or manual detection, which makes it generally scarce, incomplete, and potentially biased. This lack of reliable data imposes severe limitations for media reporting, humanitarian relief efforts, human-rights monitoring, reconstruction initiatives, and academic studies of violent conflict. This article introduces an automated method of measuring destruction in high-resolution satellite images using deep-learning techniques combined with label augmentation and spatial and temporal smoothing, which exploit the underlying spatial and temporal structure of destruction. As a proof of concept, we apply this method to the Syrian civil war and reconstruct the evolution of damage in major cities across the country. Our approach allows generating destruction data with unprecedented scope, resolution, and frequency—and makes use of the ever-higher frequency at which satellite imagery becomes available.
|
|
|
Idoia Ruiz and Joan Serrat. 2022. Hierarchical Novelty Detection for Traffic Sign Recognition. SENS, 22(12), 4389.
Abstract: Recent works have made significant progress in novelty detection, i.e., the problem of detecting samples of novel classes, never seen during training, while classifying those that belong to known classes. However, the only information this task provides about novel samples is that they are unknown. In this work, we leverage hierarchical taxonomies of classes to provide informative outputs for samples of novel classes. We predict their closest class in the taxonomy, i.e., its parent class. We address this problem, known as hierarchical novelty detection, by proposing a novel loss, namely Hierarchical Cosine Loss that is designed to learn class prototypes along with an embedding of discriminative features consistent with the taxonomy. We apply it to traffic sign recognition, where we predict the parent class semantics for new types of traffic signs. Our model beats state-of-the art approaches on two large scale traffic sign benchmarks, Mapillary Traffic Sign Dataset (MTSD) and Tsinghua-Tencent 100K (TT100K), and performs similarly on natural images benchmarks (AWA2, CUB). For TT100K and MTSD, our approach is able to detect novel samples at the correct nodes of the hierarchy with 81% and 36% of accuracy, respectively, at 80% known class accuracy.
Keywords: Novelty detection; hierarchical classification; deep learning; traffic sign recognition; autonomous driving; computer vision
|
|
|
Jose Luis Gomez, Gabriel Villalonga and Antonio Lopez. 2023. Co-Training for Unsupervised Domain Adaptation of Semantic Segmentation Models. SENS, 23(2), 621.
Abstract: Semantic image segmentation is a central and challenging task in autonomous driving, addressed by training deep models. Since this training draws to a curse of human-based image labeling, using synthetic images with automatically generated labels together with unlabeled real-world images is a promising alternative. This implies to address an unsupervised domain adaptation (UDA) problem. In this paper, we propose a new co-training procedure for synth-to-real UDA of semantic
segmentation models. It consists of a self-training stage, which provides two domain-adapted models, and a model collaboration loop for the mutual improvement of these two models. These models are then used to provide the final semantic segmentation labels (pseudo-labels) for the real-world images. The overall
procedure treats the deep models as black boxes and drives their collaboration at the level of pseudo-labeled target images, i.e., neither modifying loss functions is required, nor explicit feature alignment. We test our proposal on standard synthetic and real-world datasets for on-board semantic segmentation. Our
procedure shows improvements ranging from ∼13 to ∼26 mIoU points over baselines, so establishing new state-of-the-art results.
Keywords: Domain adaptation; semi-supervised learning; Semantic segmentation; Autonomous driving
|
|
|
Gemma Rotger, Francesc Moreno-Noguer, Felipe Lumbreras and Antonio Agudo. 2019. Detailed 3D face reconstruction from a single RGB image.
Abstract: This paper introduces a method to obtain a detailed 3D reconstruction of facial skin from a single RGB image.
To this end, we propose the exclusive use of an input image without requiring any information about the observed material nor training data to model the wrinkle properties. They are detected and characterized directly from the image via a simple and effective parametric model, determining several features such as location, orientation, width, and height. With these ingredients, we propose to minimize a photometric error to retrieve the final detailed 3D map, which is initialized by current techniques based on deep learning. In contrast with other approaches, we only require estimating a depth parameter, making our approach fast and intuitive. Extensive experimental evaluation is presented in a wide variety of synthetic and real images, including different skin properties and facial
expressions. In all cases, our method outperforms the current approaches regarding 3D reconstruction accuracy, providing striking results for both large and fine wrinkles.
Keywords: 3D Wrinkle Reconstruction; Face Analysis, Optimization.
|
|
|
Aura Hernandez-Sabate, Jose Elias Yauri, Pau Folch, Miquel Angel Piera and Debora Gil. 2022. Recognition of the Mental Workloads of Pilots in the Cockpit Using EEG Signals. APPLSCI, 12(5), 2298.
Abstract: The commercial flightdeck is a naturally multi-tasking work environment, one in which interruptions are frequent come in various forms, contributing in many cases to aviation incident reports. Automatic characterization of pilots’ workloads is essential to preventing these kind of incidents. In addition, minimizing the physiological sensor network as much as possible remains both a challenge and a requirement. Electroencephalogram (EEG) signals have shown high correlations with specific cognitive and mental states, such as workload. However, there is not enough evidence in the literature to validate how well models generalize in cases of new subjects performing tasks with workloads similar to the ones included during the model’s training. In this paper, we propose a convolutional neural network to classify EEG features across different mental workloads in a continuous performance task test that partly measures working memory and working memory capacity. Our model is valid at the general population level and it is able to transfer task learning to pilot mental workload recognition in a simulated operational environment.
Keywords: Cognitive states; Mental workload; EEG analysis; Neural networks; Multimodal data fusion
|
|
|
M. Altillawi, S. Li, S.M. Prakhya, Z. Liu and Joan Serrat. 2024. Implicit Learning of Scene Geometry From Poses for Global Localization. ROBOTAUTOMLET, 9(2), 955–962.
Abstract: Global visual localization estimates the absolute pose of a camera using a single image, in a previously mapped area. Obtaining the pose from a single image enables many robotics and augmented/virtual reality applications. Inspired by latest advances in deep learning, many existing approaches directly learn and regress 6 DoF pose from an input image. However, these methods do not fully utilize the underlying scene geometry for pose regression. The challenge in monocular relocalization is the minimal availability of supervised training data, which is just the corresponding 6 DoF poses of the images. In this letter, we propose to utilize these minimal available labels (i.e., poses) to learn the underlying 3D geometry of the scene and use the geometry to estimate the 6 DoF camera pose. We present a learning method that uses these pose labels and rigid alignment to learn two 3D geometric representations ( X, Y, Z coordinates ) of the scene, one in camera coordinate frame and the other in global coordinate frame. Given a single image, it estimates these two 3D scene representations, which are then aligned to estimate a pose that matches the pose label. This formulation allows for the active inclusion of additional learning constraints to minimize 3D alignment errors between the two 3D scene representations, and 2D re-projection errors between the 3D global scene representation and 2D image pixels, resulting in improved localization accuracy. During inference, our model estimates the 3D scene geometry in camera and global frames and aligns them rigidly to obtain pose in real-time. We evaluate our work on three common visual localization datasets, conduct ablation studies, and show that our method exceeds state-of-the-art regression methods' pose accuracy on all datasets.
Keywords: Localization; Localization and mapping; Deep learning for visual perception; Visual learning
|
|
|
Enric Marti, Carme Julia and Debora Gil. 2006. A PBL Experience in the Teaching of Computer Graphics. CGF, 25(1), 95–103.
Abstract: Project-Based Learning (PBL) is an educational strategy to improve student’s learning capability that, in recent years, has had a progressive acceptance in undergraduate studies. This methodology is based on solving a problem or project in a student working group. In this way, PBL focuses on learning the necessary tools to correctly find a solution to given problems. Since the learning initiative is transferred to the student, the PBL method promotes students own abilities. This allows a better assessment of the true workload that carries out the student in the subject. It follows that the methodology conforms to the guidelines of the Bologna document, which quantifies the student workload in a subject by means of the European credit transfer system (ECTS). PBL is currently applied in undergraduate studies needing strong practical training such as medicine, nursing or law sciences. Although this is also the case in engineering studies, amazingly, few experiences have been reported. In this paper we propose to use PBL in the educational organization of the Computer Graphics subjects in the Computer Science degree. Our PBL project focuses in the development of a C++ graphical environment based on the OpenGL libraries for visualization and handling of different graphical objects. The starting point is a basic skeleton that already includes lighting functions, perspective projection with mouse interaction to change the point of view and three predefined objects. Students have to complete this skeleton by adding their own functions to solve the project. A total number of 10 projects have been proposed and successfully solved. The exercises range from human face rendering to articulated objects, such as robot arms or puppets. In the present paper we extensively report the statement and educational objectives for two of the projects: solar system visualization and a chess game. We report our earlier educational experience based on the standard classroom theoretical, problem and practice sessions and the reasons that motivated searching for other learning methods. We have mainly chosen PBL because it improves the student learning initiative. We have applied the PBL educational model since the beginning of the second semester. The student’s feedback increases in his interest for the subject. We present a comparative study of the teachers’ and students’ workload between PBL and the classic teaching approach, which suggests that the workload increase in PBL is not as high as it seems.
|
|
|
Joan Serrat, Felipe Lumbreras and Antonio Lopez. 2013. Cost estimation of custom hoses from STL files and CAD drawings. COMPUTIND, 64(3), 299–309.
Abstract: We present a method for the cost estimation of custom hoses from CAD models. They can come in two formats, which are easy to generate: a STL file or the image of a CAD drawing showing several orthogonal projections. The challenges in either cases are, first, to obtain from them a high level 3D description of the shape, and second, to learn a regression function for the prediction of the manufacturing time, based on geometric features of the reconstructed shape. The chosen description is the 3D line along the medial axis of the tube and the diameter of the circular sections along it. In order to extract it from STL files, we have adapted RANSAC, a robust parametric fitting algorithm. As for CAD drawing images, we propose a new technique for 3D reconstruction from data entered on any number of orthogonal projections. The regression function is a Gaussian process, which does not constrain the function to adopt any specific form and is governed by just two parameters. We assess the accuracy of the manufacturing time estimation by k-fold cross validation on 171 STL file models for which the time is provided by an expert. The results show the feasibility of the method, whereby the relative error for 80% of the testing samples is below 15%.
Keywords: On-line quotation; STL format; Regression; Gaussian process
|
|
|
Miguel Oliveira, Victor Santos, Angel Sappa, P. Dias and A. Moreira. 2016. Incremental Scenario Representations for Autonomous Driving using Geometric Polygonal Primitives. RAS, 83, 312–325.
Abstract: When an autonomous vehicle is traveling through some scenario it receives a continuous stream of sensor data. This sensor data arrives in an asynchronous fashion and often contains overlapping or redundant information. Thus, it is not trivial how a representation of the environment observed by the vehicle can be created and updated over time. This paper presents a novel methodology to compute an incremental 3D representation of a scenario from 3D range measurements. We propose to use macro scale polygonal primitives to model the scenario. This means that the representation of the scene is given as a list of large scale polygons that describe the geometric structure of the environment. Furthermore, we propose mechanisms designed to update the geometric polygonal primitives over time whenever fresh sensor data is collected. Results show that the approach is capable of producing accurate descriptions of the scene, and that it is computationally very efficient when compared to other reconstruction techniques.
Keywords: Incremental scene reconstruction; Point clouds; Autonomous vehicles; Polygonal primitives
|
|