Volume 19, Number 4, 2022
column
Display Method:
2022, vol. 19, no. 4,
pp. 257-287, doi: 10.1007/s11633-022-1349-9
Abstract:
This paper aims to conduct a comprehensive study on facial-sketch synthesis (FSS). However, due to the high cost of obtaining hand-drawn sketch datasets, there is a lack of a complete benchmark for assessing the development of FSS algorithms over the last decade. We first introduce a high-quality dataset for FSS, named FS2K, which consists of 2 104 image-sketch pairs spanning three types of sketch styles, image backgrounds, lighting conditions, skin colors, and facial attributes. FS2K differs from previous FSS datasets in difficulty, diversity, and scalability and should thus facilitate the progress of FSS research. Second, we present the largest-scale FSS investigation by reviewing 89 classic methods, including 25 handcrafted feature-based facial-sketch synthesis approaches, 29 general translation methods, and 35 image-to-sketch approaches. In addition, we elaborate comprehensive experiments on the existing 19 cutting-edge models. Third, we present a simple baseline for FSS, named FSGAN. With only two straightforward components, i.e., facial-aware masking and style-vector expansion, our FSGAN surpasses the performance of all previous state-of-the-art models on the proposed FS2K dataset by a large margin. Finally, we conclude with lessons learned over the past years and point out several unsolved challenges. Our code is available athttps://github.com/DengPingFan/FSGAN .
This paper aims to conduct a comprehensive study on facial-sketch synthesis (FSS). However, due to the high cost of obtaining hand-drawn sketch datasets, there is a lack of a complete benchmark for assessing the development of FSS algorithms over the last decade. We first introduce a high-quality dataset for FSS, named FS2K, which consists of 2 104 image-sketch pairs spanning three types of sketch styles, image backgrounds, lighting conditions, skin colors, and facial attributes. FS2K differs from previous FSS datasets in difficulty, diversity, and scalability and should thus facilitate the progress of FSS research. Second, we present the largest-scale FSS investigation by reviewing 89 classic methods, including 25 handcrafted feature-based facial-sketch synthesis approaches, 29 general translation methods, and 35 image-to-sketch approaches. In addition, we elaborate comprehensive experiments on the existing 19 cutting-edge models. Third, we present a simple baseline for FSS, named FSGAN. With only two straightforward components, i.e., facial-aware masking and style-vector expansion, our FSGAN surpasses the performance of all previous state-of-the-art models on the proposed FS2K dataset by a large margin. Finally, we conclude with lessons learned over the past years and point out several unsolved challenges. Our code is available at
2022, vol. 19, no. 4,
pp. 288-306, doi: 10.1007/s11633-022-1332-5
Abstract:
Robot-assisted microsurgery (RAMS) has many benefits compared to traditional microsurgery. Microsurgical platforms with advanced control strategies, high-quality micro-imaging modalities and micro-sensing systems are worth developing to further enhance the clinical outcomes of RAMS. Within only a few decades, microsurgical robotics has evolved into a rapidly developing research field with increasing attention all over the world. Despite the appreciated benefits, significant challenges remain to be solved. In this review paper, the emerging concepts and achievements of RAMS will be presented. We introduce the development tendency of RAMS from teleoperation to autonomous systems. We highlight the upcoming new research opportunities that require joint efforts from both clinicians and engineers to pursue further outcomes for RAMS in years to come.
Robot-assisted microsurgery (RAMS) has many benefits compared to traditional microsurgery. Microsurgical platforms with advanced control strategies, high-quality micro-imaging modalities and micro-sensing systems are worth developing to further enhance the clinical outcomes of RAMS. Within only a few decades, microsurgical robotics has evolved into a rapidly developing research field with increasing attention all over the world. Despite the appreciated benefits, significant challenges remain to be solved. In this review paper, the emerging concepts and achievements of RAMS will be presented. We introduce the development tendency of RAMS from teleoperation to autonomous systems. We highlight the upcoming new research opportunities that require joint efforts from both clinicians and engineers to pursue further outcomes for RAMS in years to come.
2022, vol. 19, no. 4,
pp. 307-318, doi: 10.1007/s11633-022-1350-3
Abstract:
Seeing through dense occlusions and reconstructing scene images is an important but challenging task. Traditional frame-based image de-occlusion methods may lead to fatal errors when facing extremely dense occlusions due to the lack of valid information available from the limited input occluded frames. Event cameras are bio-inspired vision sensors that record the brightness changes at each pixel asynchronously with high temporal resolution. However, synthesizing images solely from event streams is ill-posed since only the brightness changes are recorded in the event stream, and the initial brightness is unknown. In this paper, we propose an event-enhanced multi-modal fusion hybrid network for image de-occlusion, which uses event streams to provide complete scene information and frames to provide color and texture information. An event stream encoder based on the spiking neural network (SNN) is proposed to encode and denoise the event stream efficiently. A comparison loss is proposed to generate clearer results. Experimental results on a large-scale event-based and frame-based image de-occlusion dataset demonstrate that our proposed method achieves state-of-the-art performance.
Seeing through dense occlusions and reconstructing scene images is an important but challenging task. Traditional frame-based image de-occlusion methods may lead to fatal errors when facing extremely dense occlusions due to the lack of valid information available from the limited input occluded frames. Event cameras are bio-inspired vision sensors that record the brightness changes at each pixel asynchronously with high temporal resolution. However, synthesizing images solely from event streams is ill-posed since only the brightness changes are recorded in the event stream, and the initial brightness is unknown. In this paper, we propose an event-enhanced multi-modal fusion hybrid network for image de-occlusion, which uses event streams to provide complete scene information and frames to provide color and texture information. An event stream encoder based on the spiking neural network (SNN) is proposed to encode and denoise the event stream efficiently. A comparison loss is proposed to generate clearer results. Experimental results on a large-scale event-based and frame-based image de-occlusion dataset demonstrate that our proposed method achieves state-of-the-art performance.
2022, vol. 19, no. 4,
pp. 319-330, doi: 10.1007/s11633-022-1338-z
Abstract:
Federated learning (FedL) is a machine learning (ML) technique utilized to train deep neural networks (DeepNNs) in a distributed way without the need to share data among the federated training clients. FedL was proposed for edge computing and Internet of things (IoT) tasks in which a centralized server was responsible for coordinating and governing the training process. To remove the design limitation implied by the centralized entity, this work proposes two different solutions to decentralize existing FedL algorithms, enabling the application of FedL on networks with arbitrary communication topologies, and thus extending the domain of application of FedL to more complex scenarios and new tasks. Of the two proposed algorithms, one, called FedLCon, is developed based on results from discrete-time weighted average consensus theory and is able to reconstruct the performances of the standard centralized FedL solutions, as also shown by the reported validation tests.
Federated learning (FedL) is a machine learning (ML) technique utilized to train deep neural networks (DeepNNs) in a distributed way without the need to share data among the federated training clients. FedL was proposed for edge computing and Internet of things (IoT) tasks in which a centralized server was responsible for coordinating and governing the training process. To remove the design limitation implied by the centralized entity, this work proposes two different solutions to decentralize existing FedL algorithms, enabling the application of FedL on networks with arbitrary communication topologies, and thus extending the domain of application of FedL to more complex scenarios and new tasks. Of the two proposed algorithms, one, called FedLCon, is developed based on results from discrete-time weighted average consensus theory and is able to reconstruct the performances of the standard centralized FedL solutions, as also shown by the reported validation tests.
2022, vol. 19, no. 4,
pp. 331-346, doi: 10.1007/s11633-022-1336-1
Abstract:
Due to the popularity of group activities in social media, group recommendation becomes increasingly significant. It aims to pursue a list of preferred items for a target group. Most deep learning-based methods on group recommendation have focused on learning group representations from single interaction between groups and users. However, these methods may suffer from data sparsity problem. Except for the interaction between groups and users, there also exist other interactions that may enrich group representation, such as the interaction between groups and items. Such interactions, which take place in the range of a group, form a local view of a certain group. In addition to local information, groups with common interests may also show similar tastes on items. Therefore, group representation can be conducted according to the similarity among groups, which forms a global view of a certain group. In this paper, we propose a novel global and local information fusion neural network (GLIF) model for group recommendation. In GLIF, an attentive neural network (ANN) activates rich interactions among groups, users and items with respect to forming a group′s local representation. Moreover, our model also leverages ANN to obtain a group′s global representation based on the similarity among different groups. Then, it fuses global and local representations based on attention mechanism to form a group′s comprehensive representation. Finally, group recommendation is conducted under neural collaborative filtering (NCF) framework. Extensive experiments on three public datasets demonstrate its superiority over the state-of-the-art methods for group recommendation.
Due to the popularity of group activities in social media, group recommendation becomes increasingly significant. It aims to pursue a list of preferred items for a target group. Most deep learning-based methods on group recommendation have focused on learning group representations from single interaction between groups and users. However, these methods may suffer from data sparsity problem. Except for the interaction between groups and users, there also exist other interactions that may enrich group representation, such as the interaction between groups and items. Such interactions, which take place in the range of a group, form a local view of a certain group. In addition to local information, groups with common interests may also show similar tastes on items. Therefore, group representation can be conducted according to the similarity among groups, which forms a global view of a certain group. In this paper, we propose a novel global and local information fusion neural network (GLIF) model for group recommendation. In GLIF, an attentive neural network (ANN) activates rich interactions among groups, users and items with respect to forming a group′s local representation. Moreover, our model also leverages ANN to obtain a group′s global representation based on the similarity among different groups. Then, it fuses global and local representations based on attention mechanism to form a group′s comprehensive representation. Finally, group recommendation is conducted under neural collaborative filtering (NCF) framework. Extensive experiments on three public datasets demonstrate its superiority over the state-of-the-art methods for group recommendation.
