Continual Egocentric Activity Recognition by Multi-level Multimodal Feature Calibration and Temporal Aggregation
-
Abstract
Continual egocentric activity recognition (CEAR) aims to understand new activities from the multimodal first-person data sequentially captured by sensors without forgetting the knowledge of old activities. Most methods of CEAR treat each frame of multimodal egocentric activity data equally when learning the new activities, ignoring the temporal information of egocentric activities. Besides, these methods utilize the multimodal features sampled from old classes to train the classifier, suffering from the multimodal semantic drift problem caused by the incremental change of the feature space. To tackle these problems, we propose the multi-level multimodal feature calibration and temporal aggregation for CEAR. Our method comprises the multimodal temporal aggregation adapter that learns the unified-modal temporal aggregated tokens to represent the temporal information of egocentric activities and maintains the temporal perception ability on old activities through temporal consistency distillation. Moreover, to further mitigate the forgetting problem, we propose multi-level feature calibration to compensate for the multimodal semantic drift, which not only modulates the gradient of an alignment modulation module at the modal level, but also supervises the semantic compensation module to learn the semantic difference between old and new tasks at the semantic level. Extensive experiments on UESTC-MMEA-CL, EPIC-KITCHENS-12-CL, and EPIC-KITCHENS-20-CL datasets demonstrate that our method significantly outperforms existing CEAR methods in various task settings.
Full Text Translation (powered by iFLYTEK)
Full Text Translation (powered by iFLYTEK)
-
-