A Survey on End-to-end Perception and Prediction for Autonomous Driving

Traditional autonomous driving research decomposes the problem into five distinct subtasks: perception, tracking, prediction, planning, and control. Despite the recent significant progress, the paradigm faces challenges due to limitations in computational capacity and the propagation of cumulative errors, resulting in unsatisfactory outcomes in real-world scenarios. Recently, researchers have shifted their focus toward a novel paradigm: end-to-end methods for perception and prediction (PnP). This approach integrates concepts from multi-object tracking and joint perception and prediction models, promoting synergistic enhancements across various tasks to achieve superior performance. In this paper, we conduct a comprehensive survey of the PnP research, aiming to encompass the entirety of the work in this area. First, we introduce the PnP pipeline and provide an overview of our survey. We then delve into PnP methods by modality, including LiDAR, camera, and multi-modal data, offering detailed insights into their architectures. Furthermore, we discuss the future directions of this field in new scenarios.