Abstract: Robust object detection in 3D point clouds faces the challenges caused by sparse range data. Accumulating multi-frame data could densify the 3D point clouds and greatly benefit detection task. However, accurately aligning the point clouds before the detecting process is a difficult task since there may exist moving objects in the scene. In this paper a novel scene flow based multi-frame network named SDP-Net is proposed. It is able to perform multiple tasks such as self-alignment, 3D object detection, prediction and tracking simultaneously. Thanks to the design of scene flow and the scheme of multi-task, our network is capable of working effectively with a simple network backbone. We further improve the annotations on KITTI RAW dataset by supplementing the ground truth. Experimental results show that our approach greatly outperforms the state-of-the-art and can perform multiple tasks in real-time.