Skip to main content

Multi-view 3D Object Detection Network for Autonomous Driving

Citations: 1,536 | Paper Published: 21 Jul 2017 | Institutions: Tsinghua University, Baidu | Authors: Xiaozhi Chen, Huimin Ma, Ji Wan, Bo Li, Xia Tian

Abstract in 2 Lines

This paper proposes a framework called Multi-View 3D networks (MV3D) for highly accurate 3D object detection in autonomous driving scenarios. It takes both LIDAR point cloud and RGB images as input and predicts oriented 3D bounding boxes.

Contributions​

  • Proposing a Multi-View 3D object detection network (MV3D) that takes both LIDAR and image data as input and predicts the full 3D extent of objects in 3D space.
  • Outperforming the state-of-the-art by around 25% and 30% AP on the tasks of 3D localization and 3D detection, respectively, on the challenging KITTI benchmark. Additionally, for 2D detection, the proposed approach obtains 14.9% higher AP than the state-of-the-art on the hard data among the LIDAR-based methods.

Practical Implications​

  • The proposed MV3D framework can be used for highly accurate 3D object detection in autonomous driving scenarios, which is crucial for the safety of self-driving cars.
  • The use of both LIDAR and image data as input can improve the accuracy of object detection and reduce false positives.
  • The proposed approach outperforms the state-of-the-art methods on the challenging KITTI benchmark, which is a widely used benchmark for autonomous driving research. This indicates that the proposed approach can be used as a strong baseline for future research in this area.

Methods​

Methodology
  • Multi-View 3D networks (MV3D) framework for 3D object detection in autonomous driving scenarios.
  • Encoding the sparse 3D point cloud with a compact multi-view representation.
  • Two subnetworks: one for 3D object proposal generation and another for multi-view feature fusion.
  • Deep fusion scheme to combine region-wise features from multiple views and enable interactions between intermediate layers of different paths.
  • Comparison of the proposed deep fusion network with early/late fusion approaches.

Data​

The paper uses the KITTI dataset, which is a widely used benchmark for autonomous driving research. The dataset includes LIDAR point clouds and RGB images of road scenes, as well as ground truth annotations for 3D object detection and tracking.

Results​

Performance Analysis
  • The proposed MV3D framework outperforms the state-of-the-art methods on the challenging KITTI benchmark for 3D object detection and localization tasks.
  • The proposed approach achieves around 25% and 30% higher average precision (AP) for 3D localization and 3D detection tasks, respectively, compared to the state-of-the-art methods.
  • For 2D detection, the proposed approach obtains 14.9% higher AP than the state-of-the-art on the hard data among the LIDAR-based methods.
  • The proposed deep fusion network outperforms early/late fusion approaches, indicating the effectiveness of the proposed method for multi-view feature fusion.

Limitations​

  • The proposed MV3D framework requires both LIDAR and image data as input, which may not be available in all autonomous driving scenarios.
  • The proposed approach is computationally expensive and may not be suitable for real-time applications on low-power devices.
  • The proposed approach may not generalize well to other datasets or scenarios, as it is trained and evaluated on the KITTI dataset.

Table of Contents