Camera is the most widely used sensor in intelligent transportation systems (ITS), but the decline of detection and positioning accuracy caused by target occlusion and external environment interference has always been an important factor restricting the development of ITS. In order to solve this problem, a multi-mode fusion target detection and tracking method based on millimeter-wave radar (MWR) and camera on highway side was designed. Compared with cameras, high-resolution MWR had better measurement accuracy and weather robustness, serving as a better complement to camera perception. The proposed method used the center point-based radar and camera fusion algorithm for target detection and adopted the greedy algorithm for target association. The test results show that on the public dataset nuScenes and the self-built multi-modal dataset of highway side, the proposed method achieves an A_MOTA (multi-object tracking accuracy) performance of 69.1% on the nuScenes dataset, outperforming all visual-based 3D tracking benchmark methods; the self-built dataset verifies its good applicability and accuracy, and the processing time for a single image is 35 ms.