Abstract: The detection and tracking of vehicle in traffic surveillance video is the important part of the intelligent transportation system. This paper proposes a vehicle detection method by combining the W4 background subtraction and optical flow method. Regional filtering and holes filling technologies are utilized to improve the accuracy of target detection. Moreover, the constraint conditions on centroid distance and target size are employed to attain the matching between target regions and the tracking of vehicles. Finally, by means of the characteristics of geometric shape, some abnormal vehicles such as motorcycles are detected and labeled. The test results on real video of highway show that the proposed algorithm can effectively track and label motor vehicle, as well as detecting abnormal vehicles such as motorcycles.