Abstract: Visible imaging is easily affected by illumination at night or in bad weather conditions, which reduces the performance of semantic segmentation system. The above problem can be alleviated by simultaneously using visible cameras and thermal Infrared Ray (IR) sensors. Although there are some jobs involved in the semantic segmentation of RGB-IR images, few of them attempted to improve the segmentation results by enhancing the discriminability of the fusion feature. Therefore, this paper proposes a novel framework of multi-spectral image semantic segmentation based on the process of supervised feature fusion. Firstly, the independent feature fusion modules constitute a whole fusion branch in the semantic segmentation network so that the fusion of high-level features can guide the fusion of lower-level features. Secondly, the segmentation supervision signal is directly applied to the end of the fusion branch to improve the discriminability of the fusion feature and the convergence speed of the network. Finally, in order to improve the segmentation of small objects, a dice loss instead of the cycle loss is specially introduced into the fusion branch to achieve a hybrid supervision of training process. It is shown via the experimental results on two multi-spectral datasets that, compared with other multi-spectral semantic segmentation methods, the proposed method can achieve better segmentation results and have more favorable for small objects segmentation.