The work aims to propose a lightweight semantic segmentation network incorporating efficient channel attention mechanism to solve the problem of large number of model parameters and low segmentation accuracy when Convolutional Neural Network performs semantic segmentation in RGB-D images. Based on RefineNet, the network model was lightened by Depthwise Separable Convolution. In addition, an efficient channel attention mechanism was applied to the encoding network and the decoding network. Firstly, the features of RGB image and depth image were extracted by the encoder network with channel attention mechanism. Secondly, the two features were fused in multiple dimensions by the fusion module. Finally, the segmentation results were obtained by the lightweight decoder network and compared with the segmentation results of 6 networks such as RefineNet. The proposed algorithm was tested on public datasets commonly used in semantic segmentation networks. The experimental results showed that the parameters of the proposed network model were only 90.41 MB, and the mIoU was 1.7% higher than that of RefineNet network, reaching 45.3%. The experimental results show that the proposed network can improve the precision of semantic segmentation even when the number of parameters is greatly reduced.