The work aims to propose a two-branch feature extraction network to improve the segmentation accuracy of color images and solve the problem of huge computing costs and redundant parameters in color image segmentation. The two-branch feature extraction network was mainly composed of semantic information branch and spatial detail branch. The semantic information branch obtained the context information of different scales of the input image by setting different hole convolution rates in the asymmetric residual module. The spatial detail branch was a shallow and simple network, which was used to establish the local dependency of each pixel to retain the details. A feature aggregation module was connected behind the two branches to effectively combine the outputs of the two branches. Without any pre-training and post-processing, on a single RTX2080Ti GPU, only 0.91 M parameters were used to achieve 75.1% segmentation accuracy on Cityscapes dataset at the speed of 97 FPS, and 70.5% segmentation effect was achieved on Camvid dataset at the reasoning speed of 107 FPS. A large number of experiments have proved that the proposed model achieves a good balance between segmentation accuracy and efficiency.