The work aims to design an image retrieval scheme based on multi-level visual semantic feature fusion, for the purpose of solving such problems as the semantic gap between the low layer features and the middle semantic properties, and the reduced retrieval accuracy caused by the information easily lost in the process of converting low layer features into semantic properties. Firstly, three kinds of image features (deep convolutional neural network (DCNN), Fisher vector and sparse coding spatial pyramid matching (SCSPM) feature) were extracted from the middle level. Secondly, in order to effectively integrate the three kinds of features, a graph based semi supervised learning model was defined to integrate the extracted three middle features to form a multi-level visual semantic feature, so that it could improve the image feature description and thus reduce the semantic gap of the retrieval algorithm by effectively combining the complementary information of three different middle features. Finally, the distance function with visual and semantic unity was introduced and the similarity measure between the query image and the training image was calculated based on the extracted multi-layer visual semantic features to finish the image retrieval task. The ex-perimental results showed that the proposed algorithm had higher retrieval precision and efficiency compared with current popular retrieval methods. The proposed algorithm has good retrieval accuracy, and it has certain reference value in the fields of medical treatment and packaging trademark.