The work aims to study the influence of the training sample selection method in the process of spectral reconstruction on the spectral reconstruction accuracy. Munsell and ColorChecker SG (test samples) were reconstructed by using the method of pseudo inverse. Training samples were selected from unscreened Munsell sets and the Munsell sets selected through dynamic clustering and dual dynamic selection proposed in the paper. Then the spectral reconstruction accuracy was obtained by comparing three sample selection methods. The experimental results showed that the reconstruction accuracy of training samples subject to double dynamic selection was apparently higher than that of the samples analyzed by dynamic clustering and the unscreened samples, whether it was root-mean-square error (RMSE), goodness of fit (GFC) or color chromatic error under different light sources (A, D50, and F2). A new sample selection method is proposed. The selection method brings good effects and it is advanced to some extent.