The work aims to solve the problem that the existing solutions either require manually specified landmarks for corresponding points in the images, or are restricted to specific objects or shape deformations. A low-dimensional representation of images for simultaneously recovering color, appearance and shape was proposed. The proposed algorithm further reduced sample complexity of manifold learning as the manifolds of shape and appearance were restricted to low-dimensional subspaces. The proposed method significantly outperformed the current typical methods of robust optical flow and SIFT flow. Our qualitative results in some related tasks such as image deformation and joint learning are encouraging.