Abstract: Aiming at the deficiency that model high level semantic motion pattern and semantic movement evolution in the research of traditional video action recognition, this paper proposes a part semantic based video action recognition method. First, it extracts high level semantic features based on the position information of the body parts. Second, we propose a feature descriptor for action recognition based on human part semantic and its movement evolution. The method is implemented on Caffe. The experiment results display good performance.