Abstract: This paper first compares HSV space for shadow detection and RGB space for shadow detection and analyze the effect on eliminating the shadow. The author concludes that RGB space for shadow detection is more suitable for detection of moving targets. What's more, in this paper, a couple of criterions are defined to analyze whether people disappear, climb, or fall. Experiments demonstrate that the approach which is proposed on abnormal behavior recognition is easy, fast and effective and aehieves good recognition effect.