Abstract: The quality of a Kinect depth image is easy to be decreased due to the influence of the instability of depth data,image noise,holes with unknown depth and bias between depth and color image,this paper proposed a novel algorithm to inpaint Kinect depth image.Firstly,this paper designed a Kinect calibration scheme and used the calibration results to align the depth image and the color image.Secondly,according to the temporal correlation of depth data,an optimization was applied to improve the data stability.Thirdly,this paper proposed a hole-filling algorithm by combing the spatial correlation of depth data and the color consistency constraint.Finally,an improved bilateral filter was used to smooth the image as well as reserve image edges.Experimental results show that holes are reduced and a smooth with significant depth -edge image could be obtained.