Abstract: CNN is a deep learning method, where its aim is to the goal of artificial intelligence that could perform any intellectual tasks.This paper proposes to optimize CNN with dropout regularization that it can some units became zero using simulated annealing algorithm based on cross entropy loss function, also proposes model averaging method that it combines retaining probability and probability of each unit within pooling region at test time.We do experiments on MNIST handwritten database and part of CMU-PIE database.Under the same structure and the same number of iterations, the method is superior to other methods.It can get a better recognition rate and can be better to overcome the over-fitting problem.