With the development of convolutional neural network, the research of pedestrian detection based on multi-spectral images has made great progress and has been widely used. Recent studies have shown that the fusion of image information captured by multispectral sensors (visible and thermal imaging cameras) makes pedestrian detection robust in both good and poor lighting conditions. However, there is still a lack of further research on how to effectively fuse image information according to lighting conditions. This paper presents a novel multi-feature extraction method aiming at extracting useful information from different feature levels. Further, this paper also proposes a novel classification score fusion mechanism. By measuring the illumination in-formation of the input multispectral images and using a fusion function, the classification results output from the two-stream classification network and RPN are combined to improve the performance of pedestrian de-tection. In conclusion, compared with the latest multispectral pedestrian detectors through experiments, the proposed multispectral illumination-aware detection R-CNN (MIAD-RCNN) achieves a lower miss rate and a faster speed during both day time and night time.