Abstract: In order to solve the problem of individual emotion recognition misjudged easily by speech or some kind of physiological signal,a multimodal emotion recognition research method of fusing speech and pulse is proposed.The features of MEL-frequency cepstral coefficient is extracted from the speech signals, which have experienced pretreaments, and a speech emotion recognition is constructed according to the hidden Markov model. The K value of pulse signals and energy value of wavelet packet coefficient are calculated, and the results inputed into least square support vector machine are judged. Then, the two results of the two models are carried out decision-level fusion. The experiment results show that as for sadness and calculates, speech recognition reates is higher than others, which can reach to 100% after fusion, as for happiness and anger, speech recognition tates are 75% and 80%, which can be increased to 95% and 90% after fusion.