Abstract: Compared with the traditional orthogonal frequency division multiplexing (OFDM) modulation scheme, filter OFDM (F-OFDM) enjoys the superiorities of flexible subcarrier bandwidth and reduced out-of-band emission, which makes it one of the key candidate waveforms for future wireless communication systems. However, under the scenarios with high mobility, the Doppler frequency shifts (DFOs) can seriously degrade the performance of F-OFDM systems. To cope with this problem, an iterative message passing detection (MPD) algorithm based on transform domain is proposed in this paper. The MPD algorithm is based on a sparse factor graph and utilizes iterative massage transmission and update to achieve the suppression of the DFOs. Besides, the interference in the detection can be modeled as a Gaussian variable, which has the benefit to reduce the complexity of the MPD. Furthermore, based on the transform domain, the proposed iterative MPD algorithm can make full use of the enhanced channel sparsity in the transform domain, which effectively decreases the branches between the transceiver nodes in the MPD, and further reduces the complexity of the algorithm. Finally, based on the F-OFDM system, the simulation results show than the proposed scheme enjoys an improvement over the computational complexity and also the system performance compared with the traditional counterpart.