Abstract: Variable Block Size Motion Estimation (VBSME) is the most important part of H.264 standard. It is not only computationally heavy, but also takes the longest time. In order to reduce the time and computation amount of motion estimation, this paper adopts the hardware implementation method, and proposes a data reuse scheme using the tree structure of Sum of Absolute Differences (SAD) calculation. The scheme makes the data flow between each unit clear and the structure is simpler. At the same time, this paper also considers the feasibility of parallel calculation of Mode Decision (MD) and SAD, and designs the corresponding parallel pipeline structure. Xilinx xc7v585tffg1761-1 development board was used for simulation verification. The results show that the scheme can process 256 pixels of input data at one time, improve the real-time performance, and achieve 100% data utilization. In addition, the scheme supports a maximum resolution of 1920×1080, a frame rate of 60fps, and has low coding delay, which meets the real-time requirements of most occasions.