In this paper, we systematically presented a series of graphics processing unit (GPU) based data processing methods for ultrahigh speed, real-time Fourier Domain optical coherence tomography (FD-OCT): GPU based algorithms including high-speed linear/cubic interpolation, non-uniform fast Fourier transform (NUFFT), numerical dispersion compensation, and multi-GPU implementation were developed to improve the image quality and stability of the system. Full-range complex-conjugate-free FD-OCT was also implemented on the GPU architecture to double the imaging range and to improve SNR. The maximum processing speed of > 3.0 Giga-Voxel/second (> 6.0 Mega-A-scan/second of 1024-pixel FD-OCT) was achieved using NVIDIA's latest GPU modules. The GPU-based volume rendering enabled real-time 4D (3D+ time) FD-OCT imaging, and a 5 volume/second 4D FD-OCT system was demonstrated. These GPU technologies were highly effective in circumventing the imaging reconstruction and visualization bottlenecks exist among current ultra-high speed FD-OCT systems and could significantly facilitate the interventional OCT imaging.