Pedestrian trajectory prediction is widely used in various applications, such as intelligent transportation systems, autonomous driving, and social robotics. Precisely forecasting surrounding pedestrians' future trajectories can assist intelligent agents in achieving better motion planning. Currently, deep learning-based trajectory prediction methods have demonstrated superior prediction performance to traditional approaches by learning from trajectory data. However, these methods still face many challenges in improving prediction accuracy, efficiency, and reliability. In this survey, we research the main challenges in deep learning-based pedestrian trajectory prediction methods and study this problem and its solutions through literature collection and analysis. Specifically, we first investigate and analyze the existing literature and surveys on pedestrian trajectory prediction. On this basis, we summarize several main challenges faced by deep learning-based pedestrian trajectory prediction, including motion uncertainty, interaction modeling, scene understanding, data-related issues, and the interpretability of prediction models. We then summarize solutions for each challenge. Subsequently, we introduce mainstream trajectory prediction datasets and analyze the state-of-the-art (SOTA) results reported on them. Finally, we discuss potential research prospects in trajectory prediction, aiming to promote the trajectory prediction community.