A Recommendation Algorithm Based on a Self-supervised Learning Pretrain Transformer

Click-through rate (CTR) prediction is crucial research direction for the recommendation, with the goal of predicting the probability that users will click on candidate items. There are studies indicates that users’ next click behavior is influenced by their last few clicks; therefore, effective modeling of user behavior sequences to extract user interest representations is an important research topic in CTR prediction. Various networks such as RNN, and transformer, have been applied to implicitly extract user interest in the sequence. However, these studies focus on designing complex network structures for better user behavior modeling, while ignoring the fact that the training methods used in current CTR prediction models may limit the model performance. Specifically, owing to the single training objective of the CTR prediction model, the sequence interest extractor component in the model will not be fully trained due to overemphasis on the final prediction effect during the training process. To solve this issue, this paper proposes a recommendation model based on self-supervised learning to pretrain the transformer (SSPT4Rec), which divides the training into two phases: pretraining and fine-tuning. The transformer is trained by a four-classification pretext task in the pretraining phase, and the weights obtained from the pretraining are used to initialize the transformer in the fine-tuning phase and to fine-tune it in the recommendation task. Extensive experiments on four publicly available datasets reveal that the SSPT4Rec method improves the feature extraction capability of transformer as an interest extractor and outperforms the existing model.