PT-Tuning: Bridging the Gap between Time Series Masked Reconstruction and Forecasting via Prompt Token Tuning

Self-supervised learning has been actively studied in time series domain, especially for masked reconstruction. Most of these methods follow the "Pre-training + Fine-tuning" paradigm in which a new decoder replaces the pre-trained decoder to fit for a specific downstream task, leading to inconsistency of upstream and downstream tasks. In this paper, we point out that the unification of task objectives and adaptation for task difficulty are critical for bridging the gap between masked reconstruction and forecasting. By reserving pre-trained mask token during fine-tuning stage, forecasting can be taken as a special case of masked reconstruction, where future values are masked and reconstructed based on history values. It guarantees the consistency of task objectives but there is still a gap in task difficulty. Because masked reconstruction can utilize contextual information while forecasting can only use historical information to reconstruct. To further mitigate the existed gap, we propose a simple yet effective prompt token tuning (PT-Tuning) paradigm, in which all pre-trained parameters are frozen and only a few trainable prompt tokens are added to extended mask tokens in element-wise manner. Extensive experiments on real-world datasets demonstrate the superiority of proposed paradigm with state-of-the-art performance compared to representation learning and end-to-end forecasting methods.