y-Tuning: an efficient tuning paradigm for large-scale pre-trained models via label representation learning

With current success of large-scale pre-trained models (PTMs), how efficiently adapting PTMs to downstream tasks has attracted tremendous attention, especially for PTMs with billions of parameters. Previous work focuses on designing parameter-efficient tuning paradigms but needs to save and compute the gradient of the whole computational graph. In this paper, we propose y-Tuning, an efficient yet effective paradigm to adapt frozen large-scale PTMs to specific downstream tasks. y-Tuning learns dense representations for labels y defined in a given task and aligns them to fixed feature representation. Without computing the gradients of text encoder at training phrase, y-Tuning is not only parameter-efficient but also training-efficient. Experimental results show that for DeBERTa(XXL) with 1.6 billion parameters, y-Tuning achieves performance more than 96% of full fine-tuning on GLUE Benchmark with only 2% tunable parameters and much fewer training costs.