NetKD: Towards Resource-Efficient Encrypted Traffic Classification Using Knowledge Distillation for Language Models

Encrypted traffic classification is a challenging task characterized by its invisibility of content, class imbalance, and limited number of labeled samples. In current research, the application of pre-trained language models has achieved significant success. However, employing language models for encrypted traffic classification on resource-constrained devices requires reducing model size and inference cost while preserving accuracy. In this paper, we proposed an unsupervised, task-agnostic, transformer-based language model knowledge distillation method for encrypted traffic data, called NetKD. It leverages Multi-Head Self-Attention Relation Alignment, Masked BURST Model, and Same-origin BURST Prediction tasks, to effectively transfer the byte-level contextual knowledge about encrypted traffic data learned by a large-scale "teacher" BERT model, to a compact "student" NetKD-BERT model. The student model only requires a small amount of supervised data for fine-tuning then can be applied to downstream tasks. Experiments show that compared to the state-of-the-art baseline model, our model uses only 4.61% of the parameters, achieves on average 99.10% of its F1 score in 4 downstream tasks, with a classification speed of 0.93ms/packet. Its memory footprint is as low as 7.33% of the baseline model's, leading to a significant 14.3 times faster inference speed.