Formatted Stateful Greybox Fuzzing of TLS server

The TLS protocol is one of the most crucial foundations for ensuring internet security. Consequently, vulnerabilities within the TLS protocol have a significant impact on the Internet security. This paper aims to explore more efficient methods of discovering vulnerabilities in the TLS protocol. Fuzzing stands out as one of the most important techniques for vulnerability discovery in the TLS protocol. To tackle the high complexity of the TLS protocol, stateful greybox fuzzers such as AFLnet have been introduced to enable stateful fuzzing of TLS servers. However, these mutation-based fuzzers often encounter challenges in preserving the message format information during the mutation process, which can undermine the testing results. As a result, this paper proposes a novel approach that incorporates a formatted mutation strategy into the stateful greybox fuzzing process, with the aim of achieving more efficient mutation results. The evaluation process involves four mainstream fuzzers, with OpenSSL's TLS server serving as the target. The results demonstrate that the proposed method significantly enhances the quality of generated seeds, code coverage, and state coverage across all four fuzzers.