For the modulation classification problems, the deep learning approaches can determine the unknown modulation formats in high confidence. However, it has been maliciously used by eavesdroppers. In this paper, we consider the wireless communication scenario, in which Alice intends to communicate with Bob confidentially in the threat of Eve, who tries to determine the unknown modulation formats of Alice using some deep learning approach. Recent advancements in adversarial machine learning have demonstrated that the deep learning techniques are vulnerable to crafted perturbations. To prevent Eve from classifying Alice's modulation formats, Alice transmits the modulation signal with the well-designed adversarial perturbation. We first formulate an optimization problem to determine the optimized adversarial perturbation, in which the objective is to mislead the modulation classifier of Eve subject to the communication constraints, i.e., the power efficiency, the achievable rate, and the reliability. Then, the augmented Lagrangian method is adopted to solve the perturbation optimization problem, in which the implicit objective is evaluated using the Monte Carlo method, and the gradients of the implicit constraints are obtained using the Gaussian-based estimation algorithm. We further extend the perturbation design to the both cases of Alice having and not having the prior knowledge of Eve. Finally, the input-independent universal perturbation for the specific modulation type is proposed, which is deployed via a lookup table method. Numerical results show that the designed perturbation with 10% power of the modulated signal can attack Eve's modulation classifier with the great success while ensuring both the achievable rate and the reliability close to the ideal case (say, no perturbation). Compared to the existing methods, the designed perturbation achieves the better attack performance and is robust to the filtering, the oversampling, and the time/frequency offset. Furthermore, this paper reveals that the structure type of Eve's model has a large impact on the attack performance, and verifies that the adversarial perturbation can effectively attack the modulation classifiers that resort to the expert knowledge.