The hardness is one of the important indexes of mechanical properties of materials. The traditional indentation method will damage the materials, so the nondestructive evaluation of hardness has become a research hotspot in this field. Aiming at the requirements of rapid quantitative nondestructive testing of alloy steel surface hardness, six 24CrNiMo alloy steel specimens with different heat treatment are designed to be measured. The nondestructive testing signals of the specimens are measured by magnetic Barkhausen noise testing system, and three different signal characteristic parameters are extracted. Then, the mapping relationship between different evaluation parameters and hardness is established respectively to obtain three kinds of single parameter evaluation models of hardness. The correlation coefficient and evaluation accuracy of the single parameter evaluation model are verified and compared, and the existing problems and defects of the model are proposed. In order to further improve the accuracy and reliability of hardness evaluation of alloy steel, the multiple evaluation parameters based on the total signal characteristics are proposed, and the evaluation model of multiple parameters is established. The results show that the multivariate evaluation model based on convolution neural network with the average error 0. 97% and the maximum error 2. 78% performs better than single parameter and multiple linear regression models. The research provides a new method for rapid quantitative nondestructive evaluation of alloy steel hardness, and the evaluation accuracy, reliability and stability are improved.