Whether the underground open stope can remain stable during the planned period is the premise of safe mining. In the field of stability analysis in open stope, there is a famous graphical tool called the Mathews stability graph. However, mining in deep environments and complex conditions has become a mainstream trend, so the stability graph derived from traditional empirical or semi-empirical techniques must be effectively adjusted to ensure its reliability. In response to this problem, this study will incorporate machine learning to explore feasible alternatives to traditional methods. To improve the credibility of the research results, eight classifiers are employed to compare, and a suitable classifier, support vector machine (SVM), is finally determined. In addition, three meta-heuristic strategies, grey wolf optimizer (GWO), particle swarm optimization (PSO), and cuckoo search algorithm (CS), are introduced to further improve the classifier performance and construct the corresponding hybrid models. Through comprehensive evaluation, CS-SVM is determined as the optimal model (accuracy = 0.8462, precision = 0.8125, recall = 0.9559, and F1 score = 0.8784). Based on this hybrid model with reliable generalization ability, the new decision boundary is output to realize the accurate identification of stable and unstable classes. Compared to traditional techniques, the updated stability graph avoids interference from various subjective factors and demonstrates stronger flexibility and interpretability. In addition, a single prediction technique may lead to accidental errors, and the prediction results near the decision boundary inherently possess fuzzy attributes. To overcome these shortcomings, another aspect of this study is to derive a discriminative criterion for convenient use, which consists of two parts: the expression of the fitted curve for the updated decision boundary and the explicit expression output by the genetic programming (GP) technique. Finally, based on the above efforts, the corresponding prediction platform is constructed to provide feedback information for on-site decision-making, which has certain engineering application value. Machine learning enhances open stope stability analysis in complex geological conditions.SVM classifier and meta-heuristic strategies (GWO, PSO, CS) optimize predictive performance.CS-SVM achieves high accuracy (0.8462) and reliability in stability classification.The hybrid model provides a new decision boundary for accurate stability class identification.Developed prediction platform offers valuable feedback for on-site decision-making.