Multi-criteria Decision-Making Techniques for the Selection of Pareto-optimal Machine Learning Models in a Drinking-Water Quality Monitoring Problem

Machine learning algorithms are valuable tools for solving a wide variety of complex engineering problems. Usually, those problems have multiple criteria to fulfill, but such machine learning-based solutions are usually optimized using a single criterion. In such instances, a multi-objective optimization-based approach could bring interesting solutions by determining a set of Pareto-optimal solutions with different trade-off. Therefore, a multi-criteria decision-making process must be carried out. To the authors' present knowledge, multi-criteria decision-making is yet to be fully explored for selecting preferable Pareto-optimal machine learning models after the training step. Therefore, this paper proposes applying and comparing five different multi-criteria decision-making techniques for selecting a preferred machine learning model. Additionally, an ensemble-based framework is proposed to cope with the difficulty of selecting parameters for such techniques. Those tools are tested on a complex real-world drinking-water quality monitoring problem. Results based on the F1 score indicate that via a multi-criteria decision-making process (F1=0.56), it is possible to select better solutions than single-criterion approaches (F1=0.55). Moreover, the proposed ensemble framework is able to mitigate the difficulty in defining preferences and regions of interest, achieving competitive solutions.