Quantifying uncertainty in the prediction of soil properties using mid-infrared spectra

Soil's pivotal role in environmental and agricultural processes underscores the importance of accurate soil property predictions for informed decisions and sustainable land management. Spectroscopic techniques, particularly mid-infrared (MIR) spectroscopy, have emerged as rapid and non-destructive tools for soil analysis. Despite advances in predicting soil properties using spectroscopy, quantifying prediction uncertainties has often been overlooked. Accurate uncertainty quantification helps risk assessment and decision-making processes. This study introduces an enhanced version of the variational inference technique to capture uncertainty when using Bayesian Convolutional Neural Networks (Bayesian CNNs). This Bayesian CNNs method was evaluated against two other methods - Bootstrapped Partial Least-Squares regression (Bootsrapped PLS) and Generalised Additive Models (GAM) for their ability to quantify uncertainty in six soil property predictions (clay, sand, silt, pH, phosphorus retention, and carbon) based on MIR spectroscopy. In terms of predictive performance and quality of prediction, our evaluation indicated that both GAMs and Bayesian CNNs outperformed PLS-BS for all six soil properties. The ability of GAMs and Bayesian CNNs to capture non-linear relationships in the data allowed for better fitting to the underlying patterns. Bayesian CNNs, in particular, demonstrated superior performance by combining accurate predictions with robust uncertainty quantification. Our results also showed that, on our dataset, bootstrapping failed to provide satisfactory prediction intervals. We suggest therefore that the evaluation of models should extend beyond standard validation metrics, which typically focuses on prediction accuracy, to include an assessment of the predicted uncertainty.