Automation in structural health monitoring (SHM) has greatly benefited from computer science's recent advances. Unlike images, the existing datasets for other types of input, such as vibration-based damage data, are relatively smaller, less diverse, and highly imbalanced. Therefore, the reliability of data-driven models developed for safety-critical assessments can be questionable. This paper proposes a dual Bayesian inference where damage predictions are accompanied by measuring the model's confidence in predictions. First, it is shown how dual classification is integrated with Bayesian inference. Later, we introduce a surrogate deep learning module to transform the raw uncertainty output into an easily interpretable prediction uncertainty index (PUI). The PUI metric can be used to alarm a decision-maker of the potential mistakes. The proposed dual Bayesian models are investigated on a 2D structure with seven different sensor layouts. Our approach yields increased robustness for different metrics compared with the benchmark. In addition to the performance boost, PUI information paves the way for a risk-informed implementation of deep learning models in vibration-based damage diagnosis.