Visual inspections of large networks of bridges yield millions of data points scattered across thousands of structural elements. Alongside visual inspections, structural attributes such as age, location and traffic load provide contextual information about the deterioration patterns in the network. Leveraging this network-scale data for modeling deterioration is challenging, especially when each structural element has few inspections over along period of time. Moreover, as new bridge information and inspections are added each year, it is strictly important for deterioration models to be scalable. This paper addresses these challenges by proposing a scalable probabilistic approach for modeling deterioration of large networks of bridges. The new framework consists of state-space models (SSM) for modeling the deterioration based on visual inspections and a Bayesian neural network (BNN) that factors-in information about structural attributes. The role of the BNN model is to learn the mapping between the initial distribution of the deterioration speed and the structural attributes of each bridge. The new framework is shown to be computationally efficient and can seamlessly incorporate a large number of structural attributes, which alleviates the need for feature selection. In addition, the proposed framework incorporates anew approach for learning the inspectors' uncertainty parameters which is shown to provide better generalization. The experiments in this study are based on real data from the network of bridges in the province of Quebec, Canada.