Several web services comprise a Recommender System (RS) that assists the user to discover new products and services such as movies, books, articles, jobs, etc. It plays a significant role in filtering data and providing adequate information to users The main goal of RS is to generate the prediction probability in user preferences and user interest in the items depending on users' past choices. However, due to scalability, diversity, accuracy and data sacristy issues, neighborhood selection is more problematic with rapidly maximizing user's quantity and movies. To resolve this problem, the Improved Restricted Boltzmann Machine (IRBM-BO) method is proposed to optimize the hyperparameters of the recommender system. The Restricted Boltzmann Machine (RBM) is a non-deterministic method that is used to learn the probability distribution. The fine-tuning of a predictive model uses hyperparameter settings for enhancing the prediction quality. The RBM integrated with the Bayesian optimization (BO) algorithm forms an IRBM-BO. The IRBM-BO method is used to address the fine-tuning parameter problems of RBM using the BO algorithm. This BO algorithm optimized three hyperparameters such as learning rate, momentum, and weight-cost with fixed weights for every validation of the proposed method. The proposed IRBM-BO is verified using datasets namely MovieLens 100 K, MovieLens 1 M, and Netflix which are assessed by various evaluation measures namely Mean Absolute Error (MAE) and Root Mean Squared Error (RMSE). The experiments showed that BO approach enhance the effectiveness of IRBM-BO significantly and also enhance the prediction accuracy of a recommender system efficiently.