In recent years, human and object detection has increased research in different real-time applications. Due to improvement in the field of deep learning, various methods have been designed for human, object detection and recognition. Hence, Hybrid Deep Convolutional Neural Network (HDCNN) is developed for human and object detection from the video frames. The HDCNN is a combination of Convolutional Neural Network (CNN) and Emperor Penguin Optimization (EPO). Here, EPO is utilized to increase the system parameters of the CNN structure. Initially, pre-processing is applied to eliminate the noise presented in the image and image quality is enhanced. Here, the Gaussian filter is used for the background subtraction in the images. The three different types of databases are considered to validate the proposed methodology. The proposed HDCNN method is tested in MATLAB and compared with existing methods like Deep Neural Network (DNN), CNN and CNN-Firefly Algorithm (FA), respectively. The proposed method is justified with the statistical measurements like accuracy, precision, recall and F-Measure, respectively.