This paper presents a novel object localizer method named multi-tracker object localizer (MTOL). It detects a specific object in an image by accurately encompassing it with a bounding box (BB). MTOL operates based on a region proposal convolutional neural network (R- CNN) and the multi-tracker optimization algorithm (MTOA). First, a pre-trained R-CNN, i.e., AlexNet with edge-boxes region proposal, detects the approximate location of the target object. Then, to locate the object precisely, a secondary well-trained convolutional neural network (CNN) which estimates and returns the intersection of union (IoU) of the input BB, named IoUCNN, is employed as the fitness function of an optimization problem. Finally, MTOA is used to solve the mentioned optimization problem to find the precise location of the target object. To evaluate the performance of the MTOL, a test is conducted on several images containing specific objects. Investigating and comparing the results show the superiority of the MTOL to R-CNN in terms of accuracy. Additionally, other well-known optimization methods are utilized to evaluate the influence of the MTOA in the localization process. The optimization results reveal the superiority of MTOA.