As recycling and remanufacturing become more common, companies may simultaneously operate two production processes for producing new and remanufactured products, and these are called hybrid production systems. However, such systems bring greater managerial complexity, and thus the determination of an appropriate operation strategy to set the recycling ratio of the old product and allocate the capacities of production to the new and remanufactured products is an important issue. This study mainly focuses on a hybrid production system in which the determination of the optimal operation strategy is based on the consideration of the related, costs, the uncertainty about recycling, the demand substitution, the capacity limitation, and the component durability. Moreover, the competition between new and remanufactured products is also considered to construct the proposed model, with the aim of maximizing the profit of the manufacturer. The mathematical model are designed from several possible scenarios to estimate the expected revenue and related costs of the hybrid production system. A Hessian matrix and multivariate optimization methods are used to examine and obtain the optimal solution of the mathematical model. The impacts of related parameters are investigated by utilizing sensitivity analysis, and the results of this can give insights to the related production managers. (C) 2017 Elsevier Ltd. All rights reserved.