The penetration of renewable energy distributed generation units in the distribution systems has become widespread due to its many techno-economic and environmental benefits. However, avoiding the unwanted effects of unplanned integration and maximizing the resulting benefits is challenging. Thus, this research introduces a powerful method called modified coyote optimization algorithm (MCOA) for identifying the optimal installation of wind turbine farms (WFs), photovoltaic farms (PVFs), and battery energy storage systems (BESS) in IEEE 123-bus unbalanced distribution system (UDS) and Nha Be 55-bus balanced distribution system (BDS) in Nha Be District, Ho Chi Minh City, Vietnam to minimize total costs. The considered costs include (1) investment, operation, and maintenance (O&M) costs of WFs, PVFs, and BESS; (2) imported energy cost for loads and power losses from the main power grid; and (3) generated emission cost from conventional power plants considering time-varying generation and consumption. Besides, this work also suggests an open-source simulator (OpenDSS) for addressing the power flow problem and develops a co-simulation between two active software (OpenDSS and MATLAB) through the component object model (COM) interface for addressing the continuous optimization problems. The proposed solution by MCOA has demonstrated superiority over other methods through total cost savings of up to 24.13% and 27.46% in IEEE 123-bus UDS and 55-bus BDS, while the values are only 23.11% and 26.50% for salp swarm algorithm (SSA) and 23.76% and 26.78% for coyote optimization algorithm (COA), respectively, as compared to the original cases. Besides, the optimal solution also satisfied the declared constraints as well as the standards for bus voltage, line current, and harmonic distortions.