Multiferroics possess two or more switchable states such as polarization, magnetization, etc. Phonon excitations in multiferroic phase are strongly modified by magnetoelectric coupling, spin-phonon coupling, and anharmonic phonon-phonon interactions. Here, we have investigated the correlation between phonons and multiferroic order parameters in hexagonal BaMO3: M = Ti1-x(Mn/Fe)(x) systems using powder x-ray diffraction (PXRD), Raman spectroscopic, and magnetic measurements. The structural transformation from a polar tetragonal to a non-polar 6H-type hexagonal phase is observed as a function of doping (Mn/Fe). Magnetic measurements reveal that the BaTi(1-x)MnxO(3) is paramagnetic while BaTi(1-x)FexO(3) exhibits composition-dependent ferromagnetic order. Importantly, Anomalous temperature-dependence is observed for two phonons (E-1g at -152 cm(-1) and A1g at 636 cm(-1)) in both the systems exhibiting similar trend with the doping (Mn/Fe) irrespective of the differences in their magnetic ground state. Hence, we attribute the phonon anomalies in both the (Fe/Mn doped) systems to strong anharmonic phonon-phonon interactions arising from large atomic displacements involved in the vibrations. In addition, we have also observed signatures of correlation of phonons with ferroelectric phase as well as magnetically ordered state suggesting the presence of a strain-induced magnetoelectric coupling in the doped compounds. (C) 2020 Elsevier B.V. All rights reserved.