This paper investigates numerically the unsteady two-dimensional convective heat transfer flow of water, kerosene, and engine oil-based nanofluids in a rectotrapezoidal enclosure under the effect of a horizontal periodic magnetic field. An in-house-built code based on the Galerkin-type finite element method has been employed to solve the governing dimensionless equations and boundary conditions. The simulated data, validated with previously published works, generally align with the results. The graphical representation and detailed discussion of key model parameters include the dissemination of streamlines, isotherms, average Nusselt number, friction factor, and thermal efficiency index. The mean Nusselt number for the nanofluid is calculated by analyzing the response surfaces and contours obtained from the response surface methodology, in addition to line graphs that depict different flow parameters. The findings indicate that the heat transfer rate in nanofluid increases significantly with higher thermal Rayleigh numbers, more extended periods of the magnetic field, and more prominent shape factors. Furthermore, the average Nusselt number, representing the convective heat transfer at the bottom heated wall, falls as the Hartmann number increases. For engine oil and kerosene-based nanofluids, the heat transfer rate is increased by 107.53%, 107.58%, 122.91%, and 92.19%, respectively, for Co and Cu, compared to base fluid, for a 5% of nanoparticle volume fraction. The obtained numerical also shows that the Fe3O4-engine oil nanofluid, out of the nine nanofluids examined in the current study, has led to the highest heat transfer rate, about 123.13% greater than the base fluid.