Impact of hunting cooperation in predator and anti-predator behaviors in prey in a predator-prey model

In this paper, we propose a predator-prey model with hunting cooperation in predator and anti-predator behaviors in prey. The conditions for the existence and the stability of the unique positive constant equilibrium are given. It is found that with the increasing of the birth rate r0$$ {r}_0 $$ of the prey, the trivial solution loses its stability, and the semi-trivial solution emerges and also loses its stability. For the positive constant solution, we find that as the hunting cooperation b$$ b $$ in predator increases or the fear k0$$ {k}_0 $$ decreases, the positive constant equilibrium loses its stability, and Hopf bifurcation occurs. We also derive the existence of limit cycles by Poincar & eacute;-Bendixson theorem. We also study a diffusive model and derive that self-diffusion can induce Turing instability. Finally, we conduct numerical simulations to present our conclusions.