Transverse instability in nonparaxial systems with four-wave mixing

We present a two-dimensional coupled nonlinear Schrodinger-like system with spatial diffractions, degree of birefringence, and four-wave mixing. This system describes two physical contexts: optical pulse propagation beyond the paraxial approximation in a weakly birefringence waveguide and light propagation near exciton-polariton resonance in semiconductor superlattice materials. We find that such systems naturally support different types of diffraction profiles, including spherical, ellipsoidal, and hyperbolic structures. We then study the transverse instability of the two-dimensional system caused by an infinitesimal perturbation-induced continuous-wave solution. Also, we find out how various physical parameters, such as nonparaxiality, degree of birefringence, power, and four-wave mixing, affect the modulational instability (MI) process, in particular. We explore the existence of bright solitary wave solutions for the proposed system as the influence of MI is closely related to the latter in a nutshell.