A general perturbation approach to conjugate duality theory in set-valued optimization considering set criterion, where the image space is a complete lattice whose elements are evenly convex sets, is presented. This image space structure is suitable for the conjugation scheme for set-valued functions which was defined in a previous work (Fajardo in Set-Valued Var. Anal. 30:827-846, 2022), and it allows to obtain a dual problem for a general primal one verifying weak dualty. Fenchel and Lagrange dual problems are presented as examples. We also obtain zero duality gap and strong duality theorems.