We study the geometric origin of generalized Gouy phases in paraxial optical modes of arbitrary order. We focus on the specific case of cyclic beam transformations of non-astigmatic vortex beams, thereby, generalizing the well-known geometric phase shift for first-order beams with orbital angular momentum to modes of arbitrary order. Our method involves two pairs of bosonic ladder operators, which, analogous to the algebraic description of the quantum-mechanical harmonic oscillator in two dimensions, connect transverse modes of different order. Rather than studying the geometry of the infinite-dimensional space of higher-order modes, we focus on the space underlying the ladder operators. We identify overall phases of the ladder operators, thereby obtaining the phases of all higher-order modes, and show that the variation of these phases under optical elements and transformations has a geometric interpretation in terms of the other parameters involved.