A certain generalization of the algebra gl(N,W) of first-order differential operators acting on a space of inhomogeneous polynomials in R(N-1) is constructed. The generators of this (non-)Lie algebra depend on permutation operators. It is shown that the Hamiltonian of the N-body Calogero model can be represented as a second-order polynomial in the generators of this algebra. The representation given implies that the Calogero Hamiltonian possesses infinitely-many finite-dimensional invariant subspaces with explicit bases, which are closely related to the finite-dimensional representations of the above algebra. This representation is an alternative to the standard representation of the Bargmann-Fock type in terms of creation and annihilation operators.