A common tool in the theory of numerical semigroups is to interpret a desired class of semigroups as the lattice points in a rational polyhedron in order to leverage computational and enumerative techniques from polyhedral geometry. Most arguments of this type make use of a parametrization of numerical semigroups with fixed multiplicity m in terms of their m-Apery sets, giving a representation called Kunz coordinates which obey a collection of inequalities defining the Kunz polyhedron. In this work, we introduce a new class of polyhedra describing numerical semigroups in terms of a truncated addition table of their positive sporadic elements. Applying a classical theorem of Ehrhart to slices of these polyhedra, we prove that the number of numerical semigroups with n sporadic elements and Frobenius number f is polynomial up to periodicity, or quasi-polynomial, as a function of f for fixed n. We also generalize this approach to higher dimensions to demonstrate quasi-polynomial growth of the number of affine semigroups with a fixed number of elements, and all gaps, contained in an integer dilation of a fixed polytope.