We promote the Noether charge of the electric-magnetic duality symmetry of U(1) gauge theory, "G", to a quantum operator. We construct ladder operators, D-(+/-)a(dagger)(k) and D-(+/-)a((k)), which create and annihilate the simultaneous quantum eigenstates of the quantum Hamiltonian (or number) and the electric-magnetic duality operators, respectively. Therefore, all the quantum states of the U(1) gauge fields can be expressed in the form of vertical bar E, g >, where E is the energy of the state, and g is the eigenvalue of the quantum operator G, where the g is quantized in the unit of 1. We also show that ten independent bilinears comprised of the creation and the annihilation operators can form SO(2,3), which is as demonstrated in Dirac's paper published in 1962. The number operator and the electric-magnetic duality operator are members of the SO(2,3) family of generators. We note that there are two more generators that commute with the number operator(or Hamiltonian). We prove that these generators are, indeed, symmetries of the action in U(1) gauge field theory.