We report on optical and RXTE observations of a new high-latitude bursting X-ray transient, XTE J2123-058. We identified the optical counterpart and discovered a 5.9573 +/- 0.0016 hr periodic optical modulation, which was subsequently shown to be the same as the spectroscopic orbital period. From the absence of orbital X-ray modulation and the presence of partial optical eclipses we conclude that the binary inclination is between 55 degrees and 73 degrees. From the optical magnitude in quiescence and from the X-ray flux of type I X-ray bursts, we estimate that the source distance is between 4.5 and 15 kpc, which implies that the source is unusually far from the Galactic plane, since b = -36.degrees 2. Optical bursts with properties consistent with being reprocessed X-ray bursts occurred. We detected a pair of high-frequency quasi-periodic oscillations (QPOs) at 847.1 +/- 5.5 Hz and 1102 +/- 13 Hz simultaneously. According to the beat-frequency model, this QPO separation implies a neutron star spin period of 3.92 +/- 0.22 ms. A change in the energy spectrum occurred during the decay of the outburst, which may have been due to the onset of the propeller mechanism. If so, then the neutron star magnetic field strength is between 2 and 8 x 10(8) G for an assumed distance of 10 kpc. However, the changes in the timing and spectral properties observed during the decay are typical of atoll sources, which may indicate that the changes are due solely to the dynamics of the accretion disk. As the phase-averaged V-band magnitude declined from 17.26 at the peak of the outburst to 19.24, and the X-ray flux decreased from 9.6 x 10(-10) to 7.3 x 10(-11) ergs cm(-2) s(-1), the peak-to-peak amplitude of the V-band modulation increased from 0.75 to 1.49 mag. This behavior can be explained if the size of the accretion disk decreases during the decay of the outburst.