Quantum liquids-known to be realized in He-3, He-4 and electrons in metals-generally exhibit instabilities unforeseen under classical Newtonian dynamics, such as the superfluid and superconducting transitions. Recently, a new quantum liquid, now known as the quantum spin liquid, has been discovered in frustrated antiferromagnetic spin-1/2 systems(1,2). In this state, quantum fluctuations of spins prevent classical antiferromagnetic ordering even at absolute zero, similar to the situation in the well-known quantum liquids. A fundamental question that has remained open is whether instabilities other than classical ordering can occur in a quantum spin liquid, as well as in the well-known quantum liquids. Here we demonstrate experimentally that a quantum spin liquid in an organic triangular-lattice antiferromagnet undergoes an instability involving symmetry breaking and/or topological ordering(3), possibly giving rise to a new quantum state of matter. Our result reveals a new variety of quantum-liquid instability, which might become a comparable concept to the already-known fermion-liquid instabilities (such as Bardeen-Cooper-Schrieffer pairing and Peierls instability) and boson-liquid instability (Bose-Einstein condensation).