Quantum-Hall to Insulator Transition in Ultra-Low-Carrier-Density Topological Insulator Films and a Hidden Phase of the Zeroth Landau Level

A key feature of the topological surface state under a magnetic field is the presence of the zeroth Landau level at the zero energy. Nonetheless, it is challenging to probe the zeroth Landau level due to large electron-hole puddles smearing its energy landscape. Here, by developing ultra-low-carrier density topological insulator Sb2Te3 films, an extreme quantum limit of the topological surface state is reached and a hidden phase at the zeroth Landau level is uncovered. First, an unexpected quantum-Hall-to-insulator-transition near the zeroth Landau level is discovered. Then, through a detailed scaling analysis, it is found that this quantum-Hall-to-insulator-transition belongs to a new universality class, implying that the insulating phase discovered here has a fundamentally different origin from those in nontopological systems.