As a result of their unprecedented luminosity, long gamma-ray bursts (GRBs) and their after-glows pinpoint a population of high-redshift galaxies that is distinct and complementary to galaxies selected by deep sky surveys. In this way, GRBs facilitate a unique view into the high-redshift universe. The bright optical afterglow, for example, serves as a background light for absorption-line spectroscopy, and thus helps us to accurately characterize and understand the cosmochemical evolution of the interstellar matter, the properties of star-forming galaxies through cosmic time or dust extinction at redshifts up to z similar to 6. Similarly, detailed studies of the GRB explosion site, and the specific stellar population and galactic environment linked to the GRB, provide important constraints on the physical properties of the GRB progenitor like stellar mass, or on the interplay between metallicity and GRB formation. In this paper, we review recent results in both areas, focusing on novel results from extensive host observations of well-selected GRB samples.