Assessing methods for telluric removal on atmospheric retrievals of high-resolution optical exoplanetary transmission spectra

Context. Recent advancements in ultra-stable gr ound-based high-resolution spectrographs have propelled ground-based astronomy to the forefront of exoplanet detection and characterisation. However, the resultant transmission and emission spectra of exoplanetary atmospheres are inevitably contaminated by telluric absorption and emission lines due to the light's transmission through the Earth's atmosphere above the observatory. Retrieving accurate atmospheric parameters depends on accurate modelling and removal of this telluric contamination while preserving the faint underlying exoplanet signal. Aims There exist many methods to model telluric contamination, whether directly modelling the Earth's transmission spectrum via radiative transfer modelling, or using a principal component analysis (PCA)-like reconstruction to fit the time-invariant features of a spectrum, and removing these models from the observations. We aimed to assess the efficacy of these various telluric removal methods in preserving the underlying exoplanetary spectra. Methods. We compared two of the most common telluric modelling and removal methods, MOLECFIT and the PCA-like algorithm SYSREM, using planetary transmission spectra injected into three high-resolution optical observations taken with ESPRESSO. These planetary signals were injected at orbital periods of P =2 days and P = 12 days, resulting in differing changes in radial velocity during transit. We then retrieved various injected atmospheric model parameters in order to determine the efficacy of the telluric removal methods, as well as their effect on the transmission spectra of exoplanets with varying orbital architectures. Results. For the close-in, high velocity injected signal, we found that SYSREM performed better for species that are also present in the Earth's atmosphere-with accurate and precise retrieval of atmospheric abundances and T-P profiles, across each of the datasets. As we moved to slower moving signals at larger orbital separations, for one of the three datasets, SYSREM dampened the planetary H2O signal, leaving the retrieved abundance value unconstrained. In contrast, the H2O signal was preserved for the telluric modelling method, MOLECFIT. However, this behaviour was not ubiquitous across all three of the injected datasets, with another dataset showing a more precise H2O /Fe ratio when preprocessed with SYSREM. These conflicts highlight the importance of testing individual high-resolution dataset reduction routines independently to ensure real exoplanetary signals are preserved.