Astrometric planet search around southern ultracool dwarfs I. First results, including parallaxes of 20 M8-L2 dwarfs

Context. Extrasolar-planet searches that target very low-mass stars and brown dwarfs are hampered by intrinsic or instrumental limitations. Time series of astrometric measurements with precisions better than one milli-arcsecond can yield new evidence on the planet occurrence around these objects. Aims. We present first results of an astrometric search for planets around 20 nearby dwarf stars with spectral types M8-L2. Methods. Over a time-span of two years, we obtained I-band images of the target fields with the FORS2 camera at the Very Large Telescope. Using background stars as references, we monitored the targets' astrometric trajectories, which allowed us to measure parallax and proper motions, set limits on the presence of planets, and to discover the orbital motions of two binary systems. Results. We determined trigonometric parallaxes with an average accuracy of 0.09 mas (similar or equal to 0.2%), which resulted in a reference sample for the study of ultracool dwarfs at the M/L transition, whose members are located at distances of 9.5-40 pc. This sample contains two newly discovered tight binaries (DE0630-18 and DE0823-49) and one previously known wide binary (DE1520-44). Only one target shows I-band variability >5 mmag rms. We derived planet exclusion limits that set an upper limit of 9% on the occurrence of giant planets with masses greater than or similar to 5 M-J in intermediate-separation (0.01-0.8 AU) orbits around M8-L2 dwarfs. Conclusions. We demonstrate that astrometric observations with an accuracy of 120 mu as over two years are feasible from the ground and can be used for a planet-search survey. The detection of two tight very low-mass binaries shows that our search strategy is efficient and may lead to the detection of planetary-mass companions through follow-up observations.