Using pose estimation to identify regions and points on natural history specimens

A key challenge in mobilising growing numbers of digitised biological specimens for scientific research is finding high-throughput methods to extract phenotypic measurements on these datasets. In this paper, we test a pose estimation approach based on Deep Learning capable of accurately placing point labels to identify key locations on specimen images. We then apply the approach to two distinct challenges that each requires identification of key features in a 2D image: (i) identifying body region-specific plumage colouration on avian specimens and (ii) measuring morphometric shape variation in Littorina snail shells. For the avian dataset, 95% of images are correctly labelled and colour measurements derived from these predicted points are highly correlated with human-based measurements. For the Littorina dataset, more than 95% of landmarks were accurately placed relative to expert-labelled landmarks and predicted landmarks reliably captured shape variation between two distinct shell ecotypes ('crab' vs 'wave'). Overall, our study shows that pose estimation based on Deep Learning can generate high-quality and high-throughput point-based measurements for digitised image-based biodiversity datasets and could mark a step change in the mobilisation of such data. We also provide general guidelines for using pose estimation methods on large-scale biological datasets. Author summaryAs the digitisation of natural history collections continues apace, a wealth of information is waiting to be mobilised from these vast digital datasets that can help address many evolutionary and ecological questions. Deep Learning has achieved success on many real-world tasks such as face recognition and image classification. Here, we use deep learning to measure phenotypic traits of specimens by placing points on photos of birds and periwinkles. We show that the measurements produced by Deep Learning are generally accurate and very similar to manual measurements taken by experts. As Deep Learning methods vastly reduce the time required to produce these measurements, our results demonstrate the great potential of Deep Learning for future biodiversity studies.