Quality control and assessment of interpreter consistency of annual land cover reference data in an operational national monitoring program

The U.S. Geological Survey Land Change Monitoring, Assessment and Projection (USGS LCMAP) initiative is working toward a comprehensive capability to characterize land cover and land cover change using dense Landsat time series data. A suite of products including annual land cover maps and annual land cover change maps will be produced using the Landsat 4-8 data record. LCMAP products will initially be created for the conterminous United States (CONUS) and then extended to include Alaska and Hawaii. A critical component of LCMAP is the collection of reference data using the TimeSync tool, a web-based interface for manually interpreting and recording land cover from Landsat data supplemented with fine resolution imagery and other ancillary data. These reference data will be used for area estimation and validation of the LCMAP annual land cover products. Nearly 12,000 LCMAP reference sample pixels have been interpreted and a simple random subsample of these pixels has been interpreted independently by a second analyst (hereafter referred to as "duplicate interpretations"). The annual land cover reference class labels for the 1984-2016 monitoring period obtained from these duplicate interpretations are used to address the following questions: 1) How consistent are the reference class labels among interpreters overall and per class? 2) Does consistency vary by geographic region? 3) Does consistency vary as interpreters gain experience over time? 4) Does interpreter consistency change with improving availability and quality of imagery from 1984 to 2016? Overall agreement between interpreters was 88%. Class-specific agreement ranged from 46% for Disturbed to 94% for Water, with more prevalent classes (Tree Cover, Grass/Shrub and Cropland) generally having greater agreement than rare classes (Developed, Barren and Wetland). Agreement between interpreters remained approximately the same over the 12-month period during which these interpretations were completed. Increasing availability of Landsat and Google Earth fine resolution data over the 1984 to 2016 monitoring period coincided with increased interpreter consistency for the post-2000 data record. The reference data interpretation and quality assurance protocols implemented for LCMAP demonstrate the technical and practical feasibility of using the Landsat archive and intensive human interpretation to produce national, annual reference land cover data over a 30-year period. Protocols to estimate and enhance interpreter consistency are critical elements to document and ensure the quality of these reference data.