The impact of a data warehouse on the survey process

Easy access to large collections of historical survey and census data, and the associated metadata that describes it, has long been the goal of researchers and analysts. Many questions have gone unanswered because, the datasets were not readily available, access was limited, and information about the business metadata was inconsistent, not well defined, or simply unavailable. This paper focuses on the impact of a Data Warehouse on the survey process including statistical sampling, survey methodology, data analysis, data quality, and data management. Survey processes at the National Agricultural Statistics Service (NASS) have traditionally focused on program areas that were largely self-contained. This meant the data collected for that program area was also self-contained. Data standardization, cross program analysis, strategic data management initiatives, and integration were considered quite impossible. In fact, there was no clear mandate (political capital) to push for such things - the individual program areas could see no advantage that would motivate changes in their program areas. This paper explores the strategic opportunities that came from the work of combining data from many different program areas into a single and integrated database. It will discuss the conditions that existed prior to the construction and implementation of a Data Warehouse and how the NASS Data Warehouse fulfilled one of its most important goals, the creation of the political capital needed to muster the reengineering and integration of our survey and census processes. It will discuss the changes that occurred in new systems development, data quality, analysis, collaboration across program areas, and survey execution. The paper will draw on previous papers that describe how to build a Data Warehouse, and what some of the design considerations should be, but will not deal extensively with these topics.