Evaluating long term forecasts

The U.S. Department of Energy's Energy Information Administration (EIA), and its predecessor organizations, has published projections of U.S. energy production, consumption, distribution and prices annually for over 30 years. A natural issue to raise in evaluating the projections is an assessment of their accuracy compared to eventual outcomes. A related issue is the determination of the sources of "error" in the projections that are due to differences between the actual versus realized values of the associated assumptions. One way to do this would be to run the computer-based model from which the projections are derived at the time the projected values are realized, using actual rather than assumed values for model assumptions; and, compare these results to the original projections. For long term forecasts, this approach would require that the model's software and hardware configuration be archived and available for many years, possibly decades, into the future. Such archival creates many practical problems: and, in general, it is not being done. This paper reports on an alternative approach for evaluating the projections. In the alternative approach, the model is run many times for cases in which important assumptions are changed individually and in combinations. A database is assembled from the solutions and a regression analysis is conducted for each important projected variable with the associated assumptions chosen as exogenous variables. When actual data are eventually available, the regression results are then used to estimate the sources of the differences in the projections of the endogenous variables compared to their eventual outcomes. The results presented here are for residential and commercial sector natural gas and electricity consumption. (C) 2009 Elsevier B.V. All rights reserved.