GOVERNMENT INFORMATION EXPERT-SYSTEMS - A QUANTITATIVE-EVALUATION

In this article-the first published quantitative evaluation of knowledge-based systems (KBS) or so-called expert systems-the authors quantitatively compare and contrast two systems: POINTER and Government Documents Reference Aid (GDRA). In a test based on fifteen typical U.S. government document reference questions about the federal level of government, POINTER answered 65 percent of the questions correctly while GDRA answered only 37 percent correctly An analysis of keystroke efficiency revealed that POINTER required 120 strokes in the reference interview and 60 for the question negotiation phase while GDRA needed 120 keystrokes in the reference interview but only 45 during its question negotiation. The discussion and implication section should help developers of knowledge-based computer systems focus their future activities in this area and reassure human reference librarians who work with government information that these systems still have a way to go before they are truly competent systems. Nonetheless, the first generation of expert systems for depository libraries could already be playing a widespread, if modest, role in assisting with federal level reference questions.