A method to assess search engine results

Purpose - The aim of this paper is to develop a methodology for assessing search results retrieved from different sources. Design/methodology/approach - This is a two phase method, where in the first stage users select and rank the ten best search results from a randomly ordered set. In the second stage they are asked to choose the best pre-ranked result from a set of possibilities. This two-stage method allows users to consider each search result separately (in the first stage) and to express their views on the rankings as a whole, as they were retrieved by the search provider. The method was tested in a user study that compared different country-specific search results of Google and Live Search (now Bing). The users were Israelis and the search results came from six sources: Google Israel, Google.com, Google UK, Live Search Israel, Live Search US and Live Search UK. The users evaluated the results of nine pre-selected queries, created their own preferred ranking and picked the best ranking from the six sources. Findings - The results indicate that the group of users in this study preferred their local Google interface, i.e. Google succeeded in its country-specific customisation of search results. Live Search was much less successful in this aspect. Research limitations/implications - Search engines are highly dynamic, thus the findings of the case study have to be viewed cautiously. Originality/value - The main contribution of the paper is a two-phase methodology for comparing and evaluating search results from different sources.