Towards New Methodologies for Assessing Relevance of Information Retrieval from Web Search Engines on Children's Queries

This paper expands on the results of a previous study by Bilal (2012) where she employed benchmarking and intellectual relevance judgment to evaluate and compare the ranking and relevancy of hits retrieved by five web search engines on thirty queries formulated by children to find information for given tasks. Given the dynamic nature of the Web and based on the findings of the study, Bilal called for new approaches for judging relevancy of information retrieval by the engines on children's queries. In the present paper, Bilal and Boehm propose a new multi-tiered research method that could produce a more nuanced and context-based relevance assessment. This method, Reconciled Relevance (RR) combined with Reconciled Relevance Ranking (RRa) is described and challenges for implementing it are outlined. The method has implications for child-driven relevance judgment, ranking algorithms, and relevance theory, as well as for the roles of mediators in maximizing children's web experiences.