Prato: An automated taxonomy-based reviewer-proposal assignment system

Aim/Purpose: This paper reports our implementation of a prototype system, namely PRATO (Proposals Reviewers Automated Taxonomy-based Organization), for automatic assignment of proposals to reviewers based on categorized tracks and partial matching of reviewers' profiles of research interests against proposal keywords. Background The process of assigning reviewers to proposals tends to be a complicated task as it involves inspecting the matching between a given proposal and a reviewer based on different criteria. The situation becomes worse if one tries to automate this process, especially if a reviewer partially matches the domain of the paper at hand. Hence, a new controlled approach is required to facilitate the matching process. Methodology Proposals and reviewers are organized into categorized tracks as defined by a tree of hierarchical research domains which correspond to the university's colleges and departments. In addition, reviewers create their profiles of research interests (keywords) at the time of registration. Initial assignment is based on the matching of categorized sub-tracks of proposal and reviewer. Where the proposal and a reviewer fall under different categories (sub-tracks), assignment is done based on partial matching of proposal content against reviewers' research interests. Jaccard similarity coefficient scores are calculated of proposal keywords and reviewers' profiles of research interest, and the reviewer with highest score is chosen. Evaluation The system was used to automate the process of proposal-reviewer assignment at the Umm Al-Qura University during the 2017-2018 funding cycle. The list of proposal-reviewer assignments generated by the system was sent to human experts for voting and subsequently to make final assignments accordingly. With expert votes and final decisions as evaluation criteria, data system-expert agreements (in terms of “accept” or “reject”) were collected and analyzed by tallying frequencies and calculating rejection/acceptance ratios to assess the system's performance. Contribution: This work helped the Deanship of Scientific Research (DSR), a funding agency at Umm Al-Qura University, in managing the process of reviewing proposals submitted for funding. We believe the work can also benefit any organizations or conferences to automate the assignment of papers to the most appropriate reviewers. Findings Our developed prototype, PRATO, showed a considerable impact on the entire process of reviewing proposals at DSR. It automated the assignment of proposals to reviewers and resulted in 56.7% correct assignments overall. This indicates that PRATO performed considerably well at this early stage of its development. Recommendation for Practitioners It is important for funding agencies and publishers to automate reviewing process to obtain better reviewing quality in a timely manner. Recommendations for Researchers This work highlighted a new methodology to tackle the proposal-reviewer assignment task in an automated manner. More evaluation might be needed with consideration of different categories, especially for partially matched candidates. Impact on Society The new methodology and knowledge about factors influencing the implementation of automated proposal-reviewing systems will help funding agencies and publishers to improve the quality of their internal processes. Future Research In the future, we plan to examine PRATO's performance on different classification schemes where specialty areas can be represented in graphs rather than trees. With graph representation, the scope for reviewer selection can be widened to include more general fields of specialty. Moreover, we will try to record the reasons for rejection to identify accurately whether the rejection was due to improper assignment or other reasons. © 2018 Informing Science Institute. All rights reserved.