Who Should Be Selected to Perform a Task in Crowdsourced Testing?

被引:23
作者
Cui, Qiang [1 ,4 ]
Wang, Junjie [1 ]
Yang, Guowei [2 ]
Xie, Miao [1 ,4 ]
Wang, Qing [1 ,3 ,4 ]
Li, Mingshu [1 ,3 ,4 ]
机构
[1] Chinese Acad Sci, Inst Software, Lab Internet Software Technol, Beijing, Peoples R China
[2] Texas State Univ, Dept Comp Sci, San Marcos, TX USA
[3] Chinese Acad Sci, Inst Software, State Key Lab Comp Sci, Beijing, Peoples R China
[4] Univ Chinese Acad Sci, Beijing, Peoples R China
来源
2017 IEEE 41ST ANNUAL COMPUTER SOFTWARE AND APPLICATIONS CONFERENCE (COMPSAC), VOL 1 | 2017年
基金
美国国家科学基金会; 中国国家自然科学基金;
关键词
D O I
10.1109/COMPSAC.2017.265
中图分类号
TP39 [计算机的应用];
学科分类号
081203 ; 0835 ;
摘要
Crowdsourced testing is an emerging trend in software testing, which relies on crowd workers to accomplish test tasks. Due to the cost constraint, a test task usually involves a limited number of crowd workers. Furthermore, more workers does not necessarily result in detecting more bugs. Different workers, who may have different testing experience and expertise, may make much differences in the test outcomes. For example, some inappropriate workers may miss true bug, introduce false bugs or report duplicated bugs, which decreases the test quality. In current practice, a test task is usually dispatched in a random manner, and the quality of testing cannot be guaranteed. Therefore, it is important to select an appropriate subset of workers to perform a test task to ensure high bug detection rate. This paper introduces ExReDiv, a novel hybrid approach to select a set of workers for a test task. It consists of three key strategies: the experience strategy selects experienced workers; the relevance strategy selects workers with expertise relevant to the given test task; the diversity strategy selects diverse workers to avoid detecting duplicated bugs. We evaluate ExReDiv based on 42 test tasks from one of the largest crowdsourced testing platforms in China, and the experimental results show its effectiveness.
引用
收藏
页码:75 / 84
页数:10
相关论文
共 19 条
  • [11] Expert Recommendation with Usage Expertise
    Ma, David
    Schuler, David
    Zimmermann, Thomas
    Sillito, Jonathan
    [J]. 2009 IEEE INTERNATIONAL CONFERENCE ON SOFTWARE MAINTENANCE, CONFERENCE PROCEEDINGS, 2009, : 535 - +
  • [12] Mao K., 2015, RN, V15, P01
  • [13] Nebeling M., 2012, CROWDSOURCED WEB SIT
  • [14] CrowdOracles: Can the Crowd Solve the Oracle Problem?
    Pastore, Fabrizio
    Mariani, Leonardo
    Fraser, Gordon
    [J]. 2013 IEEE SIXTH INTERNATIONAL CONFERENCE ON SOFTWARE TESTING, VERIFICATION AND VALIDATION (ICST 2013), 2013, : 342 - 351
  • [15] Rothermel G., 1999, Proceedings IEEE International Conference on Software Maintenance - 1999 (ICSM'99). `Software Maintenance for Business Change' (Cat. No.99CB36360), P179, DOI 10.1109/ICSM.1999.792604
  • [16] Tamrawi A., 2011, P 19 ACM SIGSOFT S 1, P365, DOI DOI 10.31274/ETD-180810-1814
  • [17] A novel approach to collaborative testing in a crowdsourcing environment
    Tung, Yuan-Hsin
    Tseng, Shian-Shyong
    [J]. JOURNAL OF SYSTEMS AND SOFTWARE, 2013, 86 (08) : 2143 - 2153
  • [18] Wang J., 2016, ESEM 16
  • [19] Local-Based Active Classification of Test Report to Assist Crowdsourced Testing
    Wang, Junjie
    Wang, Song
    Cui, Qiang
    Wang, Qing
    [J]. 2016 31ST IEEE/ACM INTERNATIONAL CONFERENCE ON AUTOMATED SOFTWARE ENGINEERING (ASE), 2016, : 190 - 201