Towards Comprehensive Evaluation for Android Automated Testing Tools

被引：0
|
作者：
Zhong Y. [1 ,2 ]
Shi M.-Y. [1 ,2 ]
Fang C.-R. [1 ,2 ]
Zhao Z.-H. [1 ,2 ]
Chen Z.-Y. [1 ,2 ]
机构：
[1] State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing
[2] Shenzhen Institute, Nanjing University, Shenzhen
来源：
Ruan Jian Xue Bao/Journal of Software | 2023年 / 34卷 / 04期
关键词：
automated testing; fragmentation; multi-dimensional evaluation; mutation testing; UI control widget coverage;
D O I：
10.13328/j.cnki.jos.006701
中图分类号：
学科分类号：
摘要：
Automated testing tools are the primary means of quality assurance for Android applications. With the increase in Android version diversity, underlying hardware variability (fragmentation), and logical complexity, automated testing faces new challenges. Numerous automated testing tools have been developed in recent years to address the above issues. However, there are vast tools with various testing focuses, making it hard for testers to choose the right one. To help testers select the best tool for testing and achieve a unified evaluation for automated testing tools, a multi-characteristic comprehensive evaluation of the Android automated testing (CEAT) method is proposed and an easy-to-use platform is implemented for testers. CEAT introduces three widely accepted evaluation metrics: code coverage, exception detection rate, fusion multi-version compatibility score, and further introduces mutation kill rate based on the mutation testing concept, and UI control widget coverage from the perspective of the user. The five metrics constitute the whole CEAT system, thus realizing a comprehensive multi-dimensional evaluation of Android automated testing tool. To verify the effectiveness of CEAT, a set of 1,089 mutated applications is generated for testing, the experiments are deployed in a real-world cluster containing six mobile devices, and 5,040 test tasks are executed for the testing tools. The results suggest that: (i) the five indicators evaluate the automated testing tools from different perspectives, reflecting the testing performance of different tools in a more multi-dimensional way and validating the effectiveness of CEAT; (ii) CEAT supports testers to assign different weights to the five metrics and obtain comprehensive evaluation results depending on the practical testing requirements, which has certain flexibility; (iii) CEAT automatically reconstructs the APP to obtain mutant APPs and set a specific platform for testing the tool, making it convenient to operate. CEAT effectively provides a reference for testers to select the best Android automated testing tool according to different testing requirements. © 2023 Chinese Academy of Sciences. All rights reserved.
引用
收藏
页码：1630 / 1649
页数：19
相关论文
共 40 条
  • [1] Zheng W, Tang H, Chen X, Et al., State-of-the-art survey of compatibility test for Android mobile application, Computer Research and Developmen, 59, 6, pp. 1370-1387, (2022)
  • [2] Segura S, Durn A, Snchez AB, Et al., Automated metamorphic testing of variability analysis tools, Software Testing Verification & Reliability, 25, 2, (2015)
  • [3] Nguyen BN, Robbins B, Banerjee I, Et al., GUITAR: An innovative tool for automated testing of GUI-driven software, Automated Software Engineering, 21, 1, (2014)
  • [4] Joorabchi ME, Mesbah A, Kruchten P., Real challenges in mobile app development, Proc. of the IEEE Int’l Symp. on Empirical Software Engineering and Measurement, pp. 15-24, (2013)
  • [5] Machiry A, Tahiliani R, Naik M., Dynodroid: An input generation system for Android apps, Proc. of the ESEC/SIGSOFT FSE, pp. 224-234, (2013)
  • [6] Yang W, Prasad MR, Xie T., A grey-box approach for automated GUI-model generation of mobile applications, Proc. of the FASE, pp. 250-265, (2013)
  • [7] Choi W, Necula GC, Sen K., Guided GUI testing of Android apps with minimal restart and approximate learning, Proc. of the OOPSLA, pp. 623-640, (2013)
  • [8] Hao S, Liu B, Nath S, Et al., PUMA: Programmable UI-automation for large-scale dynamic analysis of mobile apps, Proc. of the MobiSys, pp. 204-217, (2014)
  • [9] Li L, Bissyande TF, Papadakis M, Et al., Static analysis of android apps: A systematic literature review, Information and Software Technology, 88, (2017)
  • [10] Xu YR., Design and Implementation of iterative Android application automation test system, (2020)
1234