Publications
publications by categories in reversed chronological order.
2025
- ISSTAEffective REST APIs Testing with Error Message AnalysisLixin Xu, Huayao Wu, Zhenyu Pan, and 4 more authorsIn International Symposium on Software Testing and Analysis, 2025
REST APIs are essential in building modern enterprise systems, while effectively examining their behaviors remains challenging due to the difficulty in inferring constraints from the specifications. To generate valid test inputs for REST APIs, existing approaches are typically feedback-driven, leveraging HTTP status codes received to guide further test input generation. However, these approaches overlook the potentially valuable information described in error messages accompanying HTTP status codes, leading to inefficiencies in exploring the input space of REST APIs. In this paper, we propose EmRest, a black-box testing approach that leverages error message analysis to enhance both valid and exceptional test input generation for REST APIs. For each operation under test, EmRest first identifies all possible value assignment strategies for each of its input parameters. It then repeatedly applies combinatorial testing to sample test inputs based on these strategies, and statistically analyzes the error messages (of 400-range status code) received to infer and exclude invalid combinations of value assignment strategies (i.e., constraints of the input space). Additionally, EmRest seeks to mutate valid value assignment strategies that are finally identified to generate test inputs for exceptional testing. The error messages (of 500-range status code) received are categorized to identify bug-prone operations, for which more testing resources are allocated. Our experimental results on 16 real-world REST APIs demonstrates the effectiveness of EmRest. It achieves higher operation coverage than state-of-the-art approaches in 50% of APIs, and detects 226 unique bugs that cannot be found by the other approaches.
@inproceedings{xu2025, title = {Effective REST APIs Testing with Error Message Analysis}, booktitle = {International Symposium on Software Testing and Analysis}, author = {Xu, Lixin and Wu, Huayao and Pan, Zhenyu and Xu, Tongtong and Wang, Shaohua and Niu, Xintao and and Changhai Nie}, year = {2025}, publisher = {ACM}, }
2024
- ISTTop-down: A better strategy for incremental covering array generationYan Wang, Xintao Niu*, Huayao Wu, and 4 more authorsInformation and Software Technology, 2024
@article{Wangyan2024, author = {Wang, Yan and Niu*, Xintao and Wu, Huayao and Nie, Changhai and Yu, Lei and Wang, Xiaoyin and Xu, Jiaxi}, title = {Top-down: A better strategy for incremental covering array generation}, journal = {Information and Software Technology}, year = {2024}, } - ISSREA Combinatorial Interaction Testing Method for Multi-Label Image ClassifierPeng Wang, Shengyou Hu, Huayao Wu, and 3 more authorsIn International Symposium on Software Reliability Engineering, 2024
@inproceedings{Wangpeng2024, author = {Wang, Peng and Hu, Shengyou and Wu, Huayao and Niu, Xintao and Nie, Changhai and Chen, Lin}, title = {A Combinatorial Interaction Testing Method for Multi-Label Image Classifier}, year = {2024}, booktitle = {International Symposium on Software Reliability Engineering}, }
2023
- ASEATOM: Automated Black-Box Testing of Multi-Label Image Classification SystemsShengyou Hu, Huayao Wu, Peng Wang, and 5 more authorsIn 2023 38th IEEE/ACM International Conference on Automated Software Engineering, 2023
@inproceedings{Hushengyou2023, author = {Hu, Shengyou and Wu, Huayao and Wang, Peng and Chang, Jing and Tu, Yongjun and Jiang, Xiu and Niu, Xintao and Nie, Changhai}, booktitle = {2023 38th IEEE/ACM International Conference on Automated Software Engineering}, title = {ATOM: Automated Black-Box Testing of Multi-Label Image Classification Systems}, year = {2023}, pages = {230-242}, keywords = {Microwave integrated circuits;Correlation;Systematics;Closed box;Artificial neural networks;Search engines;Tagging;Multi-label Image Classification Testing;Black-box Testing;Metamorphic Testing}, doi = {10.1109/ASE56229.2023.00156}, } - TSEEnhancing Fault Injection Testing of Service Systems via Fault-Tolerance BottleneckHuayao Wu, Senyao Yu, Xintao Niu, and 4 more authorsIEEE Transactions on Software Engineering, 2023
Modern large-scale service systems are usually deployed with redundant components to ensure high dependability in distributed and volatile environments. Fault Injection Testing (FIT) is a popular technique for testing such systems, while the application of FIT to validating the correctness of redundant components remains a challenging task, especially when the system’s structural information is unavailable when testing starts. In this study, we refer to a minimum set of faults that, when injected, will cut off all execution paths in a service system as a fault-tolerance bottleneck, and we propose a novel Fault-tolerance Bottleneck driven Fault Injection (FBFI) approach to the exploration and validation of redundant components without prior knowledge of the system’s business structure. The core idea of FBFI is to iteratively infer and inject bottlenecks of the business structure constructed so far. In this way, FBFI is able to discover and test redundant components by repeatedly triggering new system behaviors. The effectiveness and efficiency of FBFI is evaluated using two microservice benchmark systems with different deployment scales. The results reveal that FBFI is more practical and cost-effective than random and lineage-driven FIT approaches in testing service systems of high redundancy levels.
@article{wu2023enhance, title = {Enhancing Fault Injection Testing of Service Systems via Fault-Tolerance Bottleneck}, author = {Wu, Huayao and Yu, Senyao and Niu, Xintao and Nie, Changhai and Pei, Yu and He, Qiang and Yang, Yun}, journal = {IEEE Transactions on Software Engineering}, year = {2023}, publisher = {IEEE}, volume = {1}, number = {1}, pages = {1--1}, doi = {10.1109/TSE.2023.3285357}, } - TOSEMToward More Efficient Statistical Debugging with Abstraction RefinementZhiqiang Zuo, Xintao Niu, Siyi Zhang, and 5 more authorsACM Trans. Softw. Eng. Methodol., Mar 2023
Debugging is known to be a notoriously painstaking and time-consuming task. As one major family of automated debugging, statistical debugging approaches have been well investigated over the past decade, which collect failing and passing executions and apply statistical techniques to identify discriminative elements as potential bug causes. Most of the existing approaches instrument the entire program to produce execution profiles for debugging, thus incurring hefty instrumentation and analysis cost. However, as in fact a major part of the program code is error-free, full-scale program instrumentation is wasteful and unnecessary. This article presents a systematic abstraction refinement-based pruning technique for statistical debugging. Our technique only needs to instrument and analyze the code partially. While guided by a mathematically rigorous analysis, our technique is guaranteed to produce the same debugging results as an exhaustive analysis in deterministic settings. With the help of the effective and safe pruning, our technique greatly saves the cost of failure diagnosis without sacrificing any debugging capability. We apply this technique to two different statistical debugging scenarios: in-house and production-run statistical debugging. The comprehensive evaluations validate that our technique can significantly improve the efficiency of statistical debugging in both scenarios, while without jeopardizing the debugging capability.
@article{zzhiqiang2023, author = {Zuo, Zhiqiang and Niu, Xintao and Zhang, Siyi and Fang, Lu and Khoo, Siau Cheng and Lu, Shan and Sun, Chengnian and Xu, Guoqing Harry}, title = {Toward More Efficient Statistical Debugging with Abstraction Refinement}, year = {2023}, issue_date = {March 2023}, publisher = {Association for Computing Machinery}, address = {New York, NY, USA}, volume = {32}, number = {2}, issn = {1049-331X}, doi = {10.1145/3544790}, journal = {ACM Trans. Softw. Eng. Methodol.}, month = mar, articleno = {36}, numpages = {38}, }
2022
- ICSECombinatorial Testing of RESTful APIsHuayao Wu, Lixin Xu, Xintao Niu, and 1 more authorIn 2022 IEEE/ACM 44rd International Conference on Software Engineering, Mar 2022
This paper presents RestCT, a systematic and fully automatic approach that adopts Combinatorial Testing (CT) to test RESTful APIs. RestCT is systematic in that it covers and tests not only the interactions of a certain number of operations in RESTful APIs, but also the interactions of particular input-parameters in every single operation. This is realised by a novel two-phase test case generation approach, which first generates a constrained sequence covering array to determine the execution orders of available operations, and then applies an adaptive strategy to generate and refine several constrained covering arrays to concretise input-parameters of each operation. RestCT is also automatic in that its application relies on only a given Swagger specification of RESTful APIs. The creation of CT test models (especially, the inferring of dependency relationships in both operations and input-parameters), and the generation and execution of test cases are performed without any human intervention. Experimental results on 11 real-world RESTful APIs demonstrate the effectiveness and efficiency of RestCT. In particular, RestCT can find eight new bugs, where only one of them can be triggered by the state-of-the-art testing tool of RESTful APIs.
@inproceedings{wu2022restful, title = {Combinatorial Testing of RESTful APIs}, author = {Wu, Huayao and Xu, Lixin and Niu, Xintao and Nie, Changhai}, booktitle = {2022 IEEE/ACM 44rd International Conference on Software Engineering}, pages = {426--437}, year = {2022}, organization = {IEEE}, doi = {10.1145/3510003.3510151} } - TSEA theory of pending schemas in combinatorial testingIEEE Transactions on Software Engineering, Mar 2022
Combinatorial Testing (CT) is an effective testing technique for detecting failures which are triggered by the interactions of various factors that influence the behaviour of a system. Although many studies in CT have designed elaborate test suites (called covering arrays) to systemically check each possible factor interaction, they provide weak support to locate the concrete failure-inducing interactions, i.e., the Minimal Failure-causing Schemas (MFS). To this end, a variety of MFS identification approaches have been proposed. However, as this study reveals, these approaches suffer from various issues such as cannot identify multiple overlapping MFSs, cannot handle MFSs with high degrees, cannot be applied to systems with large number of parameters, etc. These issues are essentially caused by the exponential computing complexity of checking every interaction in the test cases. Therefore, they can only focus on a subset of all the possible interactions, resulting in many interactions unnoticed. Ignoring these unnoticed interactions could potentially cause failures that have never been systematically checked. Hence, it is beneficial for MFS identification approaches to identify these interactions. In order to account for these unnoticed interactions in CT, this study introduces the notion of pending schema, based on which a theoretical framework of CT schemas is established. In particular, we formally define the determinability of a schema in CT with respect to given information; as such, the yet-to-be determined schemas are exactly the pending schemas. The relationships between the different schemas (faulty, healthy, and pending) and test cases are also theoretically analyzed. Based on which, we further propose three formulas, along with three corresponding algorithms, for the identification of the pending schemas in failing test cases, and formally prove their correctness. As a result, we reduce the complexity of obtaining pending schemas with respect to the number of factors that may have influences on the software.
@article{niu2022theory, title = {A theory of pending schemas in combinatorial testing}, author = {Niu, Xintao and Wu, Huayao and Nie, Changhai and Lei, Yu and Wang, Xiaoyin}, journal = {IEEE Transactions on Software Engineering}, year = {2022}, publisher = {IEEE}, volume = {48}, number = {10}, pages = {4119--4151}, doi = {10.1109/TSE.2021.3113920} } - ISTAn Adaptive Penalty based Parallel Tabu Search for Constrained Covering Array GenerationYan Wang, Huayao Wu, Xintao Niu, and 2 more authorsInformation and Software Technology, Mar 2022
@article{wang2022adaptive, title = {An Adaptive Penalty based Parallel Tabu Search for Constrained Covering Array Generation}, author = {Wang, Yan and Wu, Huayao and Niu, Xintao and Nie, Changhai and Xu, Jiaxi}, journal = {Information and Software Technology}, volume = {143}, pages = {106768}, year = {2022}, publisher = {Elsevier}, doi = {10.1016/j.infsof.2021.106768} } - TSEEnhance Combinatorial Testing with Metamorphic RelationsXintao Niu, Yanjie Sun, Huayao Wu, and 4 more authorsIEEE Transactions on Software Engineering, Mar 2022
Due to the effectiveness and efficiency in detecting defects caused by interactions of multiple factors, Combinatorial Testing (CT) has received considerable scholarly attention in the last decades. Despite numerous practical test case generation techniques being developed, there remains a paucity of studies addressing the automated oracle generation problem, which holds back the overall automation of CT. As a consequence, much human intervention is inevitable, which is time-consuming and error-prone. This costly manual task also restricts the application of higher testing strength, inhibiting the full exploitation of CT in industrial practice. To bridge the gap between test designs and fully automated test flows, and to extend the applicability of CT, this paper presents a novel CT methodology, named COMER, to enhance the traditional CT by accounting for Metamorphic Relations (MRs). COMER puts a high priority on generating pairs of test cases which match the input rules of MRs, i.e., the Metamorphic Group (MG), such that the correctness can be automatically determined by verifying whether the outputs of these test cases violate their MRs. As a result, COMER can not only satisfy the t-way coverage as what CT does, but also automatically check as many test oracle violations as possible. Several empirical studies conducted on 31 real-world software projects have shown that COMER increased the number of metamorphic groups by an average factor of 75.9 and also increased the failure detection rate by an average factor of 11.3, when compared with CT, while the overall number of test cases generated by COMER barely increased.
@article{niu2021enhance, title = {Enhance Combinatorial Testing with Metamorphic Relations}, author = {Niu, Xintao and Sun, Yanjie and Wu, Huayao and Li, Gang and Nie, Changhai and Lei, Yu and Wang, Xiaoyin}, journal = {IEEE Transactions on Software Engineering}, year = {2022}, publisher = {IEEE}, volume = {48}, number = {12}, pages = {5007--5029}, doi = {10.1109/TSE.2021.3131548}, }
2021
- ICSEIdentifying Key Features from App User ReviewsHuayao Wu, Wenjun Deng, Xintao Niu, and 1 more authorIn 2021 IEEE/ACM 43rd International Conference on Software Engineering, Mar 2021
Due to the rapid growth and strong competition of mobile application (app) market, app developers should not only offer users with attractive new features, but also carefully maintain and improve existing features based on users’ feedbacks. User reviews indicate a rich source of information to plan such feature maintenance activities, and it could be of great benefit for developers to evaluate and magnify the contribution of specific features to the overall success of their apps. In this study, we refer to the features that are highly correlated to app ratings as key features, and we present KEFE, a novel approach that leverages app description and user reviews to identify key features of a given app. The application of KEFE especially relies on natural language processing, deep machine learning classifier, and regression analysis technique, which involves three main steps: 1) extracting feature-describing phrases from app description; 2) matching each app feature with its relevant user reviews; and 3) building a regression model to identify features that have significant relationships with app ratings. To train and evaluate KEFE, we collect 200 app descriptions and 1,108,148 user reviews from Chinese Apple App Store. Experimental results demonstrate the effectiveness of KEFE in feature extraction, where an average F-measure of 78.13% is achieved. The key features identified are also likely to provide hints for successful app releases, as for the releases that receive higher app ratings, 70% of features improvements are related to key features.
@inproceedings{wu2021identifying, title = {Identifying Key Features from App User Reviews}, author = {Wu, Huayao and Deng, Wenjun and Niu, Xintao and Nie, Changhai}, booktitle = {2021 IEEE/ACM 43rd International Conference on Software Engineering}, pages = {922--932}, year = {2021}, organization = {IEEE}, doi = {10.1109/ICSE43902.2021.00088} }
2020
- TSEIdentifying Failure-Causing Schemas in the Presence of Multiple FaultsIEEE Transactions on Software Engineering, Feb 2020
Combinatorial testing (CT) has been proven effective in revealing the failures caused by the interaction of factors that affect the behavior of a system. The theory of Minimal Failure-Causing Schema (MFS) has been proposed to isolate the cause of a failure after CT. Most algorithms that aim to identify MFS focus on handling a single fault in the System Under Test (SUT). However, we argue that multiple faults are more common in practice, under which masking effects may be triggered so that some failures cannot be observed. The traditional MFS theory lacks a mechanism to handle such effects; hence, they may incorrectly isolate the MFS. To address this problem, we propose a new MFS model that takes into account multiple faults. We first formally analyze the impact of the multiple faults on existing MFS identifying algorithms, especially in situations where masking effects are triggered by multiple faults. We then develop an approach that can assist traditional algorithms to better handle multiple faults. Empirical studies were conducted using several kinds of open-source software, which showed that multiple faults with masking effects do negatively affect traditional MFS identifying approaches and that our approach can help to alleviate these effects.
@article{niu2020identifying, author = {Niu, Xintao and Nie, Changhai and Lei, Yu and Leung, Hareton and Wang, Xiaoyin}, journal = {IEEE Transactions on Software Engineering}, title = {Identifying Failure-Causing Schemas in the Presence of Multiple Faults}, year = {2020}, volume = {46}, number = {2}, pages = {141-162}, keywords = {Testing;Bars;Fault diagnosis;Computer bugs;Software algorithms;Open source software;Software testing;combinatorial testing;failure-causing schemas;masking effects}, doi = {10.1109/TSE.2018.2844259}, issn = {2326-3881}, month = feb, } - TSEAn Interleaving Approach to Combinatorial Testing and Failure-Inducing Interaction IdentificationXintao Niu, Changhai Nie, Hareton Leung, and 4 more authorsIEEE Transactions on Software Engineering, Feb 2020
Combinatorial testing (CT) seeks to detect potential faults caused by various interactions of factors that can influence the software systems. When applying CT, it is a common practice to first generate a set of test cases to cover each possible interaction and then to identify the failure-inducing interaction after a failure is detected. Although this conventional procedure is simple and forthright, we conjecture that it is not the ideal choice in practice. This is because 1) testers desire to identify the root cause of failures before all the needed test cases are generated and executed 2) the early identified failure-inducing interactions can guide the remaining test case generation so that many unnecessary and invalid test cases can be avoided. For these reasons, we propose a novel CT framework that allows both generation and identification process to interact with each other. As a result, both generation and identification stages will be done more effectively and efficiently. We conducted a series of empirical studies on several open-source software, the results of which show that our framework can identify the failure-inducing interactions more quickly than traditional approaches while requiring fewer test cases.
@article{niu2020interleaving, author = {Niu, Xintao and Nie, Changhai and Leung, Hareton and Lei, Yu and Wang, Xiaoyin and Xu, Jiaxi and Wang, Yan}, journal = {IEEE Transactions on Software Engineering}, title = {An Interleaving Approach to Combinatorial Testing and Failure-Inducing Interaction Identification}, year = {2020}, volume = {46}, number = {6}, pages = {584-615}, doi = {10.1109/TSE.2018.2865772}, }
2018
- 软件学报覆盖表生成的禁忌搜索算法王燕, 聂长海, 钮鑫涛, and 2 more authorsJournal of Software, Feb 2018
@article{王燕2018覆盖表生成的禁忌搜索算法, title = {覆盖表生成的禁忌搜索算法}, author = {王燕 and 聂长海 and 钮鑫涛 and 吴化尧 and 徐家喜}, journal = {Journal of Software}, volume = {29}, number = {12}, pages = {3665--3691}, year = {2018}, }
2015
- ISTCombinatorial testing, random testing, and adaptive random testing for detecting interaction triggered failuresInformation and Software Technology, Feb 2015
@article{NIE2015198, title = {Combinatorial testing, random testing, and adaptive random testing for detecting interaction triggered failures}, journal = {Information and Software Technology}, volume = {62}, pages = {198 - 213}, year = {2015}, issn = {0950-5849}, doi = {10.1016/j.infsof.2015.02.008}, author = {Nie, Changhai and Wu, Huayao and Niu, Xintao and Kuo, Fei-Ching and Leung, Hareton and Colbourn, Charles J.}, }
2014
- 计算机学报组合测试故障定位的关系树模型钮鑫涛, 聂长海, and Chan Alvin计算机学报, Jul 2014
@article{钮鑫涛 2014 组合测试故障定位的关系树模型, author = {钮鑫涛 and 聂长海 and Alvin, Chan}, title = {组合测试故障定位的关系树模型}, journal = {计算机学报}, year = {2014}, volume = {37}, number = {12}, pages = {2505-2518}, month = jul, }
2013
- IWCTIdentifying Failure-Inducing Combinations Using Tuple RelationshipIn 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation Workshops, Mar 2013
Combinatorial testing (CT) aims at detecting interaction failures between parameters in a system. Identifying the failure-inducing combinations of a failing test configuration can help developers find the cause of this failure. However, most studies in CT focus on detecting the failures rather than identifying failure-inducing combinations. In this paper, we propose the notion of a tuple relationship tree (TRT) to describe the relationships among all the candidate parameter interactions. TRT reduces additional test configurations that need to be generated in the fault localization process, and it also provides a clear view of all possible candidate interactions. As a result, our approach will not omit any possible interaction that could be the cause of a failure. In particular, we can identify multiple failure-inducing combinations that overlap with each other. Moreover, we extend our approach to handle the case where additional failure-inducing combinations may be introduced by newly generated test configurations.
@inproceedings{niu2013ttr, author = {Niu, Xintao and Nie, Changhai and Lei, Yu and Chan, Alvin}, booktitle = {2013 IEEE Sixth International Conference on Software Testing, Verification and Validation Workshops}, title = {Identifying Failure-Inducing Combinations Using Tuple Relationship}, year = {2013}, volume = {}, number = {}, pages = {271-280}, keywords = {program testing;software fault tolerance;trees (mathematics);failure-inducing combination identification;combinatorial testing;CT;failing test configuration;TRT;tuple relationship tree;parameter interaction;fault localization process;test configuration;software system behavior;Testing;Pediatrics;Indexes;Fault diagnosis;Software;Arrays;Educational institutions;Combinatorial testing;Fault localization;Tuple relationship tree;overlapping combinations;New import combinations}, doi = {10.1109/ICSTW.2013.38}, issn = {null}, month = mar, }