Xintao Niu (钮鑫涛)

2022

1. TSE

   A theory of pending schemas in combinatorial testing

   Xintao Niu, Huayao Wu, Changhai Nie, and 2 more authors

   IEEE Transactions on Software Engineering, 2022

   Abs Bib HTML

   Combinatorial Testing (CT) is an effective testing technique for detecting failures which are triggered by the interactions of various factors that influence the behaviour of a system. Although many studies in CT have designed elaborate test suites (called covering arrays) to systemically check each possible factor interaction, they provide weak support to locate the concrete failure-inducing interactions, i.e., the Minimal Failure-causing Schemas (MFS). To this end, a variety of MFS identification approaches have been proposed. However, as this study reveals, these approaches suffer from various issues such as cannot identify multiple overlapping MFSs, cannot handle MFSs with high degrees, cannot be applied to systems with large number of parameters, etc. These issues are essentially caused by the exponential computing complexity of checking every interaction in the test cases. Therefore, they can only focus on a subset of all the possible interactions, resulting in many interactions unnoticed. Ignoring these unnoticed interactions could potentially cause failures that have never been systematically checked. Hence, it is beneficial for MFS identification approaches to identify these interactions. In order to account for these unnoticed interactions in CT, this study introduces the notion of pending schema, based on which a theoretical framework of CT schemas is established. In particular, we formally define the determinability of a schema in CT with respect to given information; as such, the yet-to-be determined schemas are exactly the pending schemas. The relationships between the different schemas (faulty, healthy, and pending) and test cases are also theoretically analyzed. Based on which, we further propose three formulas, along with three corresponding algorithms, for the identification of the pending schemas in failing test cases, and formally prove their correctness. As a result, we reduce the complexity of obtaining pending schemas with respect to the number of factors that may have influences on the software.

   @article{niu2022theory,
     title = {A theory of pending schemas in combinatorial testing},
     author = {Niu, Xintao and Wu, Huayao and Nie, Changhai and Lei, Yu and Wang, Xiaoyin},
     journal = {IEEE Transactions on Software Engineering},
     year = {2022},
     publisher = {IEEE},
     volume = {48},
     number = {10},
     pages = {4119--4151},
     doi = {10.1109/TSE.2021.3113920}
   }

2. TSE

   Enhance Combinatorial Testing with Metamorphic Relations

   Xintao Niu, Yanjie Sun, Huayao Wu, and 4 more authors

   IEEE Transactions on Software Engineering, 2022

   Abs Bib HTML

   Due to the effectiveness and efficiency in detecting defects caused by interactions of multiple factors, Combinatorial Testing (CT) has received considerable scholarly attention in the last decades. Despite numerous practical test case generation techniques being developed, there remains a paucity of studies addressing the automated oracle generation problem, which holds back the overall automation of CT. As a consequence, much human intervention is inevitable, which is time-consuming and error-prone. This costly manual task also restricts the application of higher testing strength, inhibiting the full exploitation of CT in industrial practice. To bridge the gap between test designs and fully automated test flows, and to extend the applicability of CT, this paper presents a novel CT methodology, named COMER, to enhance the traditional CT by accounting for Metamorphic Relations (MRs). COMER puts a high priority on generating pairs of test cases which match the input rules of MRs, i.e., the Metamorphic Group (MG), such that the correctness can be automatically determined by verifying whether the outputs of these test cases violate their MRs. As a result, COMER can not only satisfy the t-way coverage as what CT does, but also automatically check as many test oracle violations as possible. Several empirical studies conducted on 31 real-world software projects have shown that COMER increased the number of metamorphic groups by an average factor of 75.9 and also increased the failure detection rate by an average factor of 11.3, when compared with CT, while the overall number of test cases generated by COMER barely increased.

   @article{niu2021enhance,
     title = {Enhance Combinatorial Testing with Metamorphic Relations},
     author = {Niu, Xintao and Sun, Yanjie and Wu, Huayao and Li, Gang and Nie, Changhai and Lei, Yu and Wang, Xiaoyin},
     journal = {IEEE Transactions on Software Engineering},
     year = {2022},
     publisher = {IEEE},
     volume = {48},
     number = {12},
     pages = {5007--5029},
     doi = {10.1109/TSE.2021.3131548},
   }

2020

1. TSE

   Identifying Failure-Causing Schemas in the Presence of Multiple Faults

   Xintao Niu, Changhai Nie, Yu Lei, and 2 more authors

   IEEE Transactions on Software Engineering, Feb 2020

   Abs Bib HTML

   Combinatorial testing (CT) has been proven effective in revealing the failures caused by the interaction of factors that affect the behavior of a system. The theory of Minimal Failure-Causing Schema (MFS) has been proposed to isolate the cause of a failure after CT. Most algorithms that aim to identify MFS focus on handling a single fault in the System Under Test (SUT). However, we argue that multiple faults are more common in practice, under which masking effects may be triggered so that some failures cannot be observed. The traditional MFS theory lacks a mechanism to handle such effects; hence, they may incorrectly isolate the MFS. To address this problem, we propose a new MFS model that takes into account multiple faults. We first formally analyze the impact of the multiple faults on existing MFS identifying algorithms, especially in situations where masking effects are triggered by multiple faults. We then develop an approach that can assist traditional algorithms to better handle multiple faults. Empirical studies were conducted using several kinds of open-source software, which showed that multiple faults with masking effects do negatively affect traditional MFS identifying approaches and that our approach can help to alleviate these effects.

   @article{niu2020identifying,
     author = {Niu, Xintao and Nie, Changhai and Lei, Yu and Leung, Hareton and Wang, Xiaoyin},
     journal = {IEEE Transactions on Software Engineering},
     title = {Identifying Failure-Causing Schemas in the Presence of Multiple Faults},
     year = {2020},
     volume = {46},
     number = {2},
     pages = {141-162},
     keywords = {Testing;Bars;Fault diagnosis;Computer bugs;Software algorithms;Open source software;Software testing;combinatorial testing;failure-causing schemas;masking effects},
     doi = {10.1109/TSE.2018.2844259},
     issn = {2326-3881},
     month = feb,
   }

2. TSE

   An Interleaving Approach to Combinatorial Testing and Failure-Inducing Interaction Identification

   Xintao Niu, Changhai Nie, Hareton Leung, and 4 more authors

   IEEE Transactions on Software Engineering, Feb 2020

   Abs Bib HTML

   Combinatorial testing (CT) seeks to detect potential faults caused by various interactions of factors that can influence the software systems. When applying CT, it is a common practice to first generate a set of test cases to cover each possible interaction and then to identify the failure-inducing interaction after a failure is detected. Although this conventional procedure is simple and forthright, we conjecture that it is not the ideal choice in practice. This is because 1) testers desire to identify the root cause of failures before all the needed test cases are generated and executed 2) the early identified failure-inducing interactions can guide the remaining test case generation so that many unnecessary and invalid test cases can be avoided. For these reasons, we propose a novel CT framework that allows both generation and identification process to interact with each other. As a result, both generation and identification stages will be done more effectively and efficiently. We conducted a series of empirical studies on several open-source software, the results of which show that our framework can identify the failure-inducing interactions more quickly than traditional approaches while requiring fewer test cases.

   @article{niu2020interleaving,
     author = {Niu, Xintao and Nie, Changhai and Leung, Hareton and Lei, Yu and Wang, Xiaoyin and Xu, Jiaxi and Wang, Yan},
     journal = {IEEE Transactions on Software Engineering},
     title = {An Interleaving Approach to Combinatorial Testing and Failure-Inducing Interaction Identification},
     year = {2020},
     volume = {46},
     number = {6},
     pages = {584-615},
     doi = {10.1109/TSE.2018.2865772},
   }