The Why, When, What and How about Predictive Continuous Integration : A Simulation-Based Investigation
Journal article
Liu, Bohan, Zhang, He, Ma, Weigang, Li, Gongyuan, Li, Shanshan and Shen, Haifeng. (2023). The Why, When, What and How about Predictive Continuous Integration : A Simulation-Based Investigation. IEEE Transactions on Software Engineering. 49(12), pp. 5223-5249. https://doi.org/10.1109/TSE.2023.3330510
Authors | Liu, Bohan, Zhang, He, Ma, Weigang, Li, Gongyuan, Li, Shanshan and Shen, Haifeng |
---|---|
Abstract | Continuous Integration (CI) enables developers to detect defects early and thus reduce lead time. However, the high frequency and long duration of executing CI have a detrimental effect on this practice. Existing studies have focused on using CI outcome predictors to reduce frequency. Since there is no reported project using predictive CI, it is difficult to evaluate its economic impact. This research aims to investigate predictive CI from a process perspective, including why and when to adopt predictors, what predictors to be used, and how to practice predictive CI in real projects. We innovatively employ Software Process Simulation to simulate a predictive CI process with a Discrete-Event Simulation (DES) model and conduct simulation-based experiments. We develop the Rollback-based Identification of Defective Commits (RIDEC) method to account for the negative effects of false predictions in simulations. Experimental results show that: 1) using predictive CI generally improves the effectiveness of CI, reducing time costs by up to 36.8% and the average waiting time before executing CI by 90.5%; 2) the time-saving varies across projects, with higher commit frequency projects benefiting more; and 3) predictor performance does not strongly correlate with time savings, but the precision of both failed and passed predictions should be paid more attention. Simulation-based evaluation helps identify overlooked aspects in existing research. Predictive CI saves time and resources, but improved prediction performance has limited cost-saving benefits. The primary value of predictive CI lies in providing accurate and quick feedback to developers, aligning with the goal of CI. |
Keywords | Continuous integration; machine learning; software process simulation; discrete-event simulation |
Year | 01 Jan 2023 |
Journal | IEEE Transactions on Software Engineering |
Journal citation | 49 (12), pp. 5223-5249 |
Publisher | IEEE, Institute of Electrical and Electronics Engineers |
ISSN | 0098-5589 |
Digital Object Identifier (DOI) | https://doi.org/10.1109/TSE.2023.3330510 |
Web address (URL) | https://ieeexplore.ieee.org/document/10315109 |
Open access | Published as non-open access |
Research or scholarly | Research |
Page range | 5223-5249 |
Publisher's version | |
Output status | Published |
Publication dates | |
Dec 2023 | |
Publication process dates | |
Accepted | Nov 2022 |
Deposited | 10 Jun 2024 |
Additional information | © Copyright 2023 IEEE. All rights reserved. |
Place of publication | United States |
https://acuresearchbank.acu.edu.au/item/909xy/the-why-when-what-and-how-about-predictive-continuous-integration-a-simulation-based-investigation
Download files
53
total views29
total downloads9
views this month8
downloads this month