2022
Journal article
Open Access
Designing and testing systems of systems: from variability models to test cases passing through desirability assessment
Lonetti F., De Oliveira Neves V., Bertolino A.
In the early stages of a system of systems (SoS) conception, several constituent systems could be available that provide similar functionalities. An SoS design methodology should provide adequate means to model variability in order to support the opportunistic selection of the most desirable SoS configuration. We propose the VANTESS approach that (i) supports SoS modeling taking into account the variation points implied by the considered constituent systems; (ii) includes a heuristics to weight benefits and costs of potential architectural choices (called as SoS variants) for the selection of the constituent systems; and finally (iii) also helps test planning for the selected SoS variant by deriving a simulation model on which test objectives and scenarios can be devised. We illustrate an application example of VANTESS to the "educational" SoS and discuss its pros and cons within a focus group.Source: Journal of software (Malden, Mass. Online) (2022). doi:10.1002/smr.2427
DOI: 10.1002/smr.2427
Metrics:
Lonetti F., De Oliveira Neves V., Bertolino A.
In the early stages of a system of systems (SoS) conception, several constituent systems could be available that provide similar functionalities. An SoS design methodology should provide adequate means to model variability in order to support the opportunistic selection of the most desirable SoS configuration. We propose the VANTESS approach that (i) supports SoS modeling taking into account the variation points implied by the considered constituent systems; (ii) includes a heuristics to weight benefits and costs of potential architectural choices (called as SoS variants) for the selection of the constituent systems; and finally (iii) also helps test planning for the selected SoS variant by deriving a simulation model on which test objectives and scenarios can be devised. We illustrate an application example of VANTESS to the "educational" SoS and discuss its pros and cons within a focus group.Source: Journal of software (Malden, Mass. Online) (2022). doi:10.1002/smr.2427
DOI: 10.1002/smr.2427
Metrics:
See at:
ISTI Repository 
| onlinelibrary.wiley.com 
| CNR ExploRA
2022
Journal article
Open Access
A Delphi study to recognize and assess systems of systems vulnerabilities
Olivero M. A., Bertolino A., Dominguez-Mayo F. J., Matteucci I., María José Escalona M. J.
Context System of Systems (SoS) is an emerging paradigm by which independent systems collaborate by sharing resources and processes to achieve objectives that they could not achieve on their own. In this context, a number of emergent behaviors may arise that can undermine the security of the constituent systems. Objective We apply the Delphi method with the aims to improve our understanding of SoS security and related problems, and to investigate their possible causes and remedies. Method Experts on SoS expressed their opinions and reached consensus in a series of rounds by following a structured questionnaire. Results The results show that the experts found more consensus in disagreement than in agreement about some SoS characteristics, and on how SoS vulnerabilities could be identified and prevented. Conclusions From this study we learn that more work is needed to reach a shared understanding of SoS vulnerabilities, and we leverage expert feedback to outline some future research directions.Source: Information and software technology 146 (2022). doi:10.1016/j.infsof.2022.106874
DOI: 10.1016/j.infsof.2022.106874
Metrics:
Olivero M. A., Bertolino A., Dominguez-Mayo F. J., Matteucci I., María José Escalona M. J.
Context System of Systems (SoS) is an emerging paradigm by which independent systems collaborate by sharing resources and processes to achieve objectives that they could not achieve on their own. In this context, a number of emergent behaviors may arise that can undermine the security of the constituent systems. Objective We apply the Delphi method with the aims to improve our understanding of SoS security and related problems, and to investigate their possible causes and remedies. Method Experts on SoS expressed their opinions and reached consensus in a series of rounds by following a structured questionnaire. Results The results show that the experts found more consensus in disagreement than in agreement about some SoS characteristics, and on how SoS vulnerabilities could be identified and prevented. Conclusions From this study we learn that more work is needed to reach a shared understanding of SoS vulnerabilities, and we leverage expert feedback to outline some future research directions.Source: Information and software technology 146 (2022). doi:10.1016/j.infsof.2022.106874
DOI: 10.1016/j.infsof.2022.106874
Metrics:
See at:
ISTI Repository | CNR ExploRA | www.sciencedirect.com
2022
Report
Restricted
Unobtrusive in vivo test and rollback of Java applications
Bertolino A., De Angelis G., Miranda B., Tonella P.
Modern software systems accommodate complex configurations and execution conditions that depend on the environment where the software is run. While in house testing can exercise only a fraction of such execution contexts, in vivo testing can take advantage of the execution state observed in the field to conduct further testing activities. In this paper, we present the Groucho approach to in vivo testing. Groucho can suspend the execution, run some in vivo tests, rollback the side effects introduced by such tests, and eventually resume normal execution. Differently from the state-of-art approach Invite, Groucho can be transparently applied to the original application code, even if only available as compiled code, and is fully automated. Our empirical studies of the performance overhead introduced by Groucho under various configurations showed that this may be kept to a negligible level by activating in vivo testing with low probability. Our empirical studies about the effectiveness of the approach confirm previous findings on the existence of faults that are unlikely exposed in house and become easy to expose in the field. Moreover, we include the first study to quantify the coverage increase gained when in vivo testing is added to complement in house testing.Source: ISTI Technical Report, ISTI-2022-TR/008, 2022
DOI: 10.32079/isti-tr-2022/008
Metrics:
Bertolino A., De Angelis G., Miranda B., Tonella P.
Modern software systems accommodate complex configurations and execution conditions that depend on the environment where the software is run. While in house testing can exercise only a fraction of such execution contexts, in vivo testing can take advantage of the execution state observed in the field to conduct further testing activities. In this paper, we present the Groucho approach to in vivo testing. Groucho can suspend the execution, run some in vivo tests, rollback the side effects introduced by such tests, and eventually resume normal execution. Differently from the state-of-art approach Invite, Groucho can be transparently applied to the original application code, even if only available as compiled code, and is fully automated. Our empirical studies of the performance overhead introduced by Groucho under various configurations showed that this may be kept to a negligible level by activating in vivo testing with low probability. Our empirical studies about the effectiveness of the approach confirm previous findings on the existence of faults that are unlikely exposed in house and become easy to expose in the field. Moreover, we include the first study to quantify the coverage increase gained when in vivo testing is added to complement in house testing.Source: ISTI Technical Report, ISTI-2022-TR/008, 2022
DOI: 10.32079/isti-tr-2022/008
Metrics:
See at:
CNR ExploRA
2022
Report
Restricted
Insights from running flaky tests into the field: extended version
Barboni M., Bertolino A., De Angelis G.
Test flakiness is a topmost concern in software test automation. While conducting pre-deployment testing, those tests that are flagged as ?flaky are put aside for being either repaired or discarded. We hypothesize that some flaky tests could provide useful insights if run in the field, and could help identify hard-to-detect failures that escape testing and present themselves in operation. We present the first study to investigate the behaviour of flaky tests when moved to the field. Our experimentation over 52 test methods labelled as flaky provides a first confirmation that moving from the laboratory to the field, the behaviour of tests changes and, in particular, the failure frequency of intermittently failing tests can increase. More importantly, we could identify few cases of field failures that would have been hard to detect while testing in house.Source: ISTI Technical Report, ISTI-2022-TR/007, 2022
DOI: 10.32079/isti-tr-2022/007
Metrics:
Barboni M., Bertolino A., De Angelis G.
Test flakiness is a topmost concern in software test automation. While conducting pre-deployment testing, those tests that are flagged as ?flaky are put aside for being either repaired or discarded. We hypothesize that some flaky tests could provide useful insights if run in the field, and could help identify hard-to-detect failures that escape testing and present themselves in operation. We present the first study to investigate the behaviour of flaky tests when moved to the field. Our experimentation over 52 test methods labelled as flaky provides a first confirmation that moving from the laboratory to the field, the behaviour of tests changes and, in particular, the failure frequency of intermittently failing tests can increase. More importantly, we could identify few cases of field failures that would have been hard to detect while testing in house.Source: ISTI Technical Report, ISTI-2022-TR/007, 2022
DOI: 10.32079/isti-tr-2022/007
Metrics:
See at:
CNR ExploRA
2022
Contribution to journal
Open Access
Introduction to the special issue on automation of software test and test code quality
Bertolino A., Hong S., Mathur A. P.
Source: Journal of software (Malden, Mass. Online) (2022). doi:10.1002/smr.2453
DOI: 10.1002/smr.2453
Metrics:
Bertolino A., Hong S., Mathur A. P.
Source: Journal of software (Malden, Mass. Online) (2022). doi:10.1002/smr.2453
DOI: 10.1002/smr.2453
Metrics:
See at:
ISTI Repository | Journal of Software Evolution and Process | onlinelibrary.wiley.com | CNR ExploRA
2022
Conference article
Open Access
Comparing and combining file-based selection and similarity-based prioritization towards regression test orchestration
Greca R., Miranda B., Gligoric M., Bertolino A.
Test case selection (TCS) and test case prioritization (TCP) techniques can reduce time to detect the first test failure. Although these techniques have been extensively studied in combination and isolation, they have not been compared one against the other. In this paper, we perform an empirical study directly comparing TCS and TCP approaches, represented by the tools Ekstazi and FAST, respectively. Furthermore, we develop the first combination, named Fastazi, of file-based TCS and similarity-based TCP and evaluate its benefit and cost against each individual technique. We performed our experiments using 12 Java-based open-source projects. Our results show that, in the median case, the combined approach detects the first failure nearly two times faster than either Ekstazi alone (with random test ordering) or FAST alone (without TCS). Statistical analysis shows that the effectiveness of Fastazi is higher than that of Ekstazi, which in turn is higher than that of FAST. On the other hand, FAST adds the least overhead to testing time, while the difference between the additional time needed by Ekstazi and Fastazi is negligible. Fastazi can also improve failure detection in scenarios where the time available for testing is restricted. CCS CONCEPTS o Software and its engineering ->Software testing and debugging.Source: AST 2022 - 3rd IEEE/ACM International Conference on Automation of Software Test, pp. 115–125, Pittsburgh, USA, 17-18/05/2022
DOI: 10.1145/3524481.3527223
Metrics:
Greca R., Miranda B., Gligoric M., Bertolino A.
Test case selection (TCS) and test case prioritization (TCP) techniques can reduce time to detect the first test failure. Although these techniques have been extensively studied in combination and isolation, they have not been compared one against the other. In this paper, we perform an empirical study directly comparing TCS and TCP approaches, represented by the tools Ekstazi and FAST, respectively. Furthermore, we develop the first combination, named Fastazi, of file-based TCS and similarity-based TCP and evaluate its benefit and cost against each individual technique. We performed our experiments using 12 Java-based open-source projects. Our results show that, in the median case, the combined approach detects the first failure nearly two times faster than either Ekstazi alone (with random test ordering) or FAST alone (without TCS). Statistical analysis shows that the effectiveness of Fastazi is higher than that of Ekstazi, which in turn is higher than that of FAST. On the other hand, FAST adds the least overhead to testing time, while the difference between the additional time needed by Ekstazi and Fastazi is negligible. Fastazi can also improve failure detection in scenarios where the time available for testing is restricted. CCS CONCEPTS o Software and its engineering ->Software testing and debugging.Source: AST 2022 - 3rd IEEE/ACM International Conference on Automation of Software Test, pp. 115–125, Pittsburgh, USA, 17-18/05/2022
DOI: 10.1145/3524481.3527223
Metrics:
See at:
ISTI Repository | ZENODO | ieeexplore.ieee.org | CNR ExploRA
2022
Conference article
Open Access
Testing non-testable programs using association rules
Bertolino A., Cruciani E., Miranda B., Verdecchia R.
We propose a novel scalable approach for testing non-testable programs denoted as ARMED testing. The approach leverages efficient Association Rules Mining algorithms to determine relevant implication relations among features and actions observed while the system is in operation. These relations are used as the specification of positive and negative tests, allowing for identifying plausible or suspicious behaviors: for those cases when oracles are inherently unknownable, such as in social testing, ARMED testing introduces the novel concept of testing for plausibility. To illustrate the approach we walk-through an application example.Source: AST 2022 - 3rd ACM/IEEE International Conference on Automation of Software Test, pp. 87–91, Pittsburgh, USA, 17-18/05/2022
DOI: 10.1145/3524481.3527238
Metrics:
Bertolino A., Cruciani E., Miranda B., Verdecchia R.
We propose a novel scalable approach for testing non-testable programs denoted as ARMED testing. The approach leverages efficient Association Rules Mining algorithms to determine relevant implication relations among features and actions observed while the system is in operation. These relations are used as the specification of positive and negative tests, allowing for identifying plausible or suspicious behaviors: for those cases when oracles are inherently unknownable, such as in social testing, ARMED testing introduces the novel concept of testing for plausibility. To illustrate the approach we walk-through an application example.Source: AST 2022 - 3rd ACM/IEEE International Conference on Automation of Software Test, pp. 87–91, Pittsburgh, USA, 17-18/05/2022
DOI: 10.1145/3524481.3527238
Metrics:
See at:
ISTI Repository | ieeexplore.ieee.org | CNR ExploRA
2022
Conference article
Open Access
Self-adaptive testing in the field: are we there yet?
Silva S., Bertolino A., Pelliccione P.
Testing in the field is gaining momentum, as a means to detect those failures that escape in-house testing by continuing the testing even while a system is operating in production. Among several approaches that are proposed, this paper focuses on the important notion of self-adaptivity of testing in the field, as such techniques need to adapt in many ways their strategy to the context and the emerging behaviors of the system under test. In this work, we investigate the topic by conducting a scoping review of the literature on self-adaptive testing in the field. We rely on a taxonomy organized in some categories that include the object to adapt, the adaptation trigger, the temporal characteristics, the realization issues, the interaction concerns, the type of field-based approach, and the impact/cost. Our study sheds light on self-adaptive testing in the field by identifying related key concepts and key characteristics and extracting some knowledge gaps to better guide future research.Source: SEAMS 2022 - 17th Symposium on Software Engineering for Adaptive and Self-Managing Systems, pp. 58–69, Pittsburgh, USA, 18-20/05/2022
Silva S., Bertolino A., Pelliccione P.
Testing in the field is gaining momentum, as a means to detect those failures that escape in-house testing by continuing the testing even while a system is operating in production. Among several approaches that are proposed, this paper focuses on the important notion of self-adaptivity of testing in the field, as such techniques need to adapt in many ways their strategy to the context and the emerging behaviors of the system under test. In this work, we investigate the topic by conducting a scoping review of the literature on self-adaptive testing in the field. We rely on a taxonomy organized in some categories that include the object to adapt, the adaptation trigger, the temporal characteristics, the realization issues, the interaction concerns, the type of field-based approach, and the impact/cost. Our study sheds light on self-adaptive testing in the field by identifying related key concepts and key characteristics and extracting some knowledge gaps to better guide future research.Source: SEAMS 2022 - 17th Symposium on Software Engineering for Adaptive and Self-Managing Systems, pp. 58–69, Pittsburgh, USA, 18-20/05/2022
See at:
ISTI Repository | CNR ExploRA | www.computer.org
2021
Journal article
Open Access
A survey of field-based testing techniques
Bertolino A., Braione P., De Angelis G., Gazzola L., Kifetew F., Mariani L., Orrù M., Pezzè M., Pietrantuono R., Russo S., Tonella P.
Field testing refers to testing techniques that operate in the field to reveal those faults that escape in-house testing. Field testing techniques are becoming increasingly popular with the growing complexity of contemporary software systems. In this paper, we present the first systematic survey of field testing approaches over a body of 80 collected studies, and propose their categorization based on the environment and the system on which field testing is performed. We discuss four research questions addressing how software is tested in the field, what is tested in the field, which are the requirements, and how field tests are managed, and identify many challenging research directions.Source: ACM computing surveys (Online) 54 (2021). doi:10.1145/3447240
DOI: 10.1145/3447240
Metrics:
Bertolino A., Braione P., De Angelis G., Gazzola L., Kifetew F., Mariani L., Orrù M., Pezzè M., Pietrantuono R., Russo S., Tonella P.
Field testing refers to testing techniques that operate in the field to reveal those faults that escape in-house testing. Field testing techniques are becoming increasingly popular with the growing complexity of contemporary software systems. In this paper, we present the first systematic survey of field testing approaches over a body of 80 collected studies, and propose their categorization based on the environment and the system on which field testing is performed. We discuss four research questions addressing how software is tested in the field, what is tested in the field, which are the requirements, and how field tests are managed, and identify many challenging research directions.Source: ACM computing surveys (Online) 54 (2021). doi:10.1145/3447240
DOI: 10.1145/3447240
Metrics:
See at:
ISTI Repository | ISTI Repository | dl.acm.org | CNR ExploRA
2021
Conference article
Open Access
What we talk about when we talk about software test flakiness
Barboni M., Bertolino A., De Angelis G.
Software test flakiness is drawing increasing interest among both academic researchers and practitioners. In this work we report our findings from a scoping review of white and grey literature, highlighting variations across flaky tests key concepts. Our study clearly indicates the need of a unifying definition as well as of a more comprehensive analysis for establishing a conceptual map that can better guide future research.Source: QUATIC 2021 - 14th International Conference Quality of Information and Communications Technology, pp. 29–39, Algarve, Portugal and Online, 8-11/9/2021
DOI: 10.1007/978-3-030-85347-1_3
Metrics:
Barboni M., Bertolino A., De Angelis G.
Software test flakiness is drawing increasing interest among both academic researchers and practitioners. In this work we report our findings from a scoping review of white and grey literature, highlighting variations across flaky tests key concepts. Our study clearly indicates the need of a unifying definition as well as of a more comprehensive analysis for establishing a conceptual map that can better guide future research.Source: QUATIC 2021 - 14th International Conference Quality of Information and Communications Technology, pp. 29–39, Algarve, Portugal and Online, 8-11/9/2021
DOI: 10.1007/978-3-030-85347-1_3
Metrics:
See at:
ISTI Repository | link.springer.com | CNR ExploRA
2020
Report
Open Access
Know your neighbor: fast static prediction of test flakiness
Bertolino A., Cruciani E., Miranda B., Verdecchia R.
Flaky tests plague regression testing in Continuous Integration environments by slowing down change releases, wasting development effort, and also eroding testers trust in the test process. We present FLAST, the rst static approach to akiness detection using test code similarity. Our extensive evaluation on 24 projects taken from repositories used in three previous studies showed that FLAST can identify aky tests with up to 0.98 Median and 0.92 Mean precision. For six of those projects it could already yield 0.98 average precision values with a training set containing less than 100 tests. Besides, where known aky tests are classied according to their causes, the same approach can also predict a aky test category with alike precision values. The cost of the approach is negligible: the average train time over a dataset of 1,700 test methods is less than one second, while the average prediction time for a new test is less than one millisecond.Source: ISTI Technical Reports 001/2020, 2020, 2020
DOI: 10.32079/isti-tr-2020/001
Metrics:
Bertolino A., Cruciani E., Miranda B., Verdecchia R.
Flaky tests plague regression testing in Continuous Integration environments by slowing down change releases, wasting development effort, and also eroding testers trust in the test process. We present FLAST, the rst static approach to akiness detection using test code similarity. Our extensive evaluation on 24 projects taken from repositories used in three previous studies showed that FLAST can identify aky tests with up to 0.98 Median and 0.92 Mean precision. For six of those projects it could already yield 0.98 average precision values with a training set containing less than 100 tests. Besides, where known aky tests are classied according to their causes, the same approach can also predict a aky test category with alike precision values. The cost of the approach is negligible: the average train time over a dataset of 1,700 test methods is less than one second, while the average prediction time for a new test is less than one millisecond.Source: ISTI Technical Reports 001/2020, 2020, 2020
DOI: 10.32079/isti-tr-2020/001
Metrics:
See at:
ISTI Repository | CNR ExploRA
2020
Journal article
Closed Access
Digital persona portrayal: identifying pluridentity vulnerabilities in digital life
Olivero M. A., Bertolino A., Dominguez-Mayo F. J., Escalona M. J., Matteucci I.
The increasing use of the Internet for social purposes enriches the data available online about all of us and promotes the concept of the Digital Persona. Actually, most of us are represented online by more than one identity, what we define here as a Pluridentity. This trend brings increased risks: it is well known that the security of a Digital Persona can be exploited if its data and security are not effectively managed. In this paper, we focus specifically on a new type of digital attack that can be perpetrated by combining pieces of data belonging to one same Pluridentity in order to profile their target. Some victims can be so accurately depicted when looking at their Pluridentity that by using the gathered information attackers can execute very personalized social engineering attacks, or even bypass otherwise safe security mechanisms. We characterize these Pluridentity attacks as a security issue of a virtual System of Systems, whose constituent systems are the individual identities and the humans themselves. We present a strategy to identify vulnerabilities caused by overexposure due to the combination of data from the constituent identities of a Pluridentity. To this end we introduce the Digital Persona Portrayal Metamodel, and the related Digital Pluridentity Persona Portrayal Analysis process that supports the architecting of data from different identities: such model and process can be used to identify the vulnerabilities of a Pluridentity due to its exploitation as a System of Systems. The approach has been validated on the Pluridentities of seventeen candidates selected from a data leak, by retrieving the data of their Digital Personae, and matching them against the security mechanisms of their Pluridentities. After analyzing the results for some of the analyzed subjects we could detect several vulnerabilities.Source: Journal of Information Security and Applications 52 (2020). doi:10.1016/j.jisa.2020.102492
DOI: 10.1016/j.jisa.2020.102492
Metrics:
Olivero M. A., Bertolino A., Dominguez-Mayo F. J., Escalona M. J., Matteucci I.
The increasing use of the Internet for social purposes enriches the data available online about all of us and promotes the concept of the Digital Persona. Actually, most of us are represented online by more than one identity, what we define here as a Pluridentity. This trend brings increased risks: it is well known that the security of a Digital Persona can be exploited if its data and security are not effectively managed. In this paper, we focus specifically on a new type of digital attack that can be perpetrated by combining pieces of data belonging to one same Pluridentity in order to profile their target. Some victims can be so accurately depicted when looking at their Pluridentity that by using the gathered information attackers can execute very personalized social engineering attacks, or even bypass otherwise safe security mechanisms. We characterize these Pluridentity attacks as a security issue of a virtual System of Systems, whose constituent systems are the individual identities and the humans themselves. We present a strategy to identify vulnerabilities caused by overexposure due to the combination of data from the constituent identities of a Pluridentity. To this end we introduce the Digital Persona Portrayal Metamodel, and the related Digital Pluridentity Persona Portrayal Analysis process that supports the architecting of data from different identities: such model and process can be used to identify the vulnerabilities of a Pluridentity due to its exploitation as a System of Systems. The approach has been validated on the Pluridentities of seventeen candidates selected from a data leak, by retrieving the data of their Digital Personae, and matching them against the security mechanisms of their Pluridentities. After analyzing the results for some of the analyzed subjects we could detect several vulnerabilities.Source: Journal of Information Security and Applications 52 (2020). doi:10.1016/j.jisa.2020.102492
DOI: 10.1016/j.jisa.2020.102492
Metrics:
See at:
Journal of Information Security and Applications | CNR ExploRA | www.sciencedirect.com
2020
Journal article
Open Access
RETORCH: an approach for resource-aware orchestration of end-to-end test cases
Augusto C., Moran J., Bertolino A., De La Riva C., Tuya J.
Continuous integration practice mandates to continuously introduce incremental changes into code, but doing so may introduce new faults too. These faults could be detected automatically through regression testing, but this practice becomes prohibitive as the cost of executing the tests grows. This problem is preponderant in end-to-end testing where the whole system is requested for test execution. However, some of these test cases could be executed with fewer resources (e.g., memory, web services, computation, Cloud instances, among others), by deploying only the subsystems needed by each test. This paper is focused on the optimization of the resources employed in end-to-end testing by means of a resource-aware test orchestration technique in the context of continuous integration practices in the Cloud. The RETORCH approach proposes a novel way to identify the resources required by end-to-end test cases and to use this information to group together those tests requiring equivalent resources. Besides, the approach proposes to deploy the grouped tests in isolated and elastic environments, so that their execution can be scheduled in parallel on several machines. RETORCH is exemplified with a real-world application, and its performance evaluation shows promising savings in terms of resource usage and time.Source: Software quality journal (2020): 1–25. doi:10.1007/s11219-020-09505-2
DOI: 10.1007/s11219-020-09505-2
Project(s): ELASTEST
Metrics:
Augusto C., Moran J., Bertolino A., De La Riva C., Tuya J.
Continuous integration practice mandates to continuously introduce incremental changes into code, but doing so may introduce new faults too. These faults could be detected automatically through regression testing, but this practice becomes prohibitive as the cost of executing the tests grows. This problem is preponderant in end-to-end testing where the whole system is requested for test execution. However, some of these test cases could be executed with fewer resources (e.g., memory, web services, computation, Cloud instances, among others), by deploying only the subsystems needed by each test. This paper is focused on the optimization of the resources employed in end-to-end testing by means of a resource-aware test orchestration technique in the context of continuous integration practices in the Cloud. The RETORCH approach proposes a novel way to identify the resources required by end-to-end test cases and to use this information to group together those tests requiring equivalent resources. Besides, the approach proposes to deploy the grouped tests in isolated and elastic environments, so that their execution can be scheduled in parallel on several machines. RETORCH is exemplified with a real-world application, and its performance evaluation shows promising savings in terms of resource usage and time.Source: Software quality journal (2020): 1–25. doi:10.1007/s11219-020-09505-2
DOI: 10.1007/s11219-020-09505-2
Project(s): ELASTEST
Metrics:
See at:
Software Quality Journal | Repositorio Institucional de la Universidad de Oviedo | Software Quality Journal | link.springer.com | CNR ExploRA
2020
Journal article
Open Access
FlakyLoc: Flakiness localization for reliable test suites in web applications
Moran J., Augusto C., Bertolino A., De La Riva C., Tuya J.
Web application testing is a great challenge due to the management of complex asynchronous communications, the concurrency between the clients-servers, and the heterogeneity of resources employed. It is difficult to ensure that a test case is re-running in the same conditions because it can be executed in undesirable ways according to several environmental factors that are not easy to fine-grain control such as network bottlenecks, memory issues or screen resolution. These environmental factors can cause flakiness, which occurs when the same test case sometimes obtains one test outcome and other times another outcome in the same application due to the execution of environmental factors. The tester usually stops relying on flaky test cases because their outcome varies during the re-executions. To fix and reduce the flakiness it is very important to locate and understand which environmental factors cause the flakiness. This paper is focused on the localization of the root cause of flakiness in web applications based on the characterization of the different environmental factors that are not controlled during testing. The root cause of flakiness is located by means of spectrum-based localization techniques that analyse the test execution under different combinations of the environmental factors that can trigger the flakiness. This technique is evaluated with an educational web platform called FullTeaching. As a result, our technique was able to locate automatically the root cause of flakiness and provide enough information to both understand it and fix it.Source: Journal of web engineering 19 (2020): 267–296. doi:10.13052/jwe1540-9589.1927
DOI: 10.13052/jwe1540-9589.1927
Project(s): ELASTEST
Metrics:
Moran J., Augusto C., Bertolino A., De La Riva C., Tuya J.
Web application testing is a great challenge due to the management of complex asynchronous communications, the concurrency between the clients-servers, and the heterogeneity of resources employed. It is difficult to ensure that a test case is re-running in the same conditions because it can be executed in undesirable ways according to several environmental factors that are not easy to fine-grain control such as network bottlenecks, memory issues or screen resolution. These environmental factors can cause flakiness, which occurs when the same test case sometimes obtains one test outcome and other times another outcome in the same application due to the execution of environmental factors. The tester usually stops relying on flaky test cases because their outcome varies during the re-executions. To fix and reduce the flakiness it is very important to locate and understand which environmental factors cause the flakiness. This paper is focused on the localization of the root cause of flakiness in web applications based on the characterization of the different environmental factors that are not controlled during testing. The root cause of flakiness is located by means of spectrum-based localization techniques that analyse the test execution under different combinations of the environmental factors that can trigger the flakiness. This technique is evaluated with an educational web platform called FullTeaching. As a result, our technique was able to locate automatically the root cause of flakiness and provide enough information to both understand it and fix it.Source: Journal of web engineering 19 (2020): 267–296. doi:10.13052/jwe1540-9589.1927
DOI: 10.13052/jwe1540-9589.1927
Project(s): ELASTEST
Metrics:
See at:
Repositorio Institucional de la Universidad de Oviedo | journals.riverpublishers.com | ISTI Repository | CNR ExploRA | Journal of Web Engineering
2020
Conference article
Open Access
EDUFYSoS: A Factory of Educational System of Systems Case Studies
Bertolino A., De Angelis G., Lonetti F., De Oliveira Neves V., Olivero M. A.
We propose a factory of educational System of Systems (SoS) case studies that can be used for evaluating SoS research results, in particular in SoS testing. The factory includes a first set of constituent systems that can collaborate within different SoS architectures to accomplish different missions. In the paper, we introduce three possible SoSs and outline their missions. For more detailed descriptions, diagrams and the source code, we refer to the online repository of EDUFYSoS. The factory is meant to provide an extensible playground, which we aim to grow to include more systems and other missions with the support of the community.Source: IEEE 15th Int. Conf. of System of Systems Engineering (SoSE), Budapest, Ungheria, 2-5/06/2020
DOI: 10.1109/sose50414.2020.9130551
Metrics:
Bertolino A., De Angelis G., Lonetti F., De Oliveira Neves V., Olivero M. A.
We propose a factory of educational System of Systems (SoS) case studies that can be used for evaluating SoS research results, in particular in SoS testing. The factory includes a first set of constituent systems that can collaborate within different SoS architectures to accomplish different missions. In the paper, we introduce three possible SoSs and outline their missions. For more detailed descriptions, diagrams and the source code, we refer to the online repository of EDUFYSoS. The factory is meant to provide an extensible playground, which we aim to grow to include more systems and other missions with the support of the community.Source: IEEE 15th Int. Conf. of System of Systems Engineering (SoSE), Budapest, Ungheria, 2-5/06/2020
DOI: 10.1109/sose50414.2020.9130551
Metrics:
See at:
idUS. Depósito de Investigación Universidad de Sevilla | ieeexplore.ieee.org | ISTI Repository | CNR ExploRA | doi.org
2020
Journal article
Open Access
Testing relative to usage scope: revisiting software coverage criteria
Miranda B., Bertolino A.
Coverage criteria provide a useful and widely used means to guide software testing; however, indiscriminately pursuing full coverage may not always be convenient or meaningful, as not all entities are of interest in any usage context. We aim at introducing a more meaningful notion of coverage that takes into account how the software is going to be used. Entities that are not going to be exercised by the user should not contribute to the coverage ratio. We revisit the definition of coverage measures, introducing a notion of relative coverage. According to this notion, we provide a definition and a theoretical framework of relative coverage, within which we discuss implications on testing theory and practice. Through the evaluation of three different instances of relative coverage, we could observe that relative coverage measures provide a more effective strategy than traditional ones: we could reach higher coverage measures, and test cases selected by relative coverage could achieve higher reliability. We hint at several other useful implications of relative coverage notion on different aspects of software testing.Source: ACM transactions on software engineering and methodology 29 (2020). doi:10.1145/3389126
DOI: 10.1145/3389126
Metrics:
Miranda B., Bertolino A.
Coverage criteria provide a useful and widely used means to guide software testing; however, indiscriminately pursuing full coverage may not always be convenient or meaningful, as not all entities are of interest in any usage context. We aim at introducing a more meaningful notion of coverage that takes into account how the software is going to be used. Entities that are not going to be exercised by the user should not contribute to the coverage ratio. We revisit the definition of coverage measures, introducing a notion of relative coverage. According to this notion, we provide a definition and a theoretical framework of relative coverage, within which we discuss implications on testing theory and practice. Through the evaluation of three different instances of relative coverage, we could observe that relative coverage measures provide a more effective strategy than traditional ones: we could reach higher coverage measures, and test cases selected by relative coverage could achieve higher reliability. We hint at several other useful implications of relative coverage notion on different aspects of software testing.Source: ACM transactions on software engineering and methodology 29 (2020). doi:10.1145/3389126
DOI: 10.1145/3389126
Metrics:
See at:
dl.acm.org | ISTI Repository | CNR ExploRA | ACM Transactions on Software Engineering and Methodology
2020
Journal article
Restricted
DevOpRET: continuous reliability testing in DevOps
Bertolino A., De Angelis G., Guerriero A., Miranda B., Pietrantuono R., Russo S.
To enter the production stage, in DevOps practices candidate software releases have to pass quality gates, where they are assessed to meet established target values for key indicators of interest. We believe software reliability should be an important such indicator, as it greatly contributes to the end-user satisfaction. We proposeDevOpRET, an approach for reliability testing as part of the acceptance testing stage in DevOps.DevOpRETrelies on operational-profile-based testing, a common reliability assessment technique.DevOpRETleverages usage and failure data monitored in operations to continuously refine its estimate. We evaluate accuracy and efficiency ofDevOpRETthrough controlled experiments with a real-world open source platform and with a microservice architectures benchmark. The results show thatDevOpRETprovides accurate and efficient estimates of the true reliability over subsequent DevOps cycles.Source: Journal of software (Malden, Mass. Online) (2020). doi:10.1002/smr.2298
DOI: 10.1002/smr.2298
Metrics:
Bertolino A., De Angelis G., Guerriero A., Miranda B., Pietrantuono R., Russo S.
To enter the production stage, in DevOps practices candidate software releases have to pass quality gates, where they are assessed to meet established target values for key indicators of interest. We believe software reliability should be an important such indicator, as it greatly contributes to the end-user satisfaction. We proposeDevOpRET, an approach for reliability testing as part of the acceptance testing stage in DevOps.DevOpRETrelies on operational-profile-based testing, a common reliability assessment technique.DevOpRETleverages usage and failure data monitored in operations to continuously refine its estimate. We evaluate accuracy and efficiency ofDevOpRETthrough controlled experiments with a real-world open source platform and with a microservice architectures benchmark. The results show thatDevOpRETprovides accurate and efficient estimates of the true reliability over subsequent DevOps cycles.Source: Journal of software (Malden, Mass. Online) (2020). doi:10.1002/smr.2298
DOI: 10.1002/smr.2298
Metrics:
See at:
Journal of Software Evolution and Process | onlinelibrary.wiley.com | CNR ExploRA
2020
Conference article
Open Access
Run Java Applications and Test Them In-Vivo Meantime
Bertolino A., De Angelis G., Miranda B., Tonella P.
The outcome of test case execution depends on the state of the object under test. While testers can carefully choose meaningful and representative object states for test execution, it is unaffordable to cover the combinatorial space of possible object states exhaustively. An appealing option is to delegate part of the testing activities to the runtime and to execute test cases in the field whenever a new or uncommon state is observed. We have designed and developed Groucho, a framework for in-vivo testing of Java applications. Among the challenges that we faced, the most important ones are isolation of the test session from the user session and minimal performance overhead. Experimental results show that if the activation probability is kept reasonably small (e.g., 10 ^- 4), the impact of the framework is imperceptible(i.e., either statistically insignificant or with a negligible effect size).Source: IEEE 13th International Conference on Software Testing, Validation and Verification (ICST), pp. 454–459, Porto, 24-28/10/2020
DOI: 10.1109/icst46399.2020.00061
DOI: 10.5281/zenodo.5054694
DOI: 10.5281/zenodo.5054693
Project(s): ELASTEST, PRECRIME
Metrics:
Bertolino A., De Angelis G., Miranda B., Tonella P.
The outcome of test case execution depends on the state of the object under test. While testers can carefully choose meaningful and representative object states for test execution, it is unaffordable to cover the combinatorial space of possible object states exhaustively. An appealing option is to delegate part of the testing activities to the runtime and to execute test cases in the field whenever a new or uncommon state is observed. We have designed and developed Groucho, a framework for in-vivo testing of Java applications. Among the challenges that we faced, the most important ones are isolation of the test session from the user session and minimal performance overhead. Experimental results show that if the activation probability is kept reasonably small (e.g., 10 ^- 4), the impact of the framework is imperceptible(i.e., either statistically insignificant or with a negligible effect size).Source: IEEE 13th International Conference on Software Testing, Validation and Verification (ICST), pp. 454–459, Porto, 24-28/10/2020
DOI: 10.1109/icst46399.2020.00061
DOI: 10.5281/zenodo.5054694
DOI: 10.5281/zenodo.5054693
Project(s): ELASTEST
, PRECRIME Metrics:
See at:
ZENODO | ZENODO | ieeexplore.ieee.org | CNR ExploRA | zenodo.org | doi.org
2020
Conference article
Open Access
JTeC: A Large Collection of Java Test Classes for Test Code Analysis and Processing
Corò F., Verdecchia R., Cruciani E., Miranda B., Bertolino A.
The recent push towards test automation and test-driven development continues to scale up the dimensions of test code that needs to be maintained, analysed, and processed side-by-side with production code. As a consequence, on the one side regression testing techniques, e.g., for test suite prioritization or test case selection, capable to handle such large-scale test suites become indispensable; on the other side, as test code exposes own characteristics, specific techniques for its analysis and refactoring are actively sought. We present JTeC, a large-scale dataset of test cases that researchers can use for benchmarking the above techniques or any other type of tool expressly targeting test code. JTeC collects more than 2.5M test classes belonging to 31K+ GitHub projects and summing up to more than 430 Million SLOCs of ready-to-use real-world test code.Source: 2020 IEEE/ACM 17th International Conference on Mining Software Repositories (MSR), pp. 578–582, 29-30/06/2020
DOI: 10.1145/3379597.3387484
Metrics:
Corò F., Verdecchia R., Cruciani E., Miranda B., Bertolino A.
The recent push towards test automation and test-driven development continues to scale up the dimensions of test code that needs to be maintained, analysed, and processed side-by-side with production code. As a consequence, on the one side regression testing techniques, e.g., for test suite prioritization or test case selection, capable to handle such large-scale test suites become indispensable; on the other side, as test code exposes own characteristics, specific techniques for its analysis and refactoring are actively sought. We present JTeC, a large-scale dataset of test cases that researchers can use for benchmarking the above techniques or any other type of tool expressly targeting test code. JTeC collects more than 2.5M test classes belonging to 31K+ GitHub projects and summing up to more than 430 Million SLOCs of ready-to-use real-world test code.Source: 2020 IEEE/ACM 17th International Conference on Mining Software Repositories (MSR), pp. 578–582, 29-30/06/2020
DOI: 10.1145/3379597.3387484
Metrics:
See at:
dl.acm.org | hal.archives-ouvertes.fr | ISTI Repository | CNR ExploRA | NARCIS | Hyper Article en Ligne | NARCIS
2020
Conference article
Restricted
What is the Vocabulary of Flaky Tests?
Pinto G., Miranda B., Dissanayake S., D'Amorim M., Treude C., Bertolino A.
Flaky tests are tests whose outcomes are non-deterministic. Despite the recent research activity on this topic, no effort has been made on understanding the vocabulary of flaky tests. This work proposes to automatically classify tests as flaky or not based on their vocabulary. Static classification of flaky tests is important, for example, to detect the introduction of flaky tests and to search for flaky tests after they are introduced in regression test suites. We evaluated performance of various machine learning algorithms to solve this problem. We constructed a data set of flaky and non-flaky tests by running every test case, in a set of 64k tests, 100 times (6.4 million test executions). We then used machine learning techniques on the resulting data set to predict which tests are flaky from their source code. Based on features, such as counting stemmed tokens extracted from source code identifiers, we achieved an F-measure of 0.95 for the identification of flaky tests. The best prediction performance was obtained when using Random Forest and Support Vector Machines. In terms of the code identifiers that are most strongly associated with test flakiness, we noted that job, action, and services are commonly associated with flaky tests. Overall, our results provides initial yet strong evidence that static detection of flaky tests is effective.Source: 2020 IEEE/ACM 17th International Conference on Mining Software Repositories (MSR), pp. 492–502, 29-30/06/2020
DOI: 10.1145/3379597.3387482
Metrics:
Pinto G., Miranda B., Dissanayake S., D'Amorim M., Treude C., Bertolino A.
Flaky tests are tests whose outcomes are non-deterministic. Despite the recent research activity on this topic, no effort has been made on understanding the vocabulary of flaky tests. This work proposes to automatically classify tests as flaky or not based on their vocabulary. Static classification of flaky tests is important, for example, to detect the introduction of flaky tests and to search for flaky tests after they are introduced in regression test suites. We evaluated performance of various machine learning algorithms to solve this problem. We constructed a data set of flaky and non-flaky tests by running every test case, in a set of 64k tests, 100 times (6.4 million test executions). We then used machine learning techniques on the resulting data set to predict which tests are flaky from their source code. Based on features, such as counting stemmed tokens extracted from source code identifiers, we achieved an F-measure of 0.95 for the identification of flaky tests. The best prediction performance was obtained when using Random Forest and Support Vector Machines. In terms of the code identifiers that are most strongly associated with test flakiness, we noted that job, action, and services are commonly associated with flaky tests. Overall, our results provides initial yet strong evidence that static detection of flaky tests is effective.Source: 2020 IEEE/ACM 17th International Conference on Mining Software Repositories (MSR), pp. 492–502, 29-30/06/2020
DOI: 10.1145/3379597.3387482
Metrics:
See at:
dl.acm.org | doi.org | CNR ExploRA