not yet published
Conference article
Restricted
AdapTSoS: adaptive testing of systems of systems
Santos Da Silva Samira, Lonetti Francesca, Bertolino Antonia
Testing of System of Systems (SoS) presents many challenges due to the dynamic and continuous evolution of its Constituent Systems (CSs). Existing SoS testing strategies support the definition of test cases but fail to address the dynamic changes of CSs. New test cases need to be added or existing ones must be updated to cope with the variable functionalities of CSs. Adaptive testing is a strategy to dynamically adapt a set of test configurations or test cases to changes in the system under test over time. The main idea of this paper is to provide a new adaptive testing strategy, AdapTSoS, for the adaptive testing of SoSs in order to deal with the continuous changes of variable functionalities of the CSs. The proposed strategy leverages a variability model representing the variable functionalities of the CSs and a Discrete Time Markov Chain simulating the SoS changes over time, to then provide a set of adaptive test configurations to be executed on the SoS. We illustrate the application of AdapTSoS to an existing SoS example and discuss the benefits provided by the proposed approach.Source: LECTURE NOTES IN COMPUTER SCIENCE, vol. 16315. Lisbon, Portugal, 3-5 september 2025
Santos Da Silva Samira, Lonetti Francesca, Bertolino Antonia
Testing of System of Systems (SoS) presents many challenges due to the dynamic and continuous evolution of its Constituent Systems (CSs). Existing SoS testing strategies support the definition of test cases but fail to address the dynamic changes of CSs. New test cases need to be added or existing ones must be updated to cope with the variable functionalities of CSs. Adaptive testing is a strategy to dynamically adapt a set of test configurations or test cases to changes in the system under test over time. The main idea of this paper is to provide a new adaptive testing strategy, AdapTSoS, for the adaptive testing of SoSs in order to deal with the continuous changes of variable functionalities of the CSs. The proposed strategy leverages a variability model representing the variable functionalities of the CSs and a Discrete Time Markov Chain simulating the SoS changes over time, to then provide a set of adaptive test configurations to be executed on the SoS. We illustrate the application of AdapTSoS to an existing SoS example and discuss the benefits provided by the proposed approach.Source: LECTURE NOTES IN COMPUTER SCIENCE, vol. 16315. Lisbon, Portugal, 3-5 september 2025
2026
Journal article
Open Access
Recent developments in software engineering for systems-of-systems and software ecosystems
Lonetti Francesca, Bertolino Antonia, Antonino Pablo, Bae Doo-Hwan
The increasing scale, distribution, and interconnection of software-intensive systems continue to change the landscape of software engineering research and practice, bringing the diffusion of two similar paradigms: Systems-of-Systems (SoS) and Software Ecosystems (SECO). SoS are characterized by the integration of operationally and managerially independent systems that join together to accomplish a common mission. SoS are central to domains such as transportation, healthcare, defense, smart cities, and industrial automation where heterogeneous systems must cooperate, exchange information, and adapt to evolving missions. On the other hand, SECO describe environments in which a platform and its surrounding network of developers, partners, and organizations co-create software offerings. In SECO, technical artifacts interact with economic and social processes. Modern digital platforms, mobile operating systems, cloud services, and enterprise platforms leverage the capabilities of their ecosystems. This editorial brings together perspectives that explore the shared challenges and complementary insights of SoS and SECO research, aiming to foster a richer understanding of complex software-intensive systems and highlight new opportunities for collaboration across communities. From the long-running, successful series of the International Workshop on Software Engineering for Systems-of-Systems and Software Ecosystems (SESoS), co-located with the IEEE/ACM International Conference on Software Engineering (ICSE), we present this special issue of the Journal of Systems and Software on the topics of SESoS 2024 in Lisbon, Portugal. From a total of 18 submissions, 7 articles were accepted in this special issue. The articles in this collection address fundamental questions of SoS and SECO, including the evolution of functional relations and experimentation practices, the key factors affecting developer experience, also related to women’s inclusion, as well as the adoption of GenAI-driven approaches for vulnerability fixing. These articles offer updates on current advances of SoS and SECO engineering to researchers and practitioners, highlighting opportunities for future research.Source: THE JOURNAL OF SYSTEMS AND SOFTWARE, vol. 236 (issue 112819)
DOI: 10.1016/j.jss.2026.112819
DOI: 10.1016/j.jss.2026.112814
Metrics:
Lonetti Francesca, Bertolino Antonia, Antonino Pablo, Bae Doo-Hwan
The increasing scale, distribution, and interconnection of software-intensive systems continue to change the landscape of software engineering research and practice, bringing the diffusion of two similar paradigms: Systems-of-Systems (SoS) and Software Ecosystems (SECO). SoS are characterized by the integration of operationally and managerially independent systems that join together to accomplish a common mission. SoS are central to domains such as transportation, healthcare, defense, smart cities, and industrial automation where heterogeneous systems must cooperate, exchange information, and adapt to evolving missions. On the other hand, SECO describe environments in which a platform and its surrounding network of developers, partners, and organizations co-create software offerings. In SECO, technical artifacts interact with economic and social processes. Modern digital platforms, mobile operating systems, cloud services, and enterprise platforms leverage the capabilities of their ecosystems. This editorial brings together perspectives that explore the shared challenges and complementary insights of SoS and SECO research, aiming to foster a richer understanding of complex software-intensive systems and highlight new opportunities for collaboration across communities. From the long-running, successful series of the International Workshop on Software Engineering for Systems-of-Systems and Software Ecosystems (SESoS), co-located with the IEEE/ACM International Conference on Software Engineering (ICSE), we present this special issue of the Journal of Systems and Software on the topics of SESoS 2024 in Lisbon, Portugal. From a total of 18 submissions, 7 articles were accepted in this special issue. The articles in this collection address fundamental questions of SoS and SECO, including the evolution of functional relations and experimentation practices, the key factors affecting developer experience, also related to women’s inclusion, as well as the adoption of GenAI-driven approaches for vulnerability fixing. These articles offer updates on current advances of SoS and SECO engineering to researchers and practitioners, highlighting opportunities for future research.Source: THE JOURNAL OF SYSTEMS AND SOFTWARE, vol. 236 (issue 112819)
DOI: 10.1016/j.jss.2026.112819
DOI: 10.1016/j.jss.2026.112814
Metrics:
See at:
Journal of Systems and Software 
| CNR IRIS | www.sciencedirect.com | Journal of Systems and Software 
| CNR IRIS
2025
Conference article
Open Access
An adaptive testing approach based on field data
Silva S., Caldas R., Pelliccione P., Bertolino A.
The growing need to test systems post-release has led to extending testing activities into production environments, where uncertainty and dynamic conditions pose significant challenges. Field testing approaches, especially Self-Adaptive Testing in the Field (SATF), face hurdles like managing unpredictability, minimizing system overhead, and reducing human intervention, among others. Despite its importance, SATF remains underexplored in the literature. This work introduces AdapTA (Adaptive Testing Approach), a novel SATF strategy tailored for testing Body Sensor Networks (BSNs). BSNs are networks of wearable or implantable sensors designed to monitor physiological and environmental data. AdapTA employs an ex-vivo approach, using real-world data collected from the field to simulate patient behavior in in-house experiments. Field data are used to derive Discrete-Time Markov Chain (DTMC) models, which simulate patient profiles and generate test input data for the BSN. The BSN's outputs are compared against a proposed oracle to evaluate test outcomes. AdapTA's adaptive logic continuously monitors the system under test and the simulated patient, triggering adaptations as needed. Results demonstrate that AdapTA achieves greater effectiveness compared to a non-adaptive version of the proposed approach across three adaptation scenarios, emphasizing the value of its adaptive logic.DOI: 10.1109/ast66626.2025.00013
Metrics:
Silva S., Caldas R., Pelliccione P., Bertolino A.
The growing need to test systems post-release has led to extending testing activities into production environments, where uncertainty and dynamic conditions pose significant challenges. Field testing approaches, especially Self-Adaptive Testing in the Field (SATF), face hurdles like managing unpredictability, minimizing system overhead, and reducing human intervention, among others. Despite its importance, SATF remains underexplored in the literature. This work introduces AdapTA (Adaptive Testing Approach), a novel SATF strategy tailored for testing Body Sensor Networks (BSNs). BSNs are networks of wearable or implantable sensors designed to monitor physiological and environmental data. AdapTA employs an ex-vivo approach, using real-world data collected from the field to simulate patient behavior in in-house experiments. Field data are used to derive Discrete-Time Markov Chain (DTMC) models, which simulate patient profiles and generate test input data for the BSN. The BSN's outputs are compared against a proposed oracle to evaluate test outcomes. AdapTA's adaptive logic continuously monitors the system under test and the simulated patient, triggering adaptations as needed. Results demonstrate that AdapTA achieves greater effectiveness compared to a non-adaptive version of the proposed approach across three adaptation scenarios, emphasizing the value of its adaptive logic.DOI: 10.1109/ast66626.2025.00013
Metrics:
See at:
CNR IRIS | ieeexplore.ieee.org | CNR IRIS | CNR IRIS
2025
Journal article
Open Access
Different approaches for testing body sensor network applications
Silva S., Caldas R., Pelliccione P., Bertolino A.
Body Sensor Networks (BSNs) offer a cost-effective way to monitor patients’ health and detect potential risks. Despite the growing interest attracted by BSNs, there is a lack of testing approaches for them. Testing a Body Sensor Network (BSN) is challenging due to its evolving nature, the complexity of sensor scenarios and their fusion, the potential necessity of third-party testing for certification, and the need to prioritize critical failures given limited resources. This paper addresses these challenges by proposing three BSN testing approaches: PASTA, ValComb, and TransCov. These approaches share common characteristics, which are described through a general framework called GATE4BSN. PASTA simulates patients with sensors and models sensor trends using a Discrete Time Markov Chain (DTMC). ValComb explores various health conditions by considering all sensor risk level combinations, while TransCov ensures full coverage of DTMC transitions. We empirically evaluate these approaches, comparing them with a baseline approach in terms of failure detection. The results demonstrate that PASTA, ValComb, and TransCov uncover previously undetected failures in an open-source BSN and outperform the baseline approach. Statistical analysis reveals that PASTA is the most effective, while ValComb is 76 times faster than PASTA and nearly as effective.Source: THE JOURNAL OF SYSTEMS AND SOFTWARE, vol. 223 (issue 112336)
DOI: 10.1016/j.jss.2025.112336
DOI: 10.2139/ssrn.4876283
Metrics:
Silva S., Caldas R., Pelliccione P., Bertolino A.
Body Sensor Networks (BSNs) offer a cost-effective way to monitor patients’ health and detect potential risks. Despite the growing interest attracted by BSNs, there is a lack of testing approaches for them. Testing a Body Sensor Network (BSN) is challenging due to its evolving nature, the complexity of sensor scenarios and their fusion, the potential necessity of third-party testing for certification, and the need to prioritize critical failures given limited resources. This paper addresses these challenges by proposing three BSN testing approaches: PASTA, ValComb, and TransCov. These approaches share common characteristics, which are described through a general framework called GATE4BSN. PASTA simulates patients with sensors and models sensor trends using a Discrete Time Markov Chain (DTMC). ValComb explores various health conditions by considering all sensor risk level combinations, while TransCov ensures full coverage of DTMC transitions. We empirically evaluate these approaches, comparing them with a baseline approach in terms of failure detection. The results demonstrate that PASTA, ValComb, and TransCov uncover previously undetected failures in an open-source BSN and outperform the baseline approach. Statistical analysis reveals that PASTA is the most effective, while ValComb is 76 times faster than PASTA and nearly as effective.Source: THE JOURNAL OF SYSTEMS AND SOFTWARE, vol. 223 (issue 112336)
DOI: 10.1016/j.jss.2025.112336
DOI: 10.2139/ssrn.4876283
Metrics:
See at:
Journal of Systems and Software | CNR IRIS | www.sciencedirect.com | doi.org | CNR IRIS
2025
Conference article
Open Access
Energy-aware software testing
Verdecchia R., Cruciani E., Bertolino A., Miranda B.
Our planet urges for a more responsible use of its resources, and since information technology contributes substantially to the global energy consumption, software engineering research has promptly embraced this request and is actively working towards more sustainable processes. An indispensable activity in software development is testing, which is known to be very costly in terms of time and effort. On top of this, a recent study by Zaidman has shown that software testing can be a voracious energy consumer as well. In this work we introduce the very concept of energy-aware testing as the adoption of strategies designed to reduce the energy consumption of existing practices. We discuss some possible strategies and, as an example, we conduct a first study of an energy-aware variant of a simple similarity-based test prioritization approach considering both energy consumption and test suite effectiveness, which provides evidence of perceptible savings. We encourage future research in energy-aware software testing that needs to address further studies and to think up more strategies.Source: PROCEEDINGS - INTERNATIONAL CONFERENCE ON SOFTWARE ENGINEERING, pp. 101-105. Ottawa, Canada, 27 April 2025 - 03 May 2025
DOI: 10.1109/icse-nier66352.2025.00026
Metrics:
Verdecchia R., Cruciani E., Bertolino A., Miranda B.
Our planet urges for a more responsible use of its resources, and since information technology contributes substantially to the global energy consumption, software engineering research has promptly embraced this request and is actively working towards more sustainable processes. An indispensable activity in software development is testing, which is known to be very costly in terms of time and effort. On top of this, a recent study by Zaidman has shown that software testing can be a voracious energy consumer as well. In this work we introduce the very concept of energy-aware testing as the adoption of strategies designed to reduce the energy consumption of existing practices. We discuss some possible strategies and, as an example, we conduct a first study of an energy-aware variant of a simple similarity-based test prioritization approach considering both energy consumption and test suite effectiveness, which provides evidence of perceptible savings. We encourage future research in energy-aware software testing that needs to address further studies and to think up more strategies.Source: PROCEEDINGS - INTERNATIONAL CONFERENCE ON SOFTWARE ENGINEERING, pp. 101-105. Ottawa, Canada, 27 April 2025 - 03 May 2025
DOI: 10.1109/icse-nier66352.2025.00026
Metrics:
See at:
CNR IRIS | ieeexplore.ieee.org | CNR IRIS | CNR IRIS
2025
Conference article
Open Access
Software system testing assisted by large language models: an exploratory study
Augusto C., Morán J., Bertolino A., De La Riva C., Tuya J.
Large language models (LLMs) based on transformer architecture have revolutionized natural language processing (NLP), demonstrating excellent capabilities in understanding and generating human-like text. In Software Engineering, LLMs have been applied in code generation, documentation, and report writing tasks, to support the developer and reduce the amount of manual work. In Software Testing, one of the cornerstones of Software Engineering, LLMs have been explored for generating test code, test inputs, automating the oracle process or generating test scenarios. However, their application to high-level testing stages such as system testing, in which a deep knowledge of the business and the technological stack is needed, remains largely unexplored. This paper presents an exploratory study about how LLMs can support system test development. Given that LLM performance depends on input data quality, the study focuses on how to query general purpose LLMs to first obtain test scenarios and then derive test cases from them. The study evaluates two popular LLMs (GPT-4o and GPT-4o-mini), using as a benchmark a European project demonstrator. The study compares two different prompt strategies and employs well-established prompt patterns, showing promising results as well as room for improvement in the application of LLMs to support system testing.Source: LECTURE NOTES IN COMPUTER SCIENCE, vol. 15383, pp. 239-255. London, UK, 30/10-1/11/2024
DOI: 10.1007/978-3-031-80889-0_17
Metrics:
Augusto C., Morán J., Bertolino A., De La Riva C., Tuya J.
Large language models (LLMs) based on transformer architecture have revolutionized natural language processing (NLP), demonstrating excellent capabilities in understanding and generating human-like text. In Software Engineering, LLMs have been applied in code generation, documentation, and report writing tasks, to support the developer and reduce the amount of manual work. In Software Testing, one of the cornerstones of Software Engineering, LLMs have been explored for generating test code, test inputs, automating the oracle process or generating test scenarios. However, their application to high-level testing stages such as system testing, in which a deep knowledge of the business and the technological stack is needed, remains largely unexplored. This paper presents an exploratory study about how LLMs can support system test development. Given that LLM performance depends on input data quality, the study focuses on how to query general purpose LLMs to first obtain test scenarios and then derive test cases from them. The study evaluates two popular LLMs (GPT-4o and GPT-4o-mini), using as a benchmark a European project demonstrator. The study compares two different prompt strategies and employs well-established prompt patterns, showing promising results as well as room for improvement in the application of LLMs to support system testing.Source: LECTURE NOTES IN COMPUTER SCIENCE, vol. 15383, pp. 239-255. London, UK, 30/10-1/11/2024
DOI: 10.1007/978-3-031-80889-0_17
Metrics:
See at:
CNR IRIS | link.springer.com | doi.org | Repositorio Institucional de la Universidad de Oviedo | CNR IRIS | CNR IRIS
2025
Journal article
Open Access
Using metamorphic relations in redundancy-based fault/intrusion tolerance
Di Giandomenico F., Masetti G., Lonetti F., Bertolino A.
Redundancy is widely used as a method for fault and intrusion tolerance. However, if the redundant components lack sufficient diversity, potentially dangerous common mode failures may go undetected. To address this issue, the design diversity approach has been proposed in the literature for decades. In this paper, we take an innovative approach to this problem by introducing a broader notion of diversity, which leverages Metamorphic Relations (MRs), i.e., necessary properties that must hold among diverse inputs and diverse outputs. We define two generic categories of MRs that establish data diversity and functional diversity. Furthermore, we elaborate on two corresponding logical architectures, paying particular attention to the necessary conditions for the adjudicator component. Finally, we present an initial evaluation of the proposed architectures, which points out the advantages with respect to their counterparts based on the traditional design diversity method, and discuss future research directions for this novel conceptual approach to redundancy-based fault/intrusion tolerance.Source: ACM TRANSACTIONS ON SOFTWARE ENGINEERING AND METHODOLOGY
DOI: 10.1145/3772722
Metrics:
Di Giandomenico F., Masetti G., Lonetti F., Bertolino A.
Redundancy is widely used as a method for fault and intrusion tolerance. However, if the redundant components lack sufficient diversity, potentially dangerous common mode failures may go undetected. To address this issue, the design diversity approach has been proposed in the literature for decades. In this paper, we take an innovative approach to this problem by introducing a broader notion of diversity, which leverages Metamorphic Relations (MRs), i.e., necessary properties that must hold among diverse inputs and diverse outputs. We define two generic categories of MRs that establish data diversity and functional diversity. Furthermore, we elaborate on two corresponding logical architectures, paying particular attention to the necessary conditions for the adjudicator component. Finally, we present an initial evaluation of the proposed architectures, which points out the advantages with respect to their counterparts based on the traditional design diversity method, and discuss future research directions for this novel conceptual approach to redundancy-based fault/intrusion tolerance.Source: ACM TRANSACTIONS ON SOFTWARE ENGINEERING AND METHODOLOGY
DOI: 10.1145/3772722
Metrics:
See at:
dl.acm.org | CNR IRIS | ACM Transactions on Software Engineering and Methodology | CNR IRIS
2024
Journal article
Open Access
Self-adaptive testing in the field
Silva S, Pelliccione P, Bertolino A
We are increasingly surrounded by systems connecting us with the digital world and facilitating our life by supporting our work, leisure, activities at home, health, etc. These systems are pressed by two forces. On the one side, they operate in environments that are increasingly challenging due to uncertainty and uncontrollability. On the other side, they need to evolve, often in a continuous fashion, to meet changing needs, to ofer new functionalities, or also to fix emerging failures. To make the picture even more complex, these systems rarely work in isolation and often need to collaborate with other systems, as well as humans. All such facets call for moving their validation during operation, as offered by approaches called testing in the field.In this paper, we observe that even the field-based testing approaches should change over time to follow and adapt to the changes and evolution of collaborating systems or environments or users' behaviors. We provide a taxonomy of this new category of testing that we call self-adaptive testing in the ield (SATF), together with a reference architecture for SATF approaches. To achieve this objective, we surveyed the literature and collected feedback and contributions from experts in thedomain via a questionnaire and interviews.Source: ACM TRANSACTIONS ON AUTONOMOUS AND ADAPTIVE SYSTEMS (ONLINE), vol. 19 (issue 1), pp. 1-37
DOI: 10.1145/3627163
Metrics:
Silva S, Pelliccione P, Bertolino A
We are increasingly surrounded by systems connecting us with the digital world and facilitating our life by supporting our work, leisure, activities at home, health, etc. These systems are pressed by two forces. On the one side, they operate in environments that are increasingly challenging due to uncertainty and uncontrollability. On the other side, they need to evolve, often in a continuous fashion, to meet changing needs, to ofer new functionalities, or also to fix emerging failures. To make the picture even more complex, these systems rarely work in isolation and often need to collaborate with other systems, as well as humans. All such facets call for moving their validation during operation, as offered by approaches called testing in the field.In this paper, we observe that even the field-based testing approaches should change over time to follow and adapt to the changes and evolution of collaborating systems or environments or users' behaviors. We provide a taxonomy of this new category of testing that we call self-adaptive testing in the ield (SATF), together with a reference architecture for SATF approaches. To achieve this objective, we surveyed the literature and collected feedback and contributions from experts in thedomain via a questionnaire and interviews.Source: ACM TRANSACTIONS ON AUTONOMOUS AND ADAPTIVE SYSTEMS (ONLINE), vol. 19 (issue 1), pp. 1-37
DOI: 10.1145/3627163
Metrics:
See at:
dl.acm.org | CNR IRIS | ACM Transactions on Autonomous and Adaptive Systems | CNR IRIS | CNR IRIS
2024
Journal article
Open Access
Flakiness goes live: insights from an In Vivo testing simulation study
Barboni M, Bertolino A, De Angelis G
Context: Test flakiness is a topmost concern in software test automation. While conducting pre-deployment testing, those tests that are flagged as flaky are put aside for being either repaired or discarded.Objective: We hypothesise that some flaky tests could provide useful insights if run in the field, i.e., they could help identify failures that manifest themselves sporadically during In House testing, but are later experienced in operation.Method: We present the first simulation study to investigate the behaviour of flaky tests when moved to the field. The work compares the behaviour of known flaky tests from an open-source library when executed in the development environment vs. when executed in a simulation of the field.Results: Our experimentation over 52 test methods labelled as flaky provides a first confirmation that moving from the development environment to the field, the behaviour of tests changes. In particular, the failure frequency of intermittently failing tests can increase, and we could also identify few cases of field failures that would have been hardly detected during In House testing due to the numerous combinations of inputs and states. In most cases, such flakiness was rooted in the design of the test method itself, however we could also identify an actual bug.Conclusion: The results of our study suggest that the identification of an intermittently failing behaviour could be a valuable hint for a test engineer, and hence flaky tests should not be dismissed right away.Source: INFORMATION AND SOFTWARE TECHNOLOGY, vol. 167
DOI: 10.1016/j.infsof.2023.107373
Project(s): Solutions for Engineering Microservice Architectures
Metrics:
Barboni M, Bertolino A, De Angelis G
Context: Test flakiness is a topmost concern in software test automation. While conducting pre-deployment testing, those tests that are flagged as flaky are put aside for being either repaired or discarded.Objective: We hypothesise that some flaky tests could provide useful insights if run in the field, i.e., they could help identify failures that manifest themselves sporadically during In House testing, but are later experienced in operation.Method: We present the first simulation study to investigate the behaviour of flaky tests when moved to the field. The work compares the behaviour of known flaky tests from an open-source library when executed in the development environment vs. when executed in a simulation of the field.Results: Our experimentation over 52 test methods labelled as flaky provides a first confirmation that moving from the development environment to the field, the behaviour of tests changes. In particular, the failure frequency of intermittently failing tests can increase, and we could also identify few cases of field failures that would have been hardly detected during In House testing due to the numerous combinations of inputs and states. In most cases, such flakiness was rooted in the design of the test method itself, however we could also identify an actual bug.Conclusion: The results of our study suggest that the identification of an intermittently failing behaviour could be a valuable hint for a test engineer, and hence flaky tests should not be dismissed right away.Source: INFORMATION AND SOFTWARE TECHNOLOGY, vol. 167
DOI: 10.1016/j.infsof.2023.107373
Project(s): Solutions for Engineering Microservice Architectures
Metrics:
See at:
CNR IRIS | www.sciencedirect.com | CNR IRIS
2024
Journal article
Open Access
Automatic debugging of design faults in MapReduce applications
Morán J., Bertolino A., De La Riva C., Tuya J.
Among the current technologies to analyse large data, the MapReduce processing model stands out in Big Data. MapReduce is implemented in frameworks such as Hadoop, Spark or Flink that are able to manage the program executions according to the resources available at runtime. The developer should design the program in order to support all possible non-deterministic executions. However, the program may fail due to a design fault. Debugging these kinds of faults is difficult because the data are executed non-deterministically in parallel and the fault is not caused directly by the code, but by its design. This paper presents a framework called MRDebug which includes two debugging techniques focused on the MapReduce design faults. A spectrum- based fault localization technique locates the root cause of these faults analysing several executions of the test case, and a Delta Debugging technique isolates the data relevant to trigger the failure. An empirical evaluation with 13 programs shows that MRDebug is effective in debugging the faults, especially when the localization is done with the reduced data. In summary, MRDebug automatically provides valuable information to understand MapReduce design faults as it helps locate their root cause and obtains a minimal data that triggers the failure.Source: IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, vol. 50 (issue 4), pp. 956-978
DOI: 10.1109/tse.2024.3369766
Project(s): Progetto Integrato Cyber Security dei sistemi energetici
Metrics:
Morán J., Bertolino A., De La Riva C., Tuya J.
Among the current technologies to analyse large data, the MapReduce processing model stands out in Big Data. MapReduce is implemented in frameworks such as Hadoop, Spark or Flink that are able to manage the program executions according to the resources available at runtime. The developer should design the program in order to support all possible non-deterministic executions. However, the program may fail due to a design fault. Debugging these kinds of faults is difficult because the data are executed non-deterministically in parallel and the fault is not caused directly by the code, but by its design. This paper presents a framework called MRDebug which includes two debugging techniques focused on the MapReduce design faults. A spectrum- based fault localization technique locates the root cause of these faults analysing several executions of the test case, and a Delta Debugging technique isolates the data relevant to trigger the failure. An empirical evaluation with 13 programs shows that MRDebug is effective in debugging the faults, especially when the localization is done with the reduced data. In summary, MRDebug automatically provides valuable information to understand MapReduce design faults as it helps locate their root cause and obtains a minimal data that triggers the failure.Source: IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, vol. 50 (issue 4), pp. 956-978
DOI: 10.1109/tse.2024.3369766
Project(s): Progetto Integrato Cyber Security dei sistemi energetici
Metrics:
See at:
IEEE Transactions on Software Engineering | IRIS Cnr | IRIS Cnr | Repositorio Institucional de la Universidad de Oviedo | CNR IRIS
2024
Journal article
Open Access
RETORCH*: a cost and resource aware model for E2E testing in the cloud
Augusto C., Morán J., Bertolino A., De La Riva C., Tuya J.
Moving testing to the Cloud overcomes time/resource constraints by leveraging an unlimited and elastic infrastructure, especially for testing levels like End-to-End (E2E) that require a high number of resources and/or execution time. However, it introduces new challenges to those already faced on-premises, like selecting the most suitable Cloud infrastructure and billing scheme. We propose the RETORCH* test execution model that estimates and compares the monetary cost of executing an E2E test suite with different Cloud alternatives, billing schemes, and test configurations. RETORCH* goes beyond the mere cost billed, and selects the solution that best aligns with the test team strategy using the data of on-premises prior executions and the tester's experience. This cost is broken down into the cost incurred to execute the test suite (testing cost) and possible unused infrastructure (overprovisioning cost). Based on these distinct costs, the test team can compare different Cloud and test configurations. RETORCH* has been evaluated using a real-world application's E2E test suite. We analyze how the different decisions taken when the suite is migrated to the Cloud impact the cost, highlighting how RETORCH* can help the tester during Cloud and test configuration to make a more informed decision.Source: THE JOURNAL OF SYSTEMS AND SOFTWARE
DOI: 10.1016/j.jss.2024.112237
DOI: 10.2139/ssrn.4549726
DOI: 10.2139/ssrn.4831590
DOI: 10.2139/ssrn.4935951
Project(s): MASE RDS-PTR_22_24_P2.1 Cybersecurity (Italy)
Metrics:
Augusto C., Morán J., Bertolino A., De La Riva C., Tuya J.
Moving testing to the Cloud overcomes time/resource constraints by leveraging an unlimited and elastic infrastructure, especially for testing levels like End-to-End (E2E) that require a high number of resources and/or execution time. However, it introduces new challenges to those already faced on-premises, like selecting the most suitable Cloud infrastructure and billing scheme. We propose the RETORCH* test execution model that estimates and compares the monetary cost of executing an E2E test suite with different Cloud alternatives, billing schemes, and test configurations. RETORCH* goes beyond the mere cost billed, and selects the solution that best aligns with the test team strategy using the data of on-premises prior executions and the tester's experience. This cost is broken down into the cost incurred to execute the test suite (testing cost) and possible unused infrastructure (overprovisioning cost). Based on these distinct costs, the test team can compare different Cloud and test configurations. RETORCH* has been evaluated using a real-world application's E2E test suite. We analyze how the different decisions taken when the suite is migrated to the Cloud impact the cost, highlighting how RETORCH* can help the tester during Cloud and test configuration to make a more informed decision.Source: THE JOURNAL OF SYSTEMS AND SOFTWARE
DOI: 10.1016/j.jss.2024.112237
DOI: 10.2139/ssrn.4549726
DOI: 10.2139/ssrn.4831590
DOI: 10.2139/ssrn.4935951
Project(s): MASE RDS-PTR_22_24_P2.1 Cybersecurity (Italy)
Metrics:
See at:
Journal of Systems and Software | IRIS Cnr | IRIS Cnr | doi.org | doi.org | doi.org | CNR IRIS
2024
Journal article
Open Access
A framework for the design of fault-tolerant systems-of-systems
Cerdeira Ferreira F. H., Nakagawa E. Y., Bertolino A., Lonetti F., De Oliveira Neves V., Pereira Dos Santos R.
Context: Systems-of-Systems (SoS) increasingly permeate everyday life in various critical domains. Due to their dynamic nature, guaranteeing their fault tolerance is challenging. Fault-tolerant SoS must deal with behavioral changes in constituent systems, whether accidental or deliberate. Goal: This work proposes ReViTA, a framework to assist professionals in designing fault-tolerant SoS that can continue to provide their function even in the presence of disturbances, i.e., events that affect the ability of an SoS to fulfill its mission. Methods: By adopting ReViTA, fault tolerance can be achieved by reconfiguring an SoS architecture to meet the critical mission requirements. Results: We performed two studies to evaluate the ReViTA acceptance by professionals. In the former, we gathered perceptions and suggestions from 14 professionals through individual interviews. In the latter, we involved a group of four professionals who applied ReViTA to a real-world scenario. Conclusion: The results demonstrate that ReViTA can effectively support professionals in designing faulttolerant SoS. Employing ReViTA also brings insights into costs and planning that are crucial for implementing fault-tolerance strategies. Using ReViTA facilitates a comprehensive understanding of conflicts and weaknesses in constituent systems and fosters collaboration between domain experts and decision-makers. Employing ReViTA also improves stakeholder communication and enhances resource utilization.Source: THE JOURNAL OF SYSTEMS AND SOFTWARE, vol. 211
DOI: 10.1016/j.jss.2024.112010
Metrics:
Cerdeira Ferreira F. H., Nakagawa E. Y., Bertolino A., Lonetti F., De Oliveira Neves V., Pereira Dos Santos R.
Context: Systems-of-Systems (SoS) increasingly permeate everyday life in various critical domains. Due to their dynamic nature, guaranteeing their fault tolerance is challenging. Fault-tolerant SoS must deal with behavioral changes in constituent systems, whether accidental or deliberate. Goal: This work proposes ReViTA, a framework to assist professionals in designing fault-tolerant SoS that can continue to provide their function even in the presence of disturbances, i.e., events that affect the ability of an SoS to fulfill its mission. Methods: By adopting ReViTA, fault tolerance can be achieved by reconfiguring an SoS architecture to meet the critical mission requirements. Results: We performed two studies to evaluate the ReViTA acceptance by professionals. In the former, we gathered perceptions and suggestions from 14 professionals through individual interviews. In the latter, we involved a group of four professionals who applied ReViTA to a real-world scenario. Conclusion: The results demonstrate that ReViTA can effectively support professionals in designing faulttolerant SoS. Employing ReViTA also brings insights into costs and planning that are crucial for implementing fault-tolerance strategies. Using ReViTA facilitates a comprehensive understanding of conflicts and weaknesses in constituent systems and fosters collaboration between domain experts and decision-makers. Employing ReViTA also improves stakeholder communication and enhances resource utilization.Source: THE JOURNAL OF SYSTEMS AND SOFTWARE, vol. 211
DOI: 10.1016/j.jss.2024.112010
Metrics:
See at:
IRIS Cnr | IRIS Cnr | IRIS Cnr | Journal of Systems and Software | CNR IRIS | CNR IRIS | CNR IRIS
2024
Book
Open Access
Quality of information and communications technology
Bertolino A., Faria J. P., Lago P., Semini L.
This book constitutes the proceedings of the 17th International Conference on the Quality of Information and Communications Technology, QUATIC 2024, held in Pisa, Italy, during September 11–13, 2024. The 34 full and short papers of QUATIC 2024 included in this book were carefully reviewed and selected from 49 submissions. QUATIC is a forum for disseminating advanced methods, techniques and tools to support quality approaches to ICT engineering and management. Practitioners and researchers are encouraged to exchange ideas and approaches on how to adopt a quality culture in ICT process and product improvement and to provide practical studies in varying contexts.Source: COMMUNICATIONS IN COMPUTER AND INFORMATION SCIENCE, vol. 2178
DOI: 10.1007/978-3-031-70245-7
Metrics:
Bertolino A., Faria J. P., Lago P., Semini L.
This book constitutes the proceedings of the 17th International Conference on the Quality of Information and Communications Technology, QUATIC 2024, held in Pisa, Italy, during September 11–13, 2024. The 34 full and short papers of QUATIC 2024 included in this book were carefully reviewed and selected from 49 submissions. QUATIC is a forum for disseminating advanced methods, techniques and tools to support quality approaches to ICT engineering and management. Practitioners and researchers are encouraged to exchange ideas and approaches on how to adopt a quality culture in ICT process and product improvement and to provide practical studies in varying contexts.Source: COMMUNICATIONS IN COMPUTER AND INFORMATION SCIENCE, vol. 2178
DOI: 10.1007/978-3-031-70245-7
Metrics:
See at:
CNR IRIS | link.springer.com | doi.org | CNR IRIS
2023
Journal article
Open Access
DevOpRET: continuous reliability testing in DevOps
Bertolino A., De Angelis G., Guerriero A., Miranda B., Pietrantuono R., Russo S.
To enter the production stage, in DevOps practices candidate software releases have to pass quality gates, where they are assessed to meet established target values for key indicators of interest. We believe software reliability should be an important such indicator, as it greatly contributes to the end-user satisfaction. We proposeDevOpRET, an approach for reliability testing as part of the acceptance testing stage in DevOps.DevOpRETrelies on operational-profile-based testing, a common reliability assessment technique.DevOpRETleverages usage and failure data monitored in operations to continuously refine its estimate. We evaluate accuracy and efficiency ofDevOpRETthrough controlled experiments with a real-world open source platform and with a microservice architectures benchmark. The results show thatDevOpRETprovides accurate and efficient estimates of the true reliability over subsequent DevOps cycles.Source: JOURNAL OF SOFTWARE, vol. 35 (issue 3)
DOI: 10.1002/smr.2298
Project(s): Governing Adaptive and Unplanned Systems of Systems
Metrics:
Bertolino A., De Angelis G., Guerriero A., Miranda B., Pietrantuono R., Russo S.
To enter the production stage, in DevOps practices candidate software releases have to pass quality gates, where they are assessed to meet established target values for key indicators of interest. We believe software reliability should be an important such indicator, as it greatly contributes to the end-user satisfaction. We proposeDevOpRET, an approach for reliability testing as part of the acceptance testing stage in DevOps.DevOpRETrelies on operational-profile-based testing, a common reliability assessment technique.DevOpRETleverages usage and failure data monitored in operations to continuously refine its estimate. We evaluate accuracy and efficiency ofDevOpRETthrough controlled experiments with a real-world open source platform and with a microservice architectures benchmark. The results show thatDevOpRETprovides accurate and efficient estimates of the true reliability over subsequent DevOps cycles.Source: JOURNAL OF SOFTWARE, vol. 35 (issue 3)
DOI: 10.1002/smr.2298
Project(s): Governing Adaptive and Unplanned Systems of Systems
Metrics:
See at:
CNR IRIS | onlinelibrary.wiley.com | Journal of Software Evolution and Process | CNR IRIS | CNR IRIS
2023
Journal article
Open Access
In vivo test and rollback of Java applications as they are
Bertolino A, De Angelis G, Miranda B, Tonella P
Modern software systems accommodate complex configurations and execution conditions that depend on the environment where the software is run. While in house testing can exercise only a fraction of such execution contexts, in vivo testing can take advantage of the execution state observed in the field to conduct further testing activities. In this paper, we present the Groucho approach to in vivo testing. Groucho can suspend the execution, run some in vivo tests, rollback the side effects introduced by such tests, and eventually resume normal execution. The approach can be transparently applied to the original application, even if only available as compiled code, and it is fully automated. Our empirical studies of the performance overhead introduced by Groucho under various configurations showed that this may be kept to a negligible level by activating in vivo testing with low probability. Our empirical studies about the effectiveness of the approach confirm previous findings on the existence offaults that are unlikely exposed in house and become easy to expose in the field. Moreover, we include the first study to quantify the coverage increase gained when in vivo testing is added to complement in house testing.Source: SOFTWARE TESTING, VERIFICATION & RELIABILITY, vol. 33
DOI: 10.1002/stvr.1857
Project(s): SISMA, PRECRIME
Metrics:
Bertolino A, De Angelis G, Miranda B, Tonella P
Modern software systems accommodate complex configurations and execution conditions that depend on the environment where the software is run. While in house testing can exercise only a fraction of such execution contexts, in vivo testing can take advantage of the execution state observed in the field to conduct further testing activities. In this paper, we present the Groucho approach to in vivo testing. Groucho can suspend the execution, run some in vivo tests, rollback the side effects introduced by such tests, and eventually resume normal execution. The approach can be transparently applied to the original application, even if only available as compiled code, and it is fully automated. Our empirical studies of the performance overhead introduced by Groucho under various configurations showed that this may be kept to a negligible level by activating in vivo testing with low probability. Our empirical studies about the effectiveness of the approach confirm previous findings on the existence offaults that are unlikely exposed in house and become easy to expose in the field. Moreover, we include the first study to quantify the coverage increase gained when in vivo testing is added to complement in house testing.Source: SOFTWARE TESTING, VERIFICATION & RELIABILITY, vol. 33
DOI: 10.1002/stvr.1857
Project(s): SISMA, PRECRIME
Metrics:
See at:
CNR IRIS | onlinelibrary.wiley.com | ISTI Repository | ISTI Repository | CNR IRIS | CNR IRIS
2023
Journal article
Open Access
State of practical applicability of regression testing research: a live systematic literature review
Greca R, Miranda B, Bertolino A
Context: Software regression testing refers to rerunning test cases after the system under test is modified, ascertaining that the changes have not (re-)introduced failures. Not all researchers' approaches consider applicability and scalability concerns, and not many have produced an impact in practice. Objective: One goal is to investigate industrial relevance and applicability of proposed approaches. Another is providing a live review, open to continuous updates by the community. Method: A systematic review of regression testing studies that are clearly motivated by or validated against industrial relevance and applicability is conducted. It is complemented by follow-up surveys with authors of the selected papers and 23 practitioners. Results: A set of 79 primary studies published between 2016-2022 is collected and classified according to approaches and metrics. Aspects relative to their relevance and impact are discussed, also based on their authors' feedback. All the data are made available from the live repository that accompanies the study. Conclusions: While widely motivated by industrial relevance and applicability, not many approaches are evaluated in industrial or large-scale open-source systems, and even fewer approaches have been adopted in practice. Some challenges hindering the implementation of relevant approaches are synthesized, also based on the practitioners' feedback.Source: ACM COMPUTING SURVEYS, vol. 55 (issue 13s)
DOI: 10.1145/3579851
Metrics:
Greca R, Miranda B, Bertolino A
Context: Software regression testing refers to rerunning test cases after the system under test is modified, ascertaining that the changes have not (re-)introduced failures. Not all researchers' approaches consider applicability and scalability concerns, and not many have produced an impact in practice. Objective: One goal is to investigate industrial relevance and applicability of proposed approaches. Another is providing a live review, open to continuous updates by the community. Method: A systematic review of regression testing studies that are clearly motivated by or validated against industrial relevance and applicability is conducted. It is complemented by follow-up surveys with authors of the selected papers and 23 practitioners. Results: A set of 79 primary studies published between 2016-2022 is collected and classified according to approaches and metrics. Aspects relative to their relevance and impact are discussed, also based on their authors' feedback. All the data are made available from the live repository that accompanies the study. Conclusions: While widely motivated by industrial relevance and applicability, not many approaches are evaluated in industrial or large-scale open-source systems, and even fewer approaches have been adopted in practice. Some challenges hindering the implementation of relevant approaches are synthesized, also based on the practitioners' feedback.Source: ACM COMPUTING SURVEYS, vol. 55 (issue 13s)
DOI: 10.1145/3579851
Metrics:
See at:
dl.acm.org | CNR IRIS | ISTI Repository | ACM Computing Surveys | CNR IRIS
2023
Journal article
Open Access
Editorial: Introduction to the special issue on test automation: trends, benefits, and costs
Bertolino A, De Angelis G, Leotta M, Ricca F
Source: THE JOURNAL OF SYSTEMS AND SOFTWARE, vol. 203
DOI: 10.1016/j.jss.2023.111751
Metrics:
Bertolino A, De Angelis G, Leotta M, Ricca F
Source: THE JOURNAL OF SYSTEMS AND SOFTWARE, vol. 203
DOI: 10.1016/j.jss.2023.111751
Metrics:
See at:
CNR IRIS | ISTI Repository | www.sciencedirect.com | Journal of Systems and Software | CNR IRIS | CNR IRIS
2023
Conference article
Open Access
Orchestration strategies for regression test suites
Greca R, Miranda B, Bertolino A
Regression testing is widely studied in the literature, although most research on the topic is concerned with improving specific sub-challenges of a wider goal. Test suite orchestration proposes a more comprehensive view of the challenge of regression testing, by merging and combining different techniques with a variety of objectives, including prioritizing, selecting, reducing and amplifying tests, detecting flaky tests and potentially more. This paper presents the key approaches and techniques that form test suite orchestration, along with common evaluation metrics, and discusses how they can be used together to ultimately provide an efficient and effective regression testing strategy. To illustrate the benefits of orchestration, we provide some examples of existing papers that take steps towards this goal, even if the specific terminology is not yet used. Orchestrated strategies utilizing existing regression testing techniques provide a pathway to practicality and real-world usage of the academic literature.DOI: 10.1109/ast58925.2023.00020
Metrics:
Greca R, Miranda B, Bertolino A
Regression testing is widely studied in the literature, although most research on the topic is concerned with improving specific sub-challenges of a wider goal. Test suite orchestration proposes a more comprehensive view of the challenge of regression testing, by merging and combining different techniques with a variety of objectives, including prioritizing, selecting, reducing and amplifying tests, detecting flaky tests and potentially more. This paper presents the key approaches and techniques that form test suite orchestration, along with common evaluation metrics, and discusses how they can be used together to ultimately provide an efficient and effective regression testing strategy. To illustrate the benefits of orchestration, we provide some examples of existing papers that take steps towards this goal, even if the specific terminology is not yet used. Orchestrated strategies utilizing existing regression testing techniques provide a pathway to practicality and real-world usage of the academic literature.DOI: 10.1109/ast58925.2023.00020
Metrics:
See at:
CNR IRIS | ieeexplore.ieee.org | ISTI Repository | CNR IRIS | CNR IRIS
2023
Conference article
Open Access
Cross-coverage testing of functionally equivalent programs
Bertolino A, De Angelis G, Di Giandomenico F, Lonetti F
Cross-coverage of a program P refers to the test coverage measured over a different program Q that is functionally equivalent to P. The novel concept of cross-coverage can find useful applications in the test of redundant software. We apply here cross-coverage for test suite augmentation and show that additional test cases generated from the coverage of an equivalent program, referred to as cross tests, can increase the coverage of a program in more effective way than a random baseline. We also observe that -contrary to traditional coverage testing-cross coverage could help finding (artificially created) missing functionality faults.DOI: 10.1109/ast58925.2023.00014
Metrics:
Bertolino A, De Angelis G, Di Giandomenico F, Lonetti F
Cross-coverage of a program P refers to the test coverage measured over a different program Q that is functionally equivalent to P. The novel concept of cross-coverage can find useful applications in the test of redundant software. We apply here cross-coverage for test suite augmentation and show that additional test cases generated from the coverage of an equivalent program, referred to as cross tests, can increase the coverage of a program in more effective way than a random baseline. We also observe that -contrary to traditional coverage testing-cross coverage could help finding (artificially created) missing functionality faults.DOI: 10.1109/ast58925.2023.00014
Metrics:
See at:
CNR IRIS | ieeexplore.ieee.org | ISTI Repository | CNR IRIS | CNR IRIS
2023
Conference article
Open Access
Fault localization for reinforcement learning
Morán J, Bertolino A, De La Riva C, Tuya J
Reinforcement Learning is widely adopted in industry to approach control tasks in intelligent way. The quality of these programs is important especially when they are used for critical tasks like autonomous driving. Testing and debugging these programs are complex because they behave autonomously without providing insights about the reasons of the decisions taken. Even these decisions could be wrong if they learned from faults. In this paper, we present the first approach to automatically locate faults in Reinforcement Learning programs. This approach called SBFL4RL analyses several executions to extract those internal states that commonly reduce the performance of the program when they are covered. Locating these states can help testers to understand a known fault, or even detect an unknown fault. SBFL4RL is validated in 2 case studies locating correctly an injected fault. Initial results suggest that the faults of reinforcement learning programs can be automatically located, and there is room for further research.DOI: 10.1109/aitest58265.2023.00016
Metrics:
Morán J, Bertolino A, De La Riva C, Tuya J
Reinforcement Learning is widely adopted in industry to approach control tasks in intelligent way. The quality of these programs is important especially when they are used for critical tasks like autonomous driving. Testing and debugging these programs are complex because they behave autonomously without providing insights about the reasons of the decisions taken. Even these decisions could be wrong if they learned from faults. In this paper, we present the first approach to automatically locate faults in Reinforcement Learning programs. This approach called SBFL4RL analyses several executions to extract those internal states that commonly reduce the performance of the program when they are covered. Locating these states can help testers to understand a known fault, or even detect an unknown fault. SBFL4RL is validated in 2 case studies locating correctly an injected fault. Initial results suggest that the faults of reinforcement learning programs can be automatically located, and there is room for further research.DOI: 10.1109/aitest58265.2023.00016
Metrics:
See at:
CNR IRIS | ieeexplore.ieee.org | ISTI Repository | CNR IRIS | CNR IRIS