2024
Journal article
Open Access
Using vice and biofeedback to predict user engagement during product feedback interviews
Ferrari A., Huichapa T., Spoletini P., Novielli N., Fucci D., Girardi D.
Capturing users’ engagement is crucial for gathering feedback about the features of a software product. In a market-driven context, current approaches to collecting and analyzing users’ feedback are based on techniques leveraging information extracted from product reviews and social media. These approaches are hardly applicable in contexts where online feedback is limited, as for the majority of apps, and software in general. In such cases, companies need to resort to face-to-face interviews to get feedback on their products. In this article, we propose to utilize biometric data, in terms of physiological and voice features, to complement product feedback interviews with information about the engagement of the user on product-relevant topics. We evaluate our approach by interviewing users while gathering their physiological data (i.e., biofeedback) using an Empatica E4 wristband, and capturing their voice through the default audio-recorder of a common laptop. Our results show that we can predict users’ engagement by training supervised machine learning algorithms on biofeedback and voice data, and that voice features alone can be sufficiently effective. The best configurations evaluated achieve an average F1 ∼ 70% in terms of classification performance, and use voice features only. This work is one of the first studies in requirements engineering in which biometrics are used to identify emotions. Furthermore, this is one of the first studies in software engineering that considers voice analysis. The usage of voice features can be particularly helpful for emotion-aware feedback collection in remote communication, either performed by human analysts or voice-based chatbots, and can also be exploited to support the analysis of meetings in software engineering research.Source: ACM TRANSACTIONS ON SOFTWARE ENGINEERING AND METHODOLOGY, vol. 33 (issue 4), pp. 1-36
DOI: 10.1145/3635712
DOI: 10.5281/zenodo.11350306
DOI: 10.48550/arxiv.2104.02410
DOI: 10.5281/zenodo.11350307
Project(s): CODECS
Metrics:
Ferrari A., Huichapa T., Spoletini P., Novielli N., Fucci D., Girardi D.
Capturing users’ engagement is crucial for gathering feedback about the features of a software product. In a market-driven context, current approaches to collecting and analyzing users’ feedback are based on techniques leveraging information extracted from product reviews and social media. These approaches are hardly applicable in contexts where online feedback is limited, as for the majority of apps, and software in general. In such cases, companies need to resort to face-to-face interviews to get feedback on their products. In this article, we propose to utilize biometric data, in terms of physiological and voice features, to complement product feedback interviews with information about the engagement of the user on product-relevant topics. We evaluate our approach by interviewing users while gathering their physiological data (i.e., biofeedback) using an Empatica E4 wristband, and capturing their voice through the default audio-recorder of a common laptop. Our results show that we can predict users’ engagement by training supervised machine learning algorithms on biofeedback and voice data, and that voice features alone can be sufficiently effective. The best configurations evaluated achieve an average F1 ∼ 70% in terms of classification performance, and use voice features only. This work is one of the first studies in requirements engineering in which biometrics are used to identify emotions. Furthermore, this is one of the first studies in software engineering that considers voice analysis. The usage of voice features can be particularly helpful for emotion-aware feedback collection in remote communication, either performed by human analysts or voice-based chatbots, and can also be exploited to support the analysis of meetings in software engineering research.Source: ACM TRANSACTIONS ON SOFTWARE ENGINEERING AND METHODOLOGY, vol. 33 (issue 4), pp. 1-36
DOI: 10.1145/3635712
DOI: 10.5281/zenodo.11350306
DOI: 10.48550/arxiv.2104.02410
DOI: 10.5281/zenodo.11350307
Project(s): CODECS
Metrics:
See at:
arXiv.org e-Print Archive 
| IRIS Cnr 
| IRIS Cnr | doi.org | ZENODO | ZENODO | IRIS Cnr | CNR IRIS
2024
Journal article
Open Access
Replication in Requirements Engineering: the NLP for RE case
Abualhaija S., Aydemir F. B., Dalpiaz F., Dell'Anna D., Ferrari A., Franch X., Fucci D.
Natural language processing (NLP) techniques have been widely applied in the requirements engineering (RE) field to support tasks such as classification and ambiguity detection. Despite its empirical vocation, RE research has given limited attention to replication of NLP for RE studies. Replication is hampered by several factors, including the context specificity of the studies, the heterogeneity of the tasks involving NLP, the tasks’ inherent hairiness, and, in turn, the heterogeneous reporting structure. To address these issues, we propose a new artifact, referred to as ID-Card, whose goal is to provide a structured summary of research papers emphasizing replication-relevant information. We construct the ID-Card through a structured, iterative process based on design science. In this article: (i) we report on hands-on experiences of replication; (ii) we review the state-of-the-art and extract replication-relevant information: (iii) we identify, through focus groups, challenges across two typical dimensions of replication: data annotation and tool reconstruction; and (iv) we present the concept and structure of the ID-Card to mitigate the identified challenges. This study aims to create awareness of replication in NLP for RE. We propose an ID-Card that is intended to foster study replication but can also be used in other contexts, e.g., for educational purposes.Source: ACM TRANSACTIONS ON SOFTWARE ENGINEERING AND METHODOLOGY, vol. 33 (issue 6), pp. 1-33
DOI: 10.1145/3658669
DOI: 10.48550/arxiv.2304.10265
Project(s): CODECS
Metrics:
Abualhaija S., Aydemir F. B., Dalpiaz F., Dell'Anna D., Ferrari A., Franch X., Fucci D.
Natural language processing (NLP) techniques have been widely applied in the requirements engineering (RE) field to support tasks such as classification and ambiguity detection. Despite its empirical vocation, RE research has given limited attention to replication of NLP for RE studies. Replication is hampered by several factors, including the context specificity of the studies, the heterogeneity of the tasks involving NLP, the tasks’ inherent hairiness, and, in turn, the heterogeneous reporting structure. To address these issues, we propose a new artifact, referred to as ID-Card, whose goal is to provide a structured summary of research papers emphasizing replication-relevant information. We construct the ID-Card through a structured, iterative process based on design science. In this article: (i) we report on hands-on experiences of replication; (ii) we review the state-of-the-art and extract replication-relevant information: (iii) we identify, through focus groups, challenges across two typical dimensions of replication: data annotation and tool reconstruction; and (iv) we present the concept and structure of the ID-Card to mitigate the identified challenges. This study aims to create awareness of replication in NLP for RE. We propose an ID-Card that is intended to foster study replication but can also be used in other contexts, e.g., for educational purposes.Source: ACM TRANSACTIONS ON SOFTWARE ENGINEERING AND METHODOLOGY, vol. 33 (issue 6), pp. 1-33
DOI: 10.1145/3658669
DOI: 10.48550/arxiv.2304.10265
Project(s): CODECS
Metrics:
See at:
arXiv.org e-Print Archive | IRIS Cnr | ACM Transactions on Software Engineering and Methodology | IRIS Cnr | UPCommons. Portal del coneixement obert de la UPC | ACM Transactions on Software Engineering and Methodology | Electronic Research Archive - Blekinge Tekniska Högskola | Software Heritage | ACM Transactions on Software Engineering and Methodology | doi.org | GitHub | CNR IRIS
2024
Conference article
Restricted
The return of formal requirements engineering in the era of large language models
Spoletini P., Ferrari A.
Context and Motivation: Large Language Models (LLMs) have made remarkable advancements in emulating human linguistic capabilities, showing potential in executing various traditional software engineering tasks, including code generation. [Question/Problem] Despite their generally good performance, utilizing LLM-generated code raises legitimate concerns regarding its correctness and the assurances it can provide. [Principal Idea/Results] To address these concerns, we propose turning to formal requirements engineering—a practice currently predominantly used in developing complex systems where adherence to standards and accountability are required. [Contribution] In this vision paper, we discuss the integration of automatic formal requirements engineering techniques as a complement to LLM code generation. Additionally, we explore how LLMs can facilitate the broader acceptance of formal requirements, thus making the vision proposed in this paper realizable.Source: LECTURE NOTES IN COMPUTER SCIENCE, vol. 14588, pp. 344-353. Winterthur, Switzerland, 8-11/04/2024
DOI: 10.1007/978-3-031-57327-9_22
Metrics:
Spoletini P., Ferrari A.
Context and Motivation: Large Language Models (LLMs) have made remarkable advancements in emulating human linguistic capabilities, showing potential in executing various traditional software engineering tasks, including code generation. [Question/Problem] Despite their generally good performance, utilizing LLM-generated code raises legitimate concerns regarding its correctness and the assurances it can provide. [Principal Idea/Results] To address these concerns, we propose turning to formal requirements engineering—a practice currently predominantly used in developing complex systems where adherence to standards and accountability are required. [Contribution] In this vision paper, we discuss the integration of automatic formal requirements engineering techniques as a complement to LLM code generation. Additionally, we explore how LLMs can facilitate the broader acceptance of formal requirements, thus making the vision proposed in this paper realizable.Source: LECTURE NOTES IN COMPUTER SCIENCE, vol. 14588, pp. 344-353. Winterthur, Switzerland, 8-11/04/2024
DOI: 10.1007/978-3-031-57327-9_22
Metrics:
2024
Conference article
Open Access
Assessing the understandability and acceptance of attack-defense trees for modelling security requirements
Broccia G., Ter Beek M. H., Lluch Lafuente A., Spoletini P., Ferrari A.
Context and Motivation Attack-Defense Trees (ADTs) are a graphical notation used to model and assess security requirements. ADTs are widely popular, as they can facilitate communication between differ-ent stakeholders involved in system security evaluation, and they are for-mal enough to be verified, e.g., with model checkers. Question/Problem While the quality of this notation has been primarily assessed quanti-tatively, its understandability has never been evaluated despite being mentioned as a key factor for its success. Principal idea/Results In this paper, we conduct an experiment with 25 human subjects to assess the understandability and user acceptance of the ADT notation. The study focuses on performance-based variables and perception-based variables, with the aim of evaluating the relationship between these measures and how they might impact the practical use of the notation. The results confirm a good level of understandability of ADTs. Participants consider them useful, and they show intention to use them. Contribution This is the first study empirically supporting the understandability of ADTs, thereby contributing to the theory of security requirements engineering.Source: LECTURE NOTES IN COMPUTER SCIENCE, vol. 14588, pp. 39-56. Winterthur, Switzerland, April 8–11, 2024
DOI: 10.1007/978-3-031-57327-9_3
DOI: 10.48550/arxiv.2404.06386
Project(s): CODECS, Typeful Language Adaptation for Dynamic, Interacting and Evolving Systems
Metrics:
Broccia G., Ter Beek M. H., Lluch Lafuente A., Spoletini P., Ferrari A.
Context and Motivation Attack-Defense Trees (ADTs) are a graphical notation used to model and assess security requirements. ADTs are widely popular, as they can facilitate communication between differ-ent stakeholders involved in system security evaluation, and they are for-mal enough to be verified, e.g., with model checkers. Question/Problem While the quality of this notation has been primarily assessed quanti-tatively, its understandability has never been evaluated despite being mentioned as a key factor for its success. Principal idea/Results In this paper, we conduct an experiment with 25 human subjects to assess the understandability and user acceptance of the ADT notation. The study focuses on performance-based variables and perception-based variables, with the aim of evaluating the relationship between these measures and how they might impact the practical use of the notation. The results confirm a good level of understandability of ADTs. Participants consider them useful, and they show intention to use them. Contribution This is the first study empirically supporting the understandability of ADTs, thereby contributing to the theory of security requirements engineering.Source: LECTURE NOTES IN COMPUTER SCIENCE, vol. 14588, pp. 39-56. Winterthur, Switzerland, April 8–11, 2024
DOI: 10.1007/978-3-031-57327-9_3
DOI: 10.48550/arxiv.2404.06386
Project(s): CODECS
, Typeful Language Adaptation for Dynamic, Interacting and Evolving SystemsMetrics:
See at:
arXiv.org e-Print Archive | doi.org | doi.org | IRIS Cnr | CNR IRIS | CNR IRIS | IRIS Cnr
2024
Conference article
Open Access
ModeLLer – Enabling end-users to model systems: a case study in digital agriculture
Mannari C., Anichini E., Bacco M., Ferrari A., Turchi T., Malizia A.
Digital technologies show promising potential in the development of sustainable agriculture. For example, the combination of cloud and edge paradigms, 5G, and the Internet of Things (IoT) allows the development of sophisticated applications — e.g., for food traceability, pest detection, and automatic irrigation — with the possibility to also exploit Artificial Intelligence (AI)-powered techniques. At the same time, digitalisation in agriculture is a socio-technical process that involves several classes of stakeholders with diverse backgrounds and skills, e.g., in farming or technology. Model-driven approaches leveraging diagrammatic notations can support information exchange between different domains. In fact, the development of models can be a co-design practice involving end-users throughout all phases of creation because of their expressive power. However, current modelling platforms are typically oriented toward engineers, and there is a lack of tools accessible to end-users for designing and modelling systems. In this position paper, we present ModeLLer, a prototype of a web environment for modelling systems based on an intuitive visual language that can be exported into standard code. The aim is to increasingly involve users in modelling their digital ecosystems as a task for developing digital applications.Source: CEUR WORKSHOP PROCEEDINGS, vol. 3685. Arenzano, Italy, 4/06/2024
Project(s): CODECS
Mannari C., Anichini E., Bacco M., Ferrari A., Turchi T., Malizia A.
Digital technologies show promising potential in the development of sustainable agriculture. For example, the combination of cloud and edge paradigms, 5G, and the Internet of Things (IoT) allows the development of sophisticated applications — e.g., for food traceability, pest detection, and automatic irrigation — with the possibility to also exploit Artificial Intelligence (AI)-powered techniques. At the same time, digitalisation in agriculture is a socio-technical process that involves several classes of stakeholders with diverse backgrounds and skills, e.g., in farming or technology. Model-driven approaches leveraging diagrammatic notations can support information exchange between different domains. In fact, the development of models can be a co-design practice involving end-users throughout all phases of creation because of their expressive power. However, current modelling platforms are typically oriented toward engineers, and there is a lack of tools accessible to end-users for designing and modelling systems. In this position paper, we present ModeLLer, a prototype of a web environment for modelling systems based on an intuitive visual language that can be exported into standard code. The aim is to increasingly involve users in modelling their digital ecosystems as a task for developing digital applications.Source: CEUR WORKSHOP PROCEEDINGS, vol. 3685. Arenzano, Italy, 4/06/2024
Project(s): CODECS
See at:
ceur-ws.org | CNR IRIS | CNR IRIS
2024
Conference article
Restricted
Identifying maintenance needs with machine learning: a case study in railways
Ferdous R., Spagnolo G. O., Borselli A., Rota L., Ferrari A.
Cyber-physical systems, particularly those with extended service lives such as railways, often necessitate significant investment in maintenance activities encompassing repairs, upgrades, or inspections. These decisions are generally based on fixed schedules, or informed by the judgment of experienced maintenance staff. To improve this process, predictive maintenance (PdM) has emerged as a viable solution to anticipate maintenance needs and preempt system failures. With data-driven PdM, maintenance needs are identified through machine learning (ML) solutions that monitor the system logs and recommend interventions before a failure occurs. This paper presents preliminary findings from a case study concerning the development of a ML system for PdM in railways. We present the current maintenance process, the existing logging platform, and our strategy for leveraging log data to support PdM. Our preliminary results are promising. However, they show that, although the log dataset spans three years and three railway vehicles, in some cases the log data alone are insufficient for accurately inferring maintenance requirements. To address the problem, we discuss the necessity of employing synthetic data generation methods and rule-based, knowledge-driven strategies.DOI: 10.1109/rew61692.2024.00008
Project(s): MOST – Sustainable Mobility National Research Center and received funding from the European Union Next-GenerationEU (PIANO NAZIONALE DI RIPRESA E RESILIENZA (PNRR) – MISSIONE 4 COMPONENTE 2, INVESTIMENTO 1.4
Metrics:
Ferdous R., Spagnolo G. O., Borselli A., Rota L., Ferrari A.
Cyber-physical systems, particularly those with extended service lives such as railways, often necessitate significant investment in maintenance activities encompassing repairs, upgrades, or inspections. These decisions are generally based on fixed schedules, or informed by the judgment of experienced maintenance staff. To improve this process, predictive maintenance (PdM) has emerged as a viable solution to anticipate maintenance needs and preempt system failures. With data-driven PdM, maintenance needs are identified through machine learning (ML) solutions that monitor the system logs and recommend interventions before a failure occurs. This paper presents preliminary findings from a case study concerning the development of a ML system for PdM in railways. We present the current maintenance process, the existing logging platform, and our strategy for leveraging log data to support PdM. Our preliminary results are promising. However, they show that, although the log dataset spans three years and three railway vehicles, in some cases the log data alone are insufficient for accurately inferring maintenance requirements. To address the problem, we discuss the necessity of employing synthetic data generation methods and rule-based, knowledge-driven strategies.DOI: 10.1109/rew61692.2024.00008
Project(s): MOST – Sustainable Mobility National Research Center and received funding from the European Union Next-GenerationEU (PIANO NAZIONALE DI RIPRESA E RESILIENZA (PNRR) – MISSIONE 4 COMPONENTE 2, INVESTIMENTO 1.4
Metrics:
See at:
CNR IRIS | ieeexplore.ieee.org | CNR IRIS
2024
Book
Open Access
Preface: 7th workshop on Natural Language Processing for Requirements Engineering (NLP4RE'24)
Abualhaija S., Arora C., Dell'Anna D., Ferrari A., Ghanavati S.
Natural language processing (NLP) plays an essential role in several areas of software engineering (SE), and requirements engineering (RE) is no exception. Requirements are generally authored and communicated in textual form and different levels of formality, from structured (e.g., user stories) to unstructured natural language. Furthermore, in the last few years, the advent of massive and heterogeneous sources, such as tweets and app reviews, has attracted even more interest from the RE community, and the recent developments in large language models (LLMs) and generative AI have opened new opportunities for RE. LLMs will likely be the enabling technology for solving long-standing RE problems, such as traceability, classification, and compliance.Source: CEUR WORKSHOP PROCEEDINGS, vol. 3672
Abualhaija S., Arora C., Dell'Anna D., Ferrari A., Ghanavati S.
Natural language processing (NLP) plays an essential role in several areas of software engineering (SE), and requirements engineering (RE) is no exception. Requirements are generally authored and communicated in textual form and different levels of formality, from structured (e.g., user stories) to unstructured natural language. Furthermore, in the last few years, the advent of massive and heterogeneous sources, such as tweets and app reviews, has attracted even more interest from the RE community, and the recent developments in large language models (LLMs) and generative AI have opened new opportunities for RE. LLMs will likely be the enabling technology for solving long-standing RE problems, such as traceability, classification, and compliance.Source: CEUR WORKSHOP PROCEEDINGS, vol. 3672
See at:
ceur-ws.org | CNR IRIS | CNR IRIS
2024
Journal article
Open Access
Sustainable mobility: increase of capacity and digitisation of railway transport
Davide Basile, Maurice Ter Beek, Alessio Ferrari
Researchers from the Formal Methods and Tools (FMT) lab of CNR–ISTI work on the increase of capacity and digitisation of railway transport. The research is conducted in the context of the NextGenerationEU-funded project on “Railway Transportation” (Spoke 4), which is part of the National Centre for Sustainable Mobility (MOST).Source: ERCIM NEWS, vol. 138, pp. 8-9
Project(s): CN MOST (National Sustainable Mobility Centre) Spoke 4: Rail Transportation
Davide Basile, Maurice Ter Beek, Alessio Ferrari
Researchers from the Formal Methods and Tools (FMT) lab of CNR–ISTI work on the increase of capacity and digitisation of railway transport. The research is conducted in the context of the NextGenerationEU-funded project on “Railway Transportation” (Spoke 4), which is part of the National Centre for Sustainable Mobility (MOST).Source: ERCIM NEWS, vol. 138, pp. 8-9
Project(s): CN MOST (National Sustainable Mobility Centre) Spoke 4: Rail Transportation
See at:
ercim-news.ercim.eu | CNR IRIS | CNR IRIS
2024
Journal article
Open Access
Evaluating the understandability and user acceptance of Attack-Defense Trees: original experiment and replication
Broccia G., Ter Beek M. H., Lluch Lafuente A., Spoletini P., Fantechi A., Ferrari A.
Context: Attack-Defense Trees (ADTs) are a graphical notation used to model and evaluate security requirements. ADTs are popular because they facilitate communication among different stakeholders involved in system security evaluation and are formal enough to be verified using methods like model checking. The understandability and user-friendliness of ADTs are claimed as key factors in their success, but these aspects, along with user acceptance, have not been evaluated empirically. Objectives: This paper presents an experiment with 25 subjects designed to assess the understandability and user acceptance of the ADT notation, along with an internal replication involving 49 subjects. Methods: The experiments adapt the Method Evaluation Model (MEM) to examine understandability variables (i.e., effectiveness and efficiency in using ADTs) and user acceptance variables (i.e., ease of use, usefulness, and intention to use). The MEM is also used to evaluate the relationships between these dimensions. In addition, a comparative analysis of the results of the two experiments is carried out. Results: With some minor differences, the outcomes of the two experiments are aligned. The results demonstrate that ADTs are well understood by participants, with values of understandability variables significantly above established thresholds. They are also highly appreciated, particularly for their ease of use. The results also show that users who are more effective in using the notation tend to evaluate it better in terms of usefulness. Conclusion: These studies provide empirical evidence supporting both the understandability and perceived acceptance of ADTs, thus encouraging further adoption of the notation in industrial contexts, and development of supporting tools.Source: INFORMATION AND SOFTWARE TECHNOLOGY, vol. 178
DOI: 10.1016/j.infsof.2024.107624
Project(s): CODECS, Secure Internet of Things – Risk analysis in design and operation, Security-by-Design in Digital Denmark, Typeful Language Adaptation for Dynamic, Interacting and Evolving Systems
Metrics:
Broccia G., Ter Beek M. H., Lluch Lafuente A., Spoletini P., Fantechi A., Ferrari A.
Context: Attack-Defense Trees (ADTs) are a graphical notation used to model and evaluate security requirements. ADTs are popular because they facilitate communication among different stakeholders involved in system security evaluation and are formal enough to be verified using methods like model checking. The understandability and user-friendliness of ADTs are claimed as key factors in their success, but these aspects, along with user acceptance, have not been evaluated empirically. Objectives: This paper presents an experiment with 25 subjects designed to assess the understandability and user acceptance of the ADT notation, along with an internal replication involving 49 subjects. Methods: The experiments adapt the Method Evaluation Model (MEM) to examine understandability variables (i.e., effectiveness and efficiency in using ADTs) and user acceptance variables (i.e., ease of use, usefulness, and intention to use). The MEM is also used to evaluate the relationships between these dimensions. In addition, a comparative analysis of the results of the two experiments is carried out. Results: With some minor differences, the outcomes of the two experiments are aligned. The results demonstrate that ADTs are well understood by participants, with values of understandability variables significantly above established thresholds. They are also highly appreciated, particularly for their ease of use. The results also show that users who are more effective in using the notation tend to evaluate it better in terms of usefulness. Conclusion: These studies provide empirical evidence supporting both the understandability and perceived acceptance of ADTs, thus encouraging further adoption of the notation in industrial contexts, and development of supporting tools.Source: INFORMATION AND SOFTWARE TECHNOLOGY, vol. 178
DOI: 10.1016/j.infsof.2024.107624
Project(s): CODECS
, Secure Internet of Things – Risk analysis in design and operation, Security-by-Design in Digital Denmark, Typeful Language Adaptation for Dynamic, Interacting and Evolving SystemsMetrics:
See at:
Information and Software Technology | IRIS Cnr | IRIS Cnr | CNR IRIS
2024
Conference article
Open Access
Model generation with LLMs: from requirements to {UML} sequence diagrams
Ferrari A., Abualhaija S., Arora C.
Complementing natural language (NL) requirements with graphical models can improve stakeholders' communication and provide directions for system design. However, creating models from requirements involves manual effort. The advent of generative large language models (LLMs), ChatGPT being a notable example, offers promising avenues for automated assistance in model generation. This paper investigates the capability of ChatGPT to generate a specific type of model, i.e., UML sequence diagrams, from NL requirements. We conduct a qualitative study in which we examine the sequence diagrams generated by ChatGPT for 28 requirements documents of various types and from different domains. Observations from the analysis of the generated diagrams have systematically been captured through evaluation logs, and categorized through thematic analysis. Our results indicate that, although the models generally conform to the standard and exhibit a reasonable level of understandability, their completeness and correctness with respect to the specified requirements often present challenges. This issue is particularly pronounced in the presence of requirements smells, such as ambiguity and inconsistency. The insights derived from this study can influence the practical utilization of LLMs in the RE process, and open the door to novel RE-specific prompting strategies targeting effective model generation.DOI: 10.1109/rew61692.2024.00044
Metrics:
Ferrari A., Abualhaija S., Arora C.
Complementing natural language (NL) requirements with graphical models can improve stakeholders' communication and provide directions for system design. However, creating models from requirements involves manual effort. The advent of generative large language models (LLMs), ChatGPT being a notable example, offers promising avenues for automated assistance in model generation. This paper investigates the capability of ChatGPT to generate a specific type of model, i.e., UML sequence diagrams, from NL requirements. We conduct a qualitative study in which we examine the sequence diagrams generated by ChatGPT for 28 requirements documents of various types and from different domains. Observations from the analysis of the generated diagrams have systematically been captured through evaluation logs, and categorized through thematic analysis. Our results indicate that, although the models generally conform to the standard and exhibit a reasonable level of understandability, their completeness and correctness with respect to the specified requirements often present challenges. This issue is particularly pronounced in the presence of requirements smells, such as ambiguity and inconsistency. The insights derived from this study can influence the practical utilization of LLMs in the RE process, and open the door to novel RE-specific prompting strategies targeting effective model generation.DOI: 10.1109/rew61692.2024.00044
Metrics:
See at:
CNR IRIS | ieeexplore.ieee.org | CNR IRIS | CNR IRIS
2024
Other
Open Access
Predictive maintenance for railways: a case study
Millitarì G., Spagnolo G. O., Ferrari A.
Equipment failures, unplanned downtimes and the availability of spare parts significantly impact businesses that rely on assets, often resulting in production shutdowns and in- creased repair costs. Effective maintenance activities are essential for preventing such issues, and well-designed maintenance strategies can lead to reduced costs while enhanc- ing the efficiency and reliability of services. The railway sector, in particular, necessitates a substantial amount of maintenance to ensure smooth operations. This study presents the specific methodology developed to leverage various data provided by Trenord com- pany to assess the feasibility of implementing a predictive maintenance strategy within its decision-making process. By employing and comparing multiple classifiers, such as logis- tic regression with Lasso, KNN, random forest and boosting, alongside several rebalance methods, including SMOTE, undersampling techniques and moving threshold approach, a predictive model focused on a feature closely related to maintenance activities has been developed. The analysis aims to accurately estimate the probability that the majority of alerts on a given day would be classified as critical, based on limited information about the train’s status. The results obtained illustrate which features influence the criticality of alerts, while also highlighting the strengths and weaknesses of the various statistical learner and rebalance methods employed
Millitarì G., Spagnolo G. O., Ferrari A.
Equipment failures, unplanned downtimes and the availability of spare parts significantly impact businesses that rely on assets, often resulting in production shutdowns and in- creased repair costs. Effective maintenance activities are essential for preventing such issues, and well-designed maintenance strategies can lead to reduced costs while enhanc- ing the efficiency and reliability of services. The railway sector, in particular, necessitates a substantial amount of maintenance to ensure smooth operations. This study presents the specific methodology developed to leverage various data provided by Trenord com- pany to assess the feasibility of implementing a predictive maintenance strategy within its decision-making process. By employing and comparing multiple classifiers, such as logis- tic regression with Lasso, KNN, random forest and boosting, alongside several rebalance methods, including SMOTE, undersampling techniques and moving threshold approach, a predictive model focused on a feature closely related to maintenance activities has been developed. The analysis aims to accurately estimate the probability that the majority of alerts on a given day would be classified as critical, based on limited information about the train’s status. The results obtained illustrate which features influence the criticality of alerts, while also highlighting the strengths and weaknesses of the various statistical learner and rebalance methods employed
2024
Book
Open Access
Foreword to MO2RE 2024
Abualhaija S., Arora C., Ferrari A., Fucci D., Spoletini P.
Requirements engineering (RE) is a critical sub-field of soft- ware engineering (SE) that deals with identifying, specifying, modeling, analyzing, and validating the needs of stakeholders and constraints of a system [1]. RE covers human-related aspects, as stakeholders need to be involved in eliciting and validating the requirements, as well as more technical aspects, as requirements can be systematically collected (e.g., from app reviews) using data mining techniques and analyzed with natural language processing (NLP) approaches, e.g., to identify quality issues or trace links [2]. Despite the broad spectrum of activities that RE covers, researchers from outside RE often have a misconception that RE is limited to writing and analyzing requirements specifications. Consequently, many researchers in the SE community working on RE-relevant problems (e.g., human-centric SE) are often unaware that such problems belong to the RE research strands. Broadly speaking, RE is under-represented and under-appreciated in the SE community.
Abualhaija S., Arora C., Ferrari A., Fucci D., Spoletini P.
Requirements engineering (RE) is a critical sub-field of soft- ware engineering (SE) that deals with identifying, specifying, modeling, analyzing, and validating the needs of stakeholders and constraints of a system [1]. RE covers human-related aspects, as stakeholders need to be involved in eliciting and validating the requirements, as well as more technical aspects, as requirements can be systematically collected (e.g., from app reviews) using data mining techniques and analyzed with natural language processing (NLP) approaches, e.g., to identify quality issues or trace links [2]. Despite the broad spectrum of activities that RE covers, researchers from outside RE often have a misconception that RE is limited to writing and analyzing requirements specifications. Consequently, many researchers in the SE community working on RE-relevant problems (e.g., human-centric SE) are often unaware that such problems belong to the RE research strands. Broadly speaking, RE is under-represented and under-appreciated in the SE community.
See at:
CNR IRIS | www.computer.org | CNR IRIS
2024
Journal article
Open Access
Editorial for the REFSQ’23 special issue
Penzenstadler B., Ferrari A.
The International Working Conference on Requirement Engineering: Foundation for Software Quality (REFSQ) is an established international forum. Its goal is to foster the establishment and maintenance of a strong Require- ments Engineering (RE) community across industry and academia through contributions that report on novel ideas and techniques to enhance the quality of RE’s products and processes.Source: REQUIREMENTS ENGINEERING, vol. 29 (issue 1), pp. 1-2
DOI: 10.1007/s00766-024-00418-1
Metrics:
Penzenstadler B., Ferrari A.
The International Working Conference on Requirement Engineering: Foundation for Software Quality (REFSQ) is an established international forum. Its goal is to foster the establishment and maintenance of a strong Require- ments Engineering (RE) community across industry and academia through contributions that report on novel ideas and techniques to enhance the quality of RE’s products and processes.Source: REQUIREMENTS ENGINEERING, vol. 29 (issue 1), pp. 1-2
DOI: 10.1007/s00766-024-00418-1
Metrics:
See at:
Requirements Engineering | IRIS Cnr | IRIS Cnr | CNR IRIS
2023
Conference article
Open Access
Strategies, benefits and challenges of app store-inspired requirements elicitation
Ferrari A, Spoletini P
App store-inspired elicitation is the practice of exploring competitors' apps, to get inspiration for requirements. This activity is common among developers, but little insight is available on its practical use, advantages and possible issues. This paper aims to empirically analyse this technique in a realistic scenario, in which it is used to extend the requirements of a product that were initially captured by means of more traditional requirements elicitation interviews. Considering this scenario, we conduct an experimental simulation with 58 analysts and collect qualitative data. We perform thematic analysis of the data to identify strategies, benefits, and challenges of app store-inspired elicitation, as well as differences with respect to interviews in the considered elicitation setting. Our results show that: (1) specific guidelines and procedures are required to better conduct app store-inspired elicitation; (2) current search features made available by app stores are not suitable for this practice, and more tool support is required to help analysts in the retrieval and evaluation of competing products; (3) while interviews focus on the why dimension of requirements engineering (i.e., goals), app store-inspired elicitation focuses on how (i.e., solutions), offering indications for implementation and improved usability. Our study provides a framework for researchers to address existing challenges and suggests possible benefits to fostering app store-inspired elicitation among practitioners.DOI: 10.1109/icse48619.2023.00114
Metrics:
Ferrari A, Spoletini P
App store-inspired elicitation is the practice of exploring competitors' apps, to get inspiration for requirements. This activity is common among developers, but little insight is available on its practical use, advantages and possible issues. This paper aims to empirically analyse this technique in a realistic scenario, in which it is used to extend the requirements of a product that were initially captured by means of more traditional requirements elicitation interviews. Considering this scenario, we conduct an experimental simulation with 58 analysts and collect qualitative data. We perform thematic analysis of the data to identify strategies, benefits, and challenges of app store-inspired elicitation, as well as differences with respect to interviews in the considered elicitation setting. Our results show that: (1) specific guidelines and procedures are required to better conduct app store-inspired elicitation; (2) current search features made available by app stores are not suitable for this practice, and more tool support is required to help analysts in the retrieval and evaluation of competing products; (3) while interviews focus on the why dimension of requirements engineering (i.e., goals), app store-inspired elicitation focuses on how (i.e., solutions), offering indications for implementation and improved usability. Our study provides a framework for researchers to address existing challenges and suggests possible benefits to fostering app store-inspired elicitation among practitioners.DOI: 10.1109/icse48619.2023.00114
Metrics:
See at:
dl.acm.org | CNR IRIS | ISTI Repository | CNR IRIS | CNR IRIS
2023
Conference article
Open Access
Remote sensing and machine learning for riparian vegetation detection and classification
Fiorentini N., Bacco F. M., Ferrari A., Rovai M., Brunori G.
Precise and reliable identification of riparian vegetation along rivers is of paramount importance for managing bodies, enabling them to accurately plan key duties, such as the design of river maintenance interventions. Nonetheless, manual mapping is significantly expensive in terms of time and human costs, especially when authorities have to manage extensive river networks. Accordingly, in the present paper, we propose a methodology for classifying and automatically mapping the riparian vegetation of urban rivers. Specifically, the calibration of an unsupervised (Isodata Clustering) and a supervised (Random Forest) machine learning algorithm (MLA) is carried out for the classification of the riparian vegetation detected in high-resolution (1m) aerial orthoimages. Riparian vegetation is classified using Normalized Difference Vegetation Index (NDVI) features. In the framework of this research, the Isodata Clustering slightly outperforms the Random Forest, achieving a higher level of predictive performance and reliability throughout all the computed performance metrics. Moreover, being unsupervised, it does not require ground truth information, which makes it particularly competitive in terms of annotation costs when compared with supervised algorithms, and definitely appropriate in case of limited resources. We encourage river authorities to use MLA-based tools, such as the ones we propose in this work, for mapping riparian vegetation, since they can bring relevant benefits, such as limited implementation costs, easy calibration, fast training, and adequate reliability.DOI: 10.1109/metroagrifor58484.2023.10424205
DOI: 10.5281/zenodo.10805001
DOI: 10.5281/zenodo.10805002
Project(s): DESIRA, CODECS
Metrics:
Fiorentini N., Bacco F. M., Ferrari A., Rovai M., Brunori G.
Precise and reliable identification of riparian vegetation along rivers is of paramount importance for managing bodies, enabling them to accurately plan key duties, such as the design of river maintenance interventions. Nonetheless, manual mapping is significantly expensive in terms of time and human costs, especially when authorities have to manage extensive river networks. Accordingly, in the present paper, we propose a methodology for classifying and automatically mapping the riparian vegetation of urban rivers. Specifically, the calibration of an unsupervised (Isodata Clustering) and a supervised (Random Forest) machine learning algorithm (MLA) is carried out for the classification of the riparian vegetation detected in high-resolution (1m) aerial orthoimages. Riparian vegetation is classified using Normalized Difference Vegetation Index (NDVI) features. In the framework of this research, the Isodata Clustering slightly outperforms the Random Forest, achieving a higher level of predictive performance and reliability throughout all the computed performance metrics. Moreover, being unsupervised, it does not require ground truth information, which makes it particularly competitive in terms of annotation costs when compared with supervised algorithms, and definitely appropriate in case of limited resources. We encourage river authorities to use MLA-based tools, such as the ones we propose in this work, for mapping riparian vegetation, since they can bring relevant benefits, such as limited implementation costs, easy calibration, fast training, and adequate reliability.DOI: 10.1109/metroagrifor58484.2023.10424205
DOI: 10.5281/zenodo.10805001
DOI: 10.5281/zenodo.10805002
Project(s): DESIRA
, CODECS Metrics:
See at:
IRIS Cnr | ZENODO | IRIS Cnr | IRIS Cnr | ISTI Repository | doi.org | ZENODO | CNR IRIS | CNR IRIS
2023
Journal article
Open Access
Zero-shot learning for requirements classification: an exploratory study
Alhoshan W, Ferrari A, Zhao L
Context: Requirements engineering researchers have been experimenting with machine learning and deep learning approaches for a range of RE tasks, such as requirements classification, requirements tracing, ambiguity detection, and modelling. However, most of today's ML/DL approaches are based on supervised learning techniques, meaning that they need to be trained using a large amount of task-specific labelled training data. This constraint poses an enormous challenge to RE researchers, as the lack of labelled data makes it difficult for them to fully exploit the benefit of advanced ML/DL technologies. Objective: This paper addresses this problem by showing how a zero-shot learning approach can be used for requirements classification without using any labelled training data. We focus on the classification task because many RE tasks can be framed as classification problems. Method: The ZSL approach used in our study employs contextual word-embeddings and transformer-based language models. We demonstrate this approach through a series of experiments to perform three classification tasks: (1)FR/NFR: classification functional requirements vs non-functional requirements; (2)NFR: identification of NFR classes; (3)Security: classification of security vs non-security requirements. Results: The study shows that the ZSL approach achieves an F1 score of 0.66 for the FR/NFR task. For the NFR task, the approach yields F1~0.72-0.80, considering the most frequent classes. For the Security task, F1~0.66. All of the aforementioned F1 scores are achieved with zero-training efforts. Conclusion: This study demonstrates the potential of ZSL for requirements classification. An important implication is that it is possible to have very little or no training data to perform classification tasks. The proposed approach thus contributes to the solution of the long-standing problem of data shortage in RE.Source: INFORMATION AND SOFTWARE TECHNOLOGY, vol. 159
DOI: 10.1016/j.infsof.2023.107202
Metrics:
Alhoshan W, Ferrari A, Zhao L
Context: Requirements engineering researchers have been experimenting with machine learning and deep learning approaches for a range of RE tasks, such as requirements classification, requirements tracing, ambiguity detection, and modelling. However, most of today's ML/DL approaches are based on supervised learning techniques, meaning that they need to be trained using a large amount of task-specific labelled training data. This constraint poses an enormous challenge to RE researchers, as the lack of labelled data makes it difficult for them to fully exploit the benefit of advanced ML/DL technologies. Objective: This paper addresses this problem by showing how a zero-shot learning approach can be used for requirements classification without using any labelled training data. We focus on the classification task because many RE tasks can be framed as classification problems. Method: The ZSL approach used in our study employs contextual word-embeddings and transformer-based language models. We demonstrate this approach through a series of experiments to perform three classification tasks: (1)FR/NFR: classification functional requirements vs non-functional requirements; (2)NFR: identification of NFR classes; (3)Security: classification of security vs non-security requirements. Results: The study shows that the ZSL approach achieves an F1 score of 0.66 for the FR/NFR task. For the NFR task, the approach yields F1~0.72-0.80, considering the most frequent classes. For the Security task, F1~0.66. All of the aforementioned F1 scores are achieved with zero-training efforts. Conclusion: This study demonstrates the potential of ZSL for requirements classification. An important implication is that it is possible to have very little or no training data to perform classification tasks. The proposed approach thus contributes to the solution of the long-standing problem of data shortage in RE.Source: INFORMATION AND SOFTWARE TECHNOLOGY, vol. 159
DOI: 10.1016/j.infsof.2023.107202
Metrics:
See at:
CNR IRIS | ISTI Repository | ISTI Repository | ISTI Repository | www.sciencedirect.com | CNR IRIS | CNR IRIS | CNR IRIS
2023
Conference article
Metadata Only Access
Digitalising agriculture: design and development of a modelling web environment for end-users
Mannari C, Anichini E, Malizia A, Turchi T, Ferrari A, Bacco M
Modern digital technologies have a promising potential in the development of sustainable agriculture. For example, cloud computers powered by 5G IoT components allow the development of sophisticated applications -- e.g., for food traceability, pest detection, automatic irrigation -- also making use of different AI-based techniques. At the same time, digitalisation in agriculture is considered a socio-technical process to be evaluated by different stakeholders through collaborative approaches. Model-based requirements engineering strategies, which leverage diagrammatic notations, can support information exchange between different domains. Based on the evaluation of different solutions, we found a lack of tools for designing and modelling systems accessible to end-users. Most professional tools are desktop applications based on complex interactions, limited user experience, or generalist web platforms needing more formal components. In our study, we developed a prototype of a web environment for modelling systems based on a visual language that can be exported into standard code. We aim to involve end-users in modelling their digital ecosystems as a preliminary task for developing further applications. The prototype was evaluated in a workshop with experts following the cognitive walkthrough methodology for usability inspection. The evaluation highlighted the UI requirements to support easy-to-understand visual elements and maximise the understanding of actions while limiting user errors.
Mannari C, Anichini E, Malizia A, Turchi T, Ferrari A, Bacco M
Modern digital technologies have a promising potential in the development of sustainable agriculture. For example, cloud computers powered by 5G IoT components allow the development of sophisticated applications -- e.g., for food traceability, pest detection, automatic irrigation -- also making use of different AI-based techniques. At the same time, digitalisation in agriculture is considered a socio-technical process to be evaluated by different stakeholders through collaborative approaches. Model-based requirements engineering strategies, which leverage diagrammatic notations, can support information exchange between different domains. Based on the evaluation of different solutions, we found a lack of tools for designing and modelling systems accessible to end-users. Most professional tools are desktop applications based on complex interactions, limited user experience, or generalist web platforms needing more formal components. In our study, we developed a prototype of a web environment for modelling systems based on a visual language that can be exported into standard code. We aim to involve end-users in modelling their digital ecosystems as a preliminary task for developing further applications. The prototype was evaluated in a workshop with experts following the cognitive walkthrough methodology for usability inspection. The evaluation highlighted the UI requirements to support easy-to-understand visual elements and maximise the understanding of actions while limiting user errors.
See at:
CNR IRIS
2023
Conference article
Open Access
Evaluating a language workbench: from working memory capacity to comprehension to acceptance
Broccia G, Ferrari A, Ter Beek Mh, Cazzola W, Favalli L, Bertolotti F
Language workbenches are tools that enable the definition, reuse and composition of programming languages and their ecosystem. This breed of frameworks aims to make the development of new languages easier and more affordable. Consequently, the comprehensibility of the language used in a language workbench (i.e., the meta-language) should be an important aspect to consider and evaluate. To the best of our knowledge, although the quantitative aspects of language workbenches are often discussed in the literature, the evaluation of comprehensibility is typically neglected.Neverlang is a language workbench that enables the definition of languages with a modular approach. This paper presents a preliminary study that intends to assess the comprehensibility of Neverlang programs, evaluated in terms of users' effectiveness and efficiency in a code comprehension task. The study also investigates the relationship between Neverlang comprehensibility and the users' working memory capacity. Furthermore, we intend to capture the relationship between Neverlang comprehensibility and users' acceptance, in terms of perceived ease of use, perceived usefulness, and intention to use. Our preliminary results on 10 subjects suggest that the users' working memory capacity may be related to the ability to comprehend Neverlang programs. On the other hand, effectiveness and efficiency do not appear to be associated with an increase in users' acceptance variables.DOI: 10.1109/icpc58990.2023.00017
Project(s): Typeful Language Adaptation for Dynamic, Interacting and Evolving Systems
Metrics:
Broccia G, Ferrari A, Ter Beek Mh, Cazzola W, Favalli L, Bertolotti F
Language workbenches are tools that enable the definition, reuse and composition of programming languages and their ecosystem. This breed of frameworks aims to make the development of new languages easier and more affordable. Consequently, the comprehensibility of the language used in a language workbench (i.e., the meta-language) should be an important aspect to consider and evaluate. To the best of our knowledge, although the quantitative aspects of language workbenches are often discussed in the literature, the evaluation of comprehensibility is typically neglected.Neverlang is a language workbench that enables the definition of languages with a modular approach. This paper presents a preliminary study that intends to assess the comprehensibility of Neverlang programs, evaluated in terms of users' effectiveness and efficiency in a code comprehension task. The study also investigates the relationship between Neverlang comprehensibility and the users' working memory capacity. Furthermore, we intend to capture the relationship between Neverlang comprehensibility and users' acceptance, in terms of perceived ease of use, perceived usefulness, and intention to use. Our preliminary results on 10 subjects suggest that the users' working memory capacity may be related to the ability to comprehend Neverlang programs. On the other hand, effectiveness and efficiency do not appear to be associated with an increase in users' acceptance variables.DOI: 10.1109/icpc58990.2023.00017
Project(s): Typeful Language Adaptation for Dynamic, Interacting and Evolving Systems
Metrics:
See at:
CNR IRIS | ieeexplore.ieee.org | ISTI Repository | CNR IRIS | CNR IRIS
2023
Other
Open Access
DESIRA - D3.5 Third set of practice abstracts
Bacco Fm, Ferrari A, Berg M, Schroth C, Rendl C, Marinoskouris C, Toli E, Koltsida P, Ortolani L, Lepore F, Townsend L, Hardy C, Fiorentini N, Brunori G
This document provides DESIRA's third set of practice abstracts (PAs) which is a compilation of seven PAs. Those PAs are based on the experiences, lessons learned, project actions and reporting of the WP3 activities that aimed at the development of scenarios and showcasing of technologies building on the concept of digital game changers. Tasks 3.5 'Use Case development' and 3.6 'Showcase Technologies', are the main contributing project tasks that provided concrete results on which these seven PAs, cited in this report, are based. The first five PAs provide a concise description of five use cases that were developed during the second period of the DESIRA project. An array of conducted activities inside the boundaries of five preselected Living Labs, and with the participation of those LL's stakeholders, were planned so that WP3 could culminate in the development of five technology adoption use cases.The last two PAs of this report, supplement the five aforementioned use case PAs by showcasing two additional promising technology solutions that have the potential to contribute to sustainable digital transition pathways, as those are defined by the DESIRA's theoretical framework and as reflectedby the examined agro-rural-forestry settings of this project. For a thorough and detailed analysis of the methodology, activities, and outcomes that contributed to the production of the use cases and showcase technology reports, it is recommended the reading of deliverables D3.3 'Use Cases Report'and D3.4 'Showcase Technology Report' which are the foundation documents on which this deliverable is based on.Project(s): DESIRA
Bacco Fm, Ferrari A, Berg M, Schroth C, Rendl C, Marinoskouris C, Toli E, Koltsida P, Ortolani L, Lepore F, Townsend L, Hardy C, Fiorentini N, Brunori G
This document provides DESIRA's third set of practice abstracts (PAs) which is a compilation of seven PAs. Those PAs are based on the experiences, lessons learned, project actions and reporting of the WP3 activities that aimed at the development of scenarios and showcasing of technologies building on the concept of digital game changers. Tasks 3.5 'Use Case development' and 3.6 'Showcase Technologies', are the main contributing project tasks that provided concrete results on which these seven PAs, cited in this report, are based. The first five PAs provide a concise description of five use cases that were developed during the second period of the DESIRA project. An array of conducted activities inside the boundaries of five preselected Living Labs, and with the participation of those LL's stakeholders, were planned so that WP3 could culminate in the development of five technology adoption use cases.The last two PAs of this report, supplement the five aforementioned use case PAs by showcasing two additional promising technology solutions that have the potential to contribute to sustainable digital transition pathways, as those are defined by the DESIRA's theoretical framework and as reflectedby the examined agro-rural-forestry settings of this project. For a thorough and detailed analysis of the methodology, activities, and outcomes that contributed to the production of the use cases and showcase technology reports, it is recommended the reading of deliverables D3.3 'Use Cases Report'and D3.4 'Showcase Technology Report' which are the foundation documents on which this deliverable is based on.Project(s): DESIRA
See at:
CNR IRIS | ISTI Repository | CNR IRIS
2023
Conference article
Open Access
ModeLLer - a prototype to support requirements elicitation in co-design environments
Mannari C., Anichini E., Bacco F. M., Ferrari A., Turchi T., Malizia A.
This contribution presents ModeLLer, a prototype of a web tool for system modelling based on a block-based visual editor. The aim of ModeLLer is to enable collaborative environments in requirements elicitation, allowing end-users to create UML class diagrams without any knowledge of the (semi-)formal UML notation.DOI: 10.1109/re57278.2023.00055
Project(s): CODECS
Metrics:
Mannari C., Anichini E., Bacco F. M., Ferrari A., Turchi T., Malizia A.
This contribution presents ModeLLer, a prototype of a web tool for system modelling based on a block-based visual editor. The aim of ModeLLer is to enable collaborative environments in requirements elicitation, allowing end-users to create UML class diagrams without any knowledge of the (semi-)formal UML notation.DOI: 10.1109/re57278.2023.00055
Project(s): CODECS
Metrics:
See at:
Archivio della Ricerca - Università di Pisa | Archivio della Ricerca - Università di Pisa | IRIS Cnr | Archivio della Ricerca - Università di Pisa | ISTI Repository | IRIS Cnr | doi.org | CNR IRIS | CNR IRIS