Replication in Requirements Engineering: the NLP for RE case
Abualhaija S., Aydemir F. B., Dalpiaz F., Dell'Anna D., Ferrari A., Franch X., Fucci D.
ID cards Àrees temàtiques de la UPC::Informàtica::Enginyeria del software NLP FOS: Computer and information sciences Language processing ID card Employment Requirement engineering Natural language processing Àrees temàtiques de la UPC::Informàtica::Intel·ligència artificial::Llenguatge natural Iterative methods tool reconstruction Requirements engineering replication annotation Programvaruteknik Natural languages Language processing techniques Natural Language Processing (NLP) Tool reconstruction Requirements Engineering Requirements Engineering (RE) Annotation Computer Science - Software Engineering Natural language processing systems Software Engineering Replication Software Engineering (cs.SE)
Natural language processing (NLP) techniques have been widely applied in the requirements engineering (RE) field to support tasks such as classification and ambiguity detection. Despite its empirical vocation, RE research has given limited attention to replication of NLP for RE studies. Replication is hampered by several factors, including the context specificity of the studies, the heterogeneity of the tasks involving NLP, the tasks’ inherent hairiness, and, in turn, the heterogeneous reporting structure. To address these issues, we propose a new artifact, referred to as ID-Card, whose goal is to provide a structured summary of research papers emphasizing replication-relevant information. We construct the ID-Card through a structured, iterative process based on design science. In this article: (i) we report on hands-on experiences of replication; (ii) we review the state-of-the-art and extract replication-relevant information: (iii) we identify, through focus groups, challenges across two typical dimensions of replication: data annotation and tool reconstruction; and (iv) we present the concept and structure of the ID-Card to mitigate the identified challenges. This study aims to create awareness of replication in NLP for RE. We propose an ID-Card that is intended to foster study replication but can also be used in other contexts, e.g., for educational purposes.
Source: ACM TRANSACTIONS ON SOFTWARE ENGINEERING AND METHODOLOGY, vol. 33 (issue 6), pp. 1-33
[2] Anaphora. 2022. Definition of anaphora. https://www.lexico.com/definition/anaphora Oxford University Press. Last accessed: 16 May 2022.
[3] Authors. 2022. “Online Annex”. Available at https://figshare.com/s/a7ca0228c7a98ab882ca,.
[4] D. Berry, E. Kamsties, and M. Krieger. 2003. From Contract Drafting to Software Specification: Linguistic Sources of Ambiguity, A Handbook. http://se.uwaterloo.ca/~dberry/handbook/ambiguityHandbook.pdf
[5] Rosanna L Breen. 2006. A practical guide to focus-group research. Journal of geography in higher education 30, 3 (2006), 463-475.
[6] Danqi Chen and Christopher Manning. 2014. A Fast and Accurate Dependency Parser using Neural Networks. In Proceedings of the 18th Conference on Empirical Methods in Natural Language Processing (EMNLP'14).
[7] Jane Cleland-Huang, Rafaella Settimi, Xuchang Zou, and Peter Solc. 2006. The detection and classification of non-functional requirements with application to early aspects. In 14th IEEE International Requirements Engineering Conference. 39-48.
[8] Jane Cleland-Huang, Rafaella Settimi, Xuchang Zou, and Peter Solc. 2007. Automated classification of non-functional requirements. Requirements engineering 12, 2 (2007), 103-120.
[9] Margarita Cruz, Beatriz Bernárdez, Amador Durán, Jose A Galindo, and Antonio Ruiz-Cortés. 2019. Replication of studies in empirical software engineering: A systematic mapping study, from 2013 to 2018. IEEE Access 8 (2019), 26773-26791.
[10] Fabio QB Da Silva, Marcos Suassuna, A César C França, Alicia M Grubb, Tatiana B Gouveia, Cleviton VF Monteiro, and Igor Ebrahim dos Santos. 2014. Replication of empirical studies in software engineering research: a systematic mapping study. Empirical Software Engineering 19, 3 (2014), 501-557.
[11] Fabiano Dalpiaz, Davide Dell'Anna, Fatma Basak Aydemir, and Sercan Çevikol. 2019. Requirements Classification with Interpretable Machine Learning and Dependency Parsing. In Proceedings of the 27th IEEE International Requirements Engineering Conference, RE 2019. 142-152. https: //doi.org/10.1109/RE.2019.00025
[12] Fabiano Dalpiaz, Davide Dell'Anna, Fatma Başak Aydemir, and Sercan Çevikol. 2019. explainable-re/RE-2019-Materials v0. https://doi.org/10.5281/ zenodo.3309669
[13] Fabiano Dalpiaz, Alessio Ferrari, Xavier Franch, and Cristina Palomares. 2018. Natural language processing for requirements engineering: The best is yet to come. IEEE software 35, 5 (2018), 115-119.
[14] Fabiano Dalpiaz, Ivor Schalk, and Garm Lucassen. 2018. Pinpointing Ambiguity and Incompleteness in Requirements Engineering via Information Visualization and NLP. In Proceedings of the 24th Working Conference on Requirements Engineering: Foundation for Software Quality (REFSQ'18).
[15] Fred D. Davis. 1989. Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology. MIS Quarterly 13, 3 (1989), 319-340. http://www.jstor.org/stable/249008
[16] Saad Ezzini, Sallam Abualhaija, Chetan Arora, and Mehrdad Sabetzadeh. 2021. taphsir v6. https://doi.org/10.5281/zenodo.6009790
[17] Saad Ezzini, Sallam Abualhaija, Chetan Arora, and Mehrdad Sabetzadeh. 2022. Automated Handling of Anaphoric Ambiguity in Requirements: A Multi-solution Study. In 2022 IEEE/ACM 44th International Conference on Software Engineering.
[18] Saad Ezzini, Sallam Abualhaija, Chetan Arora, Mehrdad Sabetzadeh, and Lionel C Briand. 2021. Using domain-specific corpora for improved handling of ambiguity in requirements. In 2021 IEEE/ACM 43rd International Conference on Software Engineering.
[19] D Méndez Fernández, Stefan Wagner, Marcos Kalinowski, Michael Felderer, Priscilla Mafra, Antonio Vetrò, Tayana Conte, M-T Christiansson, Des Greer, Casper Lassenius, et al. 2017. Naming the pain in requirements engineering. Empirical software engineering 22, 5 (2017), 2298-2338.
[20] Alessio Ferrari, Felice Dell'Orletta, Andrea Esuli, Vincenzo Gervasi, and Stefania Gnesi. 2017. Natural Language Requirements Processing: A 4D Vision. IEEE Softw. 34, 6 (2017), 28-35.
[21] Alessio Ferrari, Giorgio Oronzo Spagnolo, and Stefania Gnesi. 2017. PURE: A Dataset of Public Requirements Documents. In 2017 IEEE 25th International Requirements Engineering Conference. 502-505. https://doi.org/10.1109/RE.2017.29
[22] Vincenzo Gervasi, Alessio Ferrari, Didar Zowghi, and Paola Spoletini. 2019. Ambiguity in Requirements Engineering: Towards a Unifying Framework. In From Software Engineering to Formal Methods and Tools, and Back. Springer.
[23] Martin Glinz. 2007. On non-functional requirements. In 15th IEEE international requirements engineering conference (RE 2007). IEEE, 21-26.
[24] Ben Hermann, Stefan Winter, and Janet Siegmund. 2020. Community expectations for research artifacts and evaluation processes. In Proceedings of the 28th ACM joint meeting on european software engineering conference and symposium on the foundations of software engineering. 469-480.
[25] Natalia Juristo and Sira Vegas. 2011. The role of non-exact replications in software engineering experiments. Empirical Software Engineering 16, 3 (2011), 295-324.
[26] Erik Kamsties and Barbara Peach. 2000. Taming ambiguity in natural language requirements. In Proceedings of the 13th International Conference on Software and Systems Engineering and Applications (ICSSEA'00).
[27] Mohamad Kassab, Colin Neill, and Phillip Laplante. 2014. State of practice in requirements engineering: contemporary data. Innovations in Systems and Software Engineering 10, 4 (2014), 235-241.
[28] Nikita Kitaev and Dan Klein. 2018. Constituency Parsing with a Self-Attentive Encoder. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, Iryna Gurevych and Yusuke Miyao (Eds.). Association for Computational Linguistics, 2676-2686. https://doi.org/10.18653/v1/P18-1249
[29] Nadzeya Kiyavitskaya, Nicola Zeni, Luisa Mich, and Daniel Berry. 2008. Requirements for tools for ambiguity identification and measurement in natural language requirements specifications. Requirements Engineering 13, 3 (2008).
[30] Zijad Kurtanović and Walid Maalej. 2017. Automatically classifying functional and non-functional requirements using supervised machine learning. In 2017 IEEE 25th International Requirements Engineering Conference (RE). Ieee, 490-495.
[31] Zijad Kurtanović and Walid Maalej. 2018. On user rationale in software engineering. Requirements Engineering 23, 3 (2018), 357-379.
[32] J. Richard Landis and Gary G. Koch. 1977. An Application of Hierarchical Kappa-type Statistics in the Assessment of Majority Agreement among Multiple Observers. Biometrics 33, 2 (1977).
[33] Feng-Lin Li, Jennifer Horkof, John Mylopoulos, Renata SS Guizzardi, Giancarlo Guizzardi, Alexander Borgida, and Lin Liu. 2014. Non-functional requirements as qualities, with a spice of ontology. In 2014 IEEE 22nd International Requirements Engineering Conference (RE). IEEE, 293-302.
[34] Lloyd Montgomery, Davide Fucci, Abir Bourafa, Lisa Scholz, and Walid Maalej. 2022. Empirical research on requirements quality: a systematic mapping study. Requirements Engineering (2022), 1-27.
[35] James Pustejovsky and Amber Stubbs. 2012. Natural Language Annotation for Machine Learning: A guide to corpus-building for applications. O'Reilly Media, Inc.
[36] Martin Shepperd, Nemitari Ajienka, and Steve Counsell. 2018. The role and value of replication in empirical software engineering results. Information and Software Technology 99 (2018), 120-132.
[37] Forrest J Shull, Jefrey C Carver, Sira Vegas, and Natalia Juristo. 2008. The role of replications in empirical software engineering. Empirical software engineering 13, 2 (2008), 211-218.
[38] Rini Van Solingen, Vic Basili, Gianluigi Caldiera, and H Dieter Rombach. 2002. Goal Question Metric (GQM) approach. Encyclopedia of Software Engineering (2002).
[39] Roel J Wieringa. 2014. Design science methodology for information systems and software engineering. Springer.
[40] Jonas Winkler and Andreas Vogelsang. 2016. Automatic Classification of Requirements Based on Convolutional Neural Networks. In 24th IEEE International Requirements Engineering Conference, RE 2016, Beijing, China, September 12-16, 2016. IEEE Computer Society, 39-45. https: //doi.org/10.1109/REW.2016.021
[41] Stefan Winter, Christopher S Timperley, Ben Hermann, Jurgen Cito, Jonathan Bell, Michael Hilton, and Dirk Beyer. 2022. A retrospective study of one decade of artifact evaluations. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 145-156.
[42] Claes Wohlin, Per Runeson, Martin Höst, Magnus C Ohlsson, Björn Regnell, and Anders Wesslén. 2012. Experimentation in software engineering. Springer Science & Business Media.
[43] Hui Yang, Anne de Roeck, Vincenzo Gervasi, Alistair Willis, and Bashar Nuseibeh. 2011. Analysing anaphoric ambiguity in natural language requirements. Requirements Engineering 16, 3 (may 2011), 163. https://doi.org/10.1007/s00766-011-0119-y
[44] Hui Yang, Anne de Roeck, Alistair Willis, and Bashar Nuseibeh. 2010. A Methodology for Automatic Identification of Nocuous Ambiguity. In Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010). Coling 2010 Organizing Committee, Beijing, China, 1218-1226. https://aclanthology.org/C10-1137
[45] Liping Zhao, Waad Alhoshan, Alessio Ferrari, Keletso J Letsholo, Muideen A Ajagbe, Erol-Valeriu Chioasca, and Riza T Batista-Navarro. 2021. Natural Language Processing for Requirements Engineering: A Systematic Mapping Study. ACM Computing Surveys (CSUR) 54, 3 (2021), 1-41.
Metrics
Back to previous page
@article{oai:iris.cnr.it:20.500.14243/499633,
title = {Replication in Requirements Engineering: the NLP for RE case},
author = {Abualhaija S. and Aydemir F. B. and Dalpiaz F. and Dell'Anna D. and Ferrari A. and Franch X. and Fucci D.},
doi = {10.1145/3658669 and 10.48550/arxiv.2304.10265},
year = {2024}
}
0000-0002-0636-5663
CODECS
Maximising the CO-benefits of agricultural Digitalisation through conducive digital ECoSystems
