Inconsistency Detection in Natural Language Requirements using ChatGPT: a Preliminary Evaluation

Fantechi A., Gnesi S., Passaro L., Semini L.

ChatGPT  Natural Language Requirements  Inconsistency Detection 

With the rapid advancement of tools based on Artificial Intelligence, it is interesting to assess their usefulness in requirements engineering. In early experiments, we have seen that ChatGPT can detect inconsistency defects in natural language (NL) requirements, that traditional NLP tools cannot identify or can identify with difficulties even after domain-focused training. This study is devoted to specifically measuring the performance of ChatGPT in finding inconsistency in requirements. Positive results in this respect could lead to the use of ChatGPT to complement existing requirements analysis tools to automatically detect this important quality criterion. For this purpose, we consider GPT-3.5, the Generative Pretrained Transformer language model developed by OpenAI. We evaluate its ability to detect inconsistency by comparing its predictions with those obtained from expert judgments by students with a proven knowledge of RE issues on a few example requirements documents.

Source: RE 2023 - 31st IEEE International Requirements Engineering Conference, Hannover, Germany, 04-08/09/2023

Metrics