About the assessment of Grey Literature in Software Engineering

De Angelis G., Lonetti F.

Grey Literature  Quality assessment  Likert scale  Fuzzy rating scale 

There is an ongoing interest in the Software Engineering field for multivocal literature reviews including grey literature. However, at the same time, the role of the grey literature is still controversial, and the benefits of its inclusion in systematic reviews are object of discussion. Some of these arguments concern the quality assessment methods for grey literature entries, which is often considered a challenging and critical task. On the one hand, apart from a few proposals, there is a lack of an acknowledged methodological support for the inclusion of Software Engineering grey literature in systematic surveys. On the other hand, the unstructured shape of the grey literature contents could lead to bias in the evaluation process impacting on the quality of the surveys. This work leverages an approach on fuzzy Likert scales, and it proposes a methodology for managing the explicit uncertainties emerging during the assessment of entries from the grey literature. The methodology also strengthens the adoption of consensus policies that take into account the individual confidence level expressed for each of the collected scores.

Source: PROPSER 2021 - International Workshop on Properties of Software Engineering Research, co-located with EASE 2021 - Evaluation and Assessment in Software Engineering, pp. 373–378, Trondheim, Norway and Online, 23/06/2021

Publisher: ACM Press, New York, USA

Metrics