Insights from running flaky tests into the field: extended version

Barboni M., Bertolino A., De Angelis G.

Software testing  Flaky test  In Vivo testing  Intermittently failing test 

Test flakiness is a topmost concern in software test automation. While conducting pre-deployment testing, those tests that are flagged as ?flaky are put aside for being either repaired or discarded. We hypothesize that some flaky tests could provide useful insights if run in the field, and could help identify hard-to-detect failures that escape testing and present themselves in operation. We present the first study to investigate the behaviour of flaky tests when moved to the field. Our experimentation over 52 test methods labelled as flaky provides a first confirmation that moving from the laboratory to the field, the behaviour of tests changes and, in particular, the failure frequency of intermittently failing tests can increase. More importantly, we could identify few cases of field failures that would have been hard to detect while testing in house.

Source: ISTI Technical Report, ISTI-2022-TR/007, 2022

Metrics