Software regression testing orchestration: because so many techniques need a conductor (and not necessarily a human one)

Bertolino A.

Software testing  Regression testing 

Software regression testing aims at ensuring that introduced changes do not cause previously passing tests to fail. Testing consumes a great part of software development and maintenance effort, and regression testing is by far the most expensive among testing activities. Along three decades, research has proposed many techniques that address the cost-effective software regression testing along different dimensions, including, among others, prioritization, minimization, selection, and amplification techniques. Notwithstanding, the problem remains prominent, even more by considering continuous integration practices that promote frequent change commits and retesting. In our work we point to two issues in current research on software regression testing: it is fragmented, in that the large part of proposed approaches only address one problem dimension at a time, and it often relies on unrealistic assumptions that make most proposed solutions hardly applicable. We addressed the latter issue in a recent systematic review of literature, complemented with a survey of professional testers. As a result of such an effort, we share a live open repository of current techniques and tools. To address the former issue, we introduced the notion of regression test orchestration, which consists of combining different techniques into one synergic strategy. Some initial studies (unsurprisingly) confirm that approaching regression testing by using properly combined techniques can often provide more efficient and effective solutions. Indeed, to identify the most effective orchestration strategies we need to perform empirical studies that compare and combine the many existing techniques. We invite the community to the conduction of more similar studies. For the future in this keynote we also hint at a regression test orchestration framework that ideally can autonomously manage itself by continuously monitoring the testing process and by proposing an optimized test campaign via the embedding of suitable heuristics and learning approaches. Such an autonomic test orchestrator would alleviate testers from the effort-demanding regression testing practices, allowing them to better spend their energy into other tasks requiring human attention.

Source: ISSRE 2023 - 34th IEEE International Symposium on Software Reliability Engineering, pp. 28–29, Florence, Italy, 9-12/10/2023

Metrics