2022
Conference article
Open Access
Towards agent-based testing of 3D games using reinforcement learning
Ferdous R., Kifetew F., Prandi D., Susi A.
functional coverage game play testing Reinforcement learning Functional coverage Game play testing reinforcement learning
Computer game is a billion-dollar industry and is booming. Testing games has been recognized as a difficult task, which mainly relies on manual playing and scripting based testing. With the advances in technologies, computer games have become increasingly more interactive and complex, thus play-testing using human participants alone has become unfeasible. In recent days, play-testing of games via autonomous agents has shown great promise by accelerating and simplifying this process. Reinforcement Learning solutions have the potential of complementing current scripted and automated solutions by learning directly from playing the game without the need of human intervention. This paper presented an approach based on reinforcement learning for automated testing of 3D games. We make use of the notion of curiosity as a motivating factor to encourage an RL agent to explore its environment. The results from our exploratory study are promising and we have preliminary evidence that reinforcement learning can be adopted for automated testing of 3D games.
Publisher: Association for Computing Machinery
[1] David Adamo, Md Khorrom Khan, Sreedevi Koppula, and Renée Bryce. 2018. Reinforcement learning for android gui testing. In Proceedings of the 9th ACM SIGSOFT International Workshop on Automating TEST Case Design, Selection, and Evaluation. 2-8.
[2] Thomas H Apperley. 2006. Genre and game studies: Toward critical approach to video game genres. Simulation and Gaming 37, 1 (2006), 6-23.
[3] Sinan Ariyurek, Aysu Betin-Can, and Elif Surer. 2019. Automated video game testing using synthetic and human-like agents. IEEE Transactions on Games (2019).
[4] Arthur Aubret, Laetitia Matignon, and Salima Hassas. 2019. A survey on intrinsic motivation in reinforcement learning. arXiv preprint arXiv:1908.06976 (2019).
[5] Joakim Bergdahl, Camilo Gordillo, Konrad Tollmar, and Linus Gisslén. 2020. Augmenting automated game testing with deep reinforcement learning. In 2020 IEEE Conference on Games (CoG). IEEE, 600-603.
[6] Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemysław Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. 2019. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06680 (2019).
[7] Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A Efros. 2018. Large-scale study of curiosity-driven learning. arXiv preprint arXiv:1808.04355 (2018).
[8] Avinash K Dixit, John JF Sherrerd, et al. 1990. Optimization in economic theory. Oxford University Press on Demand.
[9] Raihana Ferdous, Fitsum Meshesha Kifetew, Davide Prandi, I. S. W. B. Prasetya, Samira Shirzadehhajimahmood, and Angelo Susi. 2021. Search-Based Automated Play Testing of Computer Games: A Model-Based Approach. In Search-Based Software Engineering - 13th International Symposium, SSBSE 2021, Bari, Italy, October 11-12, 2021, Proceedings (Lecture Notes in Computer Science, Vol. 12914), Una-May O'Reilly and Xavier Devroey (Eds.). Springer, 56-71. https://doi.org/ 10.1007/978-3-030-88106-1_5
[10] Camilo Gordillo, Joakim Bergdahl, Konrad Tollmar, and Linus Gisslén. 2021. Improving Playtesting Coverage via Curiosity Driven Reinforcement Learning Agents. arXiv preprint arXiv:2103.13798 (2021).
[11] Oliver Groth, Markus Wulfmeier, Giulia Vezzani, Vibhavari Dasagi, Tim Hertweck, Roland Hafner, Nicolas Heess, and Martin Riedmiller. 2021. Is curiosity all you need? on the utility of emergent behaviours from curious exploration. arXiv preprint arXiv:2109.08603 (2021).
[12] Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. 2016. Vime: Variational information maximizing exploration. Advances in neural information processing systems 29 (2016).
[13] Paul Jaccard. 1912. The distribution of the flora in the alpine zone. 1. New phytologist 11, 2 (1912), 37-50.
[14] Guillaume Lample and Devendra Singh Chaplot. 2017. Playing FPS games with deep reinforcement learning. In Thirty-First AAAI Conference on Artificial Intelligence.
[15] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. 2013. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 (2013).
[16] Newzoo. 2021. Global games market report. https://newzoo.com/insights/ articles/the-games-market-in-2021-the-year-in-numbers-esports-cloudgaming/
[17] Georg Ostrovski, Marc G Bellemare, Aäron Oord, and Rémi Munos. 2017. Countbased exploration with neural density models. In International conference on machine learning. PMLR, 2721-2730.
[18] Minxue Pan, An Huang, Guoxin Wang, Tian Zhang, and Xuandong Li. 2020. Reinforcement learning based curiosity-driven testing of android applications. In Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis. 153-164.
[19] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. 2017. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning. PMLR, 2778-2787.
[20] Alexandre Péré, Sébastien Forestier, Olivier Sigaud, and Pierre-Yves Oudeyer. 2018. Unsupervised learning of goal spaces for intrinsically motivated goal exploration. arXiv preprint arXiv:1803.00781 (2018).
[21] Cristiano Politowski, Fabio Petrillo, and Yann-Gaël Guéhéneuc. 2021. A survey of video game testing. In 2021 IEEE/ACM International Conference on Automation of Software Test (AST). IEEE, 90-99.
[22] Rui Prada, I. S. W. B. Prasetya, Fitsum Meshesha Kifetew, Frank Dignum, Tanja E. J. Vos, Jason Lander, Jean-Yves Donnart, Alexandre Kazmierowski, Joseph Davidson, and Pedro M. Fernandes. 2020. Agent-based Testing of Extended Reality Systems. In 13th IEEE International Conference on Software Testing, Validation and Verification, ICST 2020, Porto, Portugal, October 24-28, 2020 . IEEE, 414-417. https://doi.org/10.1109/ICST46399.2020.00051
[23] ISWB Prasetya, Mehdi Dastani, Rui Prada, Tanja EJ Vos, Frank Dignum, and Fitsum Kifetew. 2020. Aplib: Tactical Agents for Testing Computer Games. In International Workshop on Engineering Multi-Agent Systems (EMAS). Springer, 21-41.
[24] I. S. W. B. Prasetya, Mehdi Dastani, Rui Prada, Tanja E. J. Vos, Frank Dignum, and Fitsum Meshesha Kifetew. 2020. Aplib: Tactical Agents for Testing Computer Games. In Engineering Multi-Agent Systems - 8th International Workshop, EMAS 2020, Auckland, New Zealand, May 8-9, 2020, Revised Selected Papers (Lecture Notes in Computer Science, Vol. 12589), Cristina Baroglio, Jomi Fred Hübner, and Michael Winikof (Eds.). Springer, 21-41. https://doi.org/10.1007/978-3-030-66534-0_2
[25] Nikolay Savinov, Anton Raichuk, Raphaël Marinier, Damien Vincent, Marc Pollefeys, Timothy Lillicrap, and Sylvain Gelly. 2018. Episodic curiosity through reachability. arXiv preprint arXiv:1810.02274 (2018).
[26] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. 2017. Mastering the game of go without human knowledge. nature 550, 7676 (2017), 354-359.
[27] Richard S Sutton and Andrew G Barto. 2018. Reinforcement learning: An introduction. MIT press.
[28] Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAI Xi Chen, Yan Duan, John Schulman, Filip DeTurck, and Pieter Abbeel. 2017. # exploration: A study of count-based exploration for deep reinforcement learning. Advances in neural information processing systems 30 (2017).
[29] Rosalia Tufano, Simone Scalabrino, Luca Pascarella, Emad Aghajani, Rocco Oliveto, and Gabriele Bavota. 2022. Using Reinforcement Learning for Load Testing of Video Games. arXiv preprint arXiv:2201.06865 (2022).
[30] Oriol Vinyals, Timo Ewalds, Sergey Bartunov, Petko Georgiev, Alexander Sasha Vezhnevets, Michelle Yeo, Alireza Makhzani, Heinrich Küttler, John Agapiou, Julian Schrittwieser, et al. 2017. Starcraft ii: A new challenge for reinforcement learning. arXiv preprint arXiv:1708.04782 (2017).
[31] Christopher JCH Watkins and Peter Dayan. 1992. Q-learning. Machine learning 8, 3 (1992), 279-292.
[32] Yan Zheng, Xiaofei Xie, Ting Su, Lei Ma, Jianye Hao, Zhaopeng Meng, Yang Liu, Ruimin Shen, Yingfeng Chen, and Changjie Fan. 2019. Wuji: Automatic online combat game testing using evolutionary deep reinforcement learning. In 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 772-784.
[2] Thomas H Apperley. 2006. Genre and game studies: Toward critical approach to video game genres. Simulation and Gaming 37, 1 (2006), 6-23.
[3] Sinan Ariyurek, Aysu Betin-Can, and Elif Surer. 2019. Automated video game testing using synthetic and human-like agents. IEEE Transactions on Games (2019).
[4] Arthur Aubret, Laetitia Matignon, and Salima Hassas. 2019. A survey on intrinsic motivation in reinforcement learning. arXiv preprint arXiv:1908.06976 (2019).
[5] Joakim Bergdahl, Camilo Gordillo, Konrad Tollmar, and Linus Gisslén. 2020. Augmenting automated game testing with deep reinforcement learning. In 2020 IEEE Conference on Games (CoG). IEEE, 600-603.
[6] Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemysław Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. 2019. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06680 (2019).
[7] Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A Efros. 2018. Large-scale study of curiosity-driven learning. arXiv preprint arXiv:1808.04355 (2018).
[8] Avinash K Dixit, John JF Sherrerd, et al. 1990. Optimization in economic theory. Oxford University Press on Demand.
[9] Raihana Ferdous, Fitsum Meshesha Kifetew, Davide Prandi, I. S. W. B. Prasetya, Samira Shirzadehhajimahmood, and Angelo Susi. 2021. Search-Based Automated Play Testing of Computer Games: A Model-Based Approach. In Search-Based Software Engineering - 13th International Symposium, SSBSE 2021, Bari, Italy, October 11-12, 2021, Proceedings (Lecture Notes in Computer Science, Vol. 12914), Una-May O'Reilly and Xavier Devroey (Eds.). Springer, 56-71. https://doi.org/ 10.1007/978-3-030-88106-1_5
[10] Camilo Gordillo, Joakim Bergdahl, Konrad Tollmar, and Linus Gisslén. 2021. Improving Playtesting Coverage via Curiosity Driven Reinforcement Learning Agents. arXiv preprint arXiv:2103.13798 (2021).
[11] Oliver Groth, Markus Wulfmeier, Giulia Vezzani, Vibhavari Dasagi, Tim Hertweck, Roland Hafner, Nicolas Heess, and Martin Riedmiller. 2021. Is curiosity all you need? on the utility of emergent behaviours from curious exploration. arXiv preprint arXiv:2109.08603 (2021).
[12] Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. 2016. Vime: Variational information maximizing exploration. Advances in neural information processing systems 29 (2016).
[13] Paul Jaccard. 1912. The distribution of the flora in the alpine zone. 1. New phytologist 11, 2 (1912), 37-50.
[14] Guillaume Lample and Devendra Singh Chaplot. 2017. Playing FPS games with deep reinforcement learning. In Thirty-First AAAI Conference on Artificial Intelligence.
[15] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. 2013. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 (2013).
[16] Newzoo. 2021. Global games market report. https://newzoo.com/insights/ articles/the-games-market-in-2021-the-year-in-numbers-esports-cloudgaming/
[17] Georg Ostrovski, Marc G Bellemare, Aäron Oord, and Rémi Munos. 2017. Countbased exploration with neural density models. In International conference on machine learning. PMLR, 2721-2730.
[18] Minxue Pan, An Huang, Guoxin Wang, Tian Zhang, and Xuandong Li. 2020. Reinforcement learning based curiosity-driven testing of android applications. In Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis. 153-164.
[19] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. 2017. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning. PMLR, 2778-2787.
[20] Alexandre Péré, Sébastien Forestier, Olivier Sigaud, and Pierre-Yves Oudeyer. 2018. Unsupervised learning of goal spaces for intrinsically motivated goal exploration. arXiv preprint arXiv:1803.00781 (2018).
[21] Cristiano Politowski, Fabio Petrillo, and Yann-Gaël Guéhéneuc. 2021. A survey of video game testing. In 2021 IEEE/ACM International Conference on Automation of Software Test (AST). IEEE, 90-99.
[22] Rui Prada, I. S. W. B. Prasetya, Fitsum Meshesha Kifetew, Frank Dignum, Tanja E. J. Vos, Jason Lander, Jean-Yves Donnart, Alexandre Kazmierowski, Joseph Davidson, and Pedro M. Fernandes. 2020. Agent-based Testing of Extended Reality Systems. In 13th IEEE International Conference on Software Testing, Validation and Verification, ICST 2020, Porto, Portugal, October 24-28, 2020 . IEEE, 414-417. https://doi.org/10.1109/ICST46399.2020.00051
[23] ISWB Prasetya, Mehdi Dastani, Rui Prada, Tanja EJ Vos, Frank Dignum, and Fitsum Kifetew. 2020. Aplib: Tactical Agents for Testing Computer Games. In International Workshop on Engineering Multi-Agent Systems (EMAS). Springer, 21-41.
[24] I. S. W. B. Prasetya, Mehdi Dastani, Rui Prada, Tanja E. J. Vos, Frank Dignum, and Fitsum Meshesha Kifetew. 2020. Aplib: Tactical Agents for Testing Computer Games. In Engineering Multi-Agent Systems - 8th International Workshop, EMAS 2020, Auckland, New Zealand, May 8-9, 2020, Revised Selected Papers (Lecture Notes in Computer Science, Vol. 12589), Cristina Baroglio, Jomi Fred Hübner, and Michael Winikof (Eds.). Springer, 21-41. https://doi.org/10.1007/978-3-030-66534-0_2
[25] Nikolay Savinov, Anton Raichuk, Raphaël Marinier, Damien Vincent, Marc Pollefeys, Timothy Lillicrap, and Sylvain Gelly. 2018. Episodic curiosity through reachability. arXiv preprint arXiv:1810.02274 (2018).
[26] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. 2017. Mastering the game of go without human knowledge. nature 550, 7676 (2017), 354-359.
[27] Richard S Sutton and Andrew G Barto. 2018. Reinforcement learning: An introduction. MIT press.
[28] Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAI Xi Chen, Yan Duan, John Schulman, Filip DeTurck, and Pieter Abbeel. 2017. # exploration: A study of count-based exploration for deep reinforcement learning. Advances in neural information processing systems 30 (2017).
[29] Rosalia Tufano, Simone Scalabrino, Luca Pascarella, Emad Aghajani, Rocco Oliveto, and Gabriele Bavota. 2022. Using Reinforcement Learning for Load Testing of Video Games. arXiv preprint arXiv:2201.06865 (2022).
[30] Oriol Vinyals, Timo Ewalds, Sergey Bartunov, Petko Georgiev, Alexander Sasha Vezhnevets, Michelle Yeo, Alireza Makhzani, Heinrich Küttler, John Agapiou, Julian Schrittwieser, et al. 2017. Starcraft ii: A new challenge for reinforcement learning. arXiv preprint arXiv:1708.04782 (2017).
[31] Christopher JCH Watkins and Peter Dayan. 1992. Q-learning. Machine learning 8, 3 (1992), 279-292.
[32] Yan Zheng, Xiaofei Xie, Ting Su, Lei Ma, Jianye Hao, Zhaopeng Meng, Yang Liu, Ruimin Shen, Yingfeng Chen, and Changjie Fan. 2019. Wuji: Automatic online combat game testing using evolutionary deep reinforcement learning. In 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 772-784.
Metrics
Back to previous page
BibTeX entry
@inproceedings{oai:iris.cnr.it:20.500.14243/521571,
title = {Towards agent-based testing of 3D games using reinforcement learning},
author = {Ferdous R. and Kifetew F. and Prandi D. and Susi A.},
publisher = {Association for Computing Machinery},
doi = {10.1145/3551349.3560507},
year = {2022}
}CNR authorsLaboratories
CNR IRIS
Bibliographic record
Deposited version
Deposited version
dl.acm.org
Publications Open Repository TOrino
Re-Unir
Publications Open Repository TOrino
Software Heritage
Software Heritage
Software Heritage
Knowledge-Based Systems
Knowledge-Based Systems
GitHub
GitHub
GitHub
GitHub
GitHub
GitHub
Hal-Diderot
Hal
Knowledge-Based Systems
link.springer.com
link.springer.com
rd.springer.com
Recolector de Ciencia Abierta, RECOLECTA
www.jenjenson.com
Knowledge-Based Systems
