2025
Journal article
Open Access
ARTEMIS: animal recognition through enhanced multimodal integration system
Fazzari E., Romano D., Falchi F., Stefanini C.
Multi-modal deep learning Genetic algorithm Contrastive learning Ensemble Reinforcement learning Animal action recognition
This paper introduces Animal Recognition Through Enhanced Multimodal Integration System (ARTEMIS), a transformer-based framework designed for multilabel animal action recognition by fusing video, image, and textual modalities. ARTEMIS utilizes state-of-the-art captioning and language models, such as BLIP2 and Llama 3, to generate textual descriptions from video frames, which are input to the model, significantly enhancing its performance unlikely previous results that do not consider this modality. Through comprehensive ablation studies, we explore the contribution of various model components and propose optimization strategies, including genetic algorithms and reinforcement learning, to dynamically adjust ensemble weights. Our feature alignment techniques-using contrastive and cosine similarity losses-further improve multimodal integration. Evaluations on the Animal Kingdom dataset, which includes 30,100 clips across 140 action classes, demonstrate that ARTEMIS achieves a new state-of-the-art mAP of 79.82, outperforming existing methods. The combination of multimodal fusion and ensemble strategies makes ARTEMIS a robust solution for complex animal action recognition tasks. The code of our fusion method is available at https://github.com/edofazza/ARTEMIS.
Source: INTERNATIONAL JOURNAL OF MACHINE LEARNING AND CYBERNETICS
Heyes, C, Dickinson, A. The intentionality of animal action. Mind Lang. 1990; 5 (1): 87-103
Rao, Y, Jiang, M, Wang, W, Zhang, W, Wang, R. On-farm welfare monitoring system for goats based on internet of things and machine learning. Int J Distrib Sens Netw. 2020; 16 (7): 786-985
Kostarev, S, Sereda, T, Tatarnikova, N. Building a model for recognition of morphostructure pathologies in animal tissues. J Phys Conf Ser. 2020; 1515
Thenmozhi, M, Saravanan, M, Kumar, KPM, Suseela, S, Deepan, S. Improving the prediction rate of unusual behaviors of animal in a poultry using deep learning technique. Soft Comput. 2020; 24: 14491-14502
Jiang, M, Rao, Y, Zhang, J, Shen, Y. Automatic behavior recognition of group-housed goats using deep learning. Comput Electron Agric. 2020; 177
Feng, L, Zhao, Y, Sun, Y, Zhao, W, Tang, J. Action recognition using a spatial-temporal network for wild felines. Animals. 2021; 11 (2): 485
Mathis, A, Mamidanna, P, Cury, KM, Abe, T, Murthy, VN, Mathis, MW, Bethge, M. DeepLabCut: markerless pose estimation of user-defined body parts with deep learning. Nat Neurosci. 2018; 21 (9): 1281-1289
Segalin, C, Williams, J, Karigo, T, Hui, M, Zelikowsky, M, Sun, JJ, Perona, P, Anderson, DJ, Kennedy, A. The mouse action recognition system (MARS) software pipeline for automated analysis of social behaviors in mice. Elife. 2021; 10: 63720
Labuguen, R, Matsumoto, J, Negrete, SB, Nishimaru, H, Nishijo, H, Takada, M, Go, Y, Inoue, K-I, Shibata, T. Macaquepose: a novel “in the wild” macaque monkey pose dataset for markerless motion capture. Front Behav Neurosci. 2021; 14
10.Yu H, Xu Y, Zhang J, Zhao W, Guan Z, Tao D (2021) Ap-10k. A benchmark for animal pose estimation in the wild. arXiv preprint arXiv:2108.12617
11.Ng XL, Ong KE, Zheng Q, Ni Y, Yeo SY, Liu J (2022) Animal kingdom: a large and diverse dataset for animal behavior understanding. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 19023–19034
Terrasson, G, Llaria, A, Marra, A, Voaden, S. Accelerometer based solution for precision livestock farming: geolocation enhancement and animal activity identification. IOP Conf Ser Mater Sci Eng. 2016; 138
13.O’Shea K (2015) An introduction to convolutional neural networks. arXiv preprint arXiv:1511.08458
14.Mondal A, Nag S, Prada JM, Zhu X, Dutta A (2023) Actor-agnostic multi-label action recognition with multi-modal query. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 784–794
15.Soares S, Antunes C, Araújo R (2012) A genetic algorithm for designing neural network ensembles. In: Proceedings of the 14th annual conference on genetic and evolutionary computation, pp 681–688
16.Sivanandam S, Deepa S, Sivanandam S, Deepa S (2008) Genetic algorithm optimization problems. In: Introduction to genetic algorithms, pp 165–209
Mou, L, Saha, S, Hua, Y, Bovolo, F, Bruzzone, L, Zhu, XX. Deep reinforcement learning for band selection in hyperspectral image classification. IEEE Trans Geosci Remote Sens. 2021; 60: 1-14
Fazzari, E, Loughlin, HA, Stoughton, C. Controlling optical-cavity locking using reinforcement learning. Mach Learn Sci Technol. 2024; 5 (3)
Mnih, V, Kavukcuoglu, K, Silver, D, Rusu, AA, Veness, J, Bellemare, MG, Graves, A, Riedmiller, M, Fidjeland, AK, Ostrovski, G. Human-level control through deep reinforcement learning. Nature. 2015; 518 (7540): 529-533
Sun, Z, Ke, Q, Rahmani, H, Bennamoun, M, Wang, G, Liu, J. Human action recognition from various data modalities: a review. IEEE Trans Pattern Anal Mach Intell. 2022; 45 (3): 3200-3225
21.Fazzari E, Romano D, Falchi F, Stefanini C (2024) Animal behavior analysis methods using deep learning: a survey. arXiv:2405.14002
Pereira, TD, Tabris, N, Matsliah, A, Turner, DM, Li, J, Ravindranath, S, Papadoyannis, ES, Normand, E, Deutsch, DS, Wang, ZY. SLEAP: a deep learning system for multi-animal pose tracking. Nat Methods. 2022; 19 (4): 486-495
Bohnslav, JP, Wimalasena, NK, Clausing, KJ, Dai, YY, Yarmolinsky, DA, Cruz, T, Kashlan, AD, Chiappe, ME, Orefice, LL, Woolf, CJ. DeepEthogram, a machine learning pipeline for supervised behavior classification from raw pixels. Elife. 2021; 10: 63377
Odo, A, Muns, R, Boyle, L, Kyriazakis, I. Video analysis using deep learning for automated quantification of ear biting in pigs. IEEE Access. 2023; 11: 59744-59757
25.Blau A, Gebhardt C, Bendesky A, Paninski L, Wu A (2022) Semimultipose: a semi-supervised multi-animal pose estimation framework. arXiv preprint arXiv:2204.07072
Fazzari, E, Carrara, F, Falchi, F, Stefanini, C, Romano, D. Using AI to decode the behavioral responses of an insect to chemical stimuli: towards machine-animal computational technologies. Int J Mach Learn Cybern. 2024; 15 (5): 1985-1994
27.Fujimori S, Ishikawa T, Watanabe H (2020) Animal behavior classification using deeplabcut. In: 2020 IEEE 9th global conference on consumer electronics (GCCE). IEEE, pp 254–257
Arablouei, R, Wang, L, Currie, L, Yates, J, Alvarenga, FA, Bishop-Hurley, GJ. Animal behavior classification via deep learning on embedded systems. Comput Electron Agric. 2023; 207
Arablouei, R, Wang, Z, Bishop-Hurley, GJ, Liu, J. Multimodal sensor data fusion for in-situ classification of animal behavior using accelerometry and GNSS data. Smart Agric Technol. 2023; 4
30.Zhao L, Stephany RG, Han Y, Ahmmed P, Huang T-P, Bozkurt A, Jia Y (2024) A wireless multimodal physiological monitoring ASIC for animal health monitoring injectable devices. IEEE Trans Biomed Circuits Syst
Anderson, G, Johnson, A, Arguelles-Ramos, M, Ali, A. Impact of body-worn sensors on broiler chicken behavior and agonistic interactions. J Appl Anim Welf Sci. 2023; 25: 1-10
Zhao, D, Chang, Z, Guo, S. A multimodal fusion approach for image captioning. Neurocomputing. 2019; 329: 476-485
33.Gong X, Mohan S, Dhingra N, Bazin J-C, Li Y, Wang Z, Ranjan R (2023) MMG-EGO4D: multimodal generalization in egocentric action recognition. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 6481–6491
34.Xia W, Yang Y, Xue J-H, Wu B (2021) Tedigan: text-guided diverse face image generation and manipulation. In: Proceedings of the IEEE/CVF conference on computer vi sion and pattern recognition, pp 2256–2265
Fazzari, E, Romano, D, Falchi, F, Stefanini, C. Selective state models are what you need for animal action recognition. Ecol Inf. 2024; 25
36.Duporge I, Kholiavchenko M, Harel R, Wolf S, Rubenstein D, Crofoot M, Berger-Wolf T, Lee S, Barreau J, Kline J et al (2024) BaboonLand dataset: tracking primates in the wild and automating behaviour recognition from drone videos. arXiv preprint arXiv:2405.17698
37.Chen J, Hu M, Coker DJ, Berumen ML, Costelloe B, Beery S, Rohrbach A, Elhoseiny M (2023) Mammalnet: a large-scale video benchmark for mammal recognition and behavior understanding. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 13052–13061
38.Bertasius G, Wang H, Torresani L (2021) Is space-time attention all you need for video understanding? In: ICML, vol 2, p 4
39.Radford A, Kim JW, Hallacy C, Ramesh A, Goh G, Agarwal S, Sast ry G, Askell A, Mishkin P, Clark J (2021) Learning transferable visual models from natural language supervision. In: International conference on machine learning. PMLR, pp 8748–8763
40.Vaswani A (2017) Attention is all you need. Adv Neural Inf Process Syst
41.Li J, Li D, Savarese S, Hoi S (2023) Blip-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In: International conference on machine learning. PMLR, pp 19730–19742
42.Dubey A, Jauhri A, Pandey A, Kadian A, Al-Dahle A, Letman A, Mathur A, Schelten A, Yang A, Fan A et al (2024) The llama 3 herd of models. arXiv preprint arXiv:2407.21783
Akbari, H, Yuan, L, Qian, R, Chuang, W-H, Chang, S-F, Cui, Y, Gong, B. VATT: transformers for multimodal self-supervised learning from raw video, audio and text. Adv Neural Inf Process Syst. 2021; 34: 24206-24221
44.Kipf T, Pol E, Welling M (2019) Contrastive learning of structured world models. In: International conference on learning representations
45.Wang H, Wang Y, Zhou Z, Ji X, Gong D, Zhou J, Li Z, Liu W (2018) COSFACE: large margin cosine loss for deep face recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5265–5274
Sun, Q-S, Zeng, S-G, Liu, Y, Heng, P-A, Xia, D-S. A new method of feature fusion and its application in image recognition. Pattern Recognit. 2005; 38 (12): 2437-2448
Zhiyuan, L. Investigating an ensemble classifier based on multi-objective genetic algorithm for machine learning applications. Int J Adv Comput Sci Appl. 2024; 15 (5): 756
Rudnick, EM, Patel, JH, Greenstein, GS, Niermann, TM. A genetic algorithm framework for test generation. IEEE Trans Comput Aided Des Integr Circuits Syst. 1997; 16 (9): 1034-1044
49.Takahashi M, Kita H (2001) A crossover operator using independent component analysis for real-coded genetic algorithms. In: Proceedings of the 2001 congress on evolutionary computation (IEEE Cat. No. 01th8546). IEEE, vol 1, pp 643–649
50.Schulman J, Wolski F, Dhariwal P, Radford A, Klimov O (2017) Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347
Raffin, A, Hill, A, Gleave, A, Kanervisto, A, Ernestus, M, Dormann, N. Stable-baselines3: reliable reinforcement learning implementations. J Mach Learn Res. 2021; 22 (268): 1-8
52.Kingma DP (2014) ADAM: a method for stochastic optimization. arXiv preprint arXiv:1412.6980
53.Loshchilov I, Hutter F (2017) SGDR: stochastic gradient descent with warm restarts. In: International conference on learning representations. https://openreview.net/forum?id=Skq89Scxx
54.Student (1908) The probable error of a mean. Biometrika 1–25
55.Wilcoxon F (1992) Individual comparisons by ranking methods. In: Breakthroughs in statistics: methodology and distribution. Springer, New York, pp 196–202
1. Heyes C, Dickinson A (1990) The intentionality of animal action. Mind Lang 5(1):87-103
2. Rao Y, Jiang M, Wang W, Zhang W, Wang R (2020) On-farm welfare monitoring system for goats based on internet of things and machine learning. Int J Distrib Sens Netw 16(7):786-985
3. Kostarev S, Sereda T, Tatarnikova N (2020) Building a model for recognition of morphostructure pathologies in animal tissues. J Phys Conf Ser 1515:052005
4. Thenmozhi M, Saravanan M, Kumar KPM, Suseela S, Deepan S (2020) Improving the prediction rate of unusual behaviors of animal in a poultry using deep learning technique. Soft Comput 24:14491-14502
5. Jiang M, Rao Y, Zhang J, Shen Y (2020) Automatic behavior recognition of group-housed goats using deep learning. Comput Electron Agric 177:105706
6. Feng L, Zhao Y, Sun Y, Zhao W, Tang J (2021) Action recognition using a spatial-temporal network for wild felines. Animals 11(2):485
7. Mathis A, Mamidanna P, Cury KM, Abe T, Murthy VN, Mathis MW, Bethge M (2018) DeepLabCut: markerless pose estimation of user-defined body parts with deep learning. Nat Neurosci 21(9):1281-1289
8. Segalin C, Williams J, Karigo T, Hui M, Zelikowsky M, Sun JJ, Perona P, Anderson DJ, Kennedy A (2021) The mouse action recognition system (MARS) software pipeline for automated analysis of social behaviors in mice. Elife 10:63720
9. Labuguen R, Matsumoto J, Negrete SB, Nishimaru H, Nishijo H, Takada M, Go Y, Inoue K-I, Shibata T (2021) Macaquepose: a novel “in the wild'' macaque monkey pose dataset for markerless motion capture. Front Behav Neurosci 14:581154
10. Yu H, Xu Y, Zhang J, Zhao W, Guan Z, Tao D (2021) Ap-10k. A benchmark for animal pose estimation in the wild. arXiv preprint arXiv:2108.12617
11. Ng XL, Ong KE, Zheng Q, Ni Y, Yeo SY, Liu J (2022) Animal kingdom: a large and diverse dataset for animal behavior understanding. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 19023-19034
12. Terrasson G, Llaria A, Marra A, Voaden S (2016) Accelerometer based solution for precision livestock farming: geolocation enhancement and animal activity identification. IOP Conf Ser Mater Sci Eng 138:012004
13. O'Shea K (2015) An introduction to convolutional neural networks. arXiv preprint arXiv:1511.08458
14. Mondal A, Nag S, Prada JM, Zhu X, Dutta A (2023) Actor-agnostic multi-label action recognition with multi-modal query. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 784-794
15. Soares S, Antunes C, Araújo R (2012) A genetic algorithm for designing neural network ensembles. In: Proceedings of the 14th annual conference on genetic and evolutionary computation, pp 681-688
16. Sivanandam S, Deepa S, Sivanandam S, Deepa S (2008) Genetic algorithm optimization problems. In: Introduction to genetic algorithms, pp 165-209
17. Mou L, Saha S, Hua Y, Bovolo F, Bruzzone L, Zhu XX (2021) Deep reinforcement learning for band selection in hyperspectral image classification. IEEE Trans Geosci Remote Sens 60:1-14
18. Fazzari E, Loughlin HA, Stoughton C (2024) Controlling optical-cavity locking using reinforcement learning. Mach Learn Sci Technol 5(3):035027
19. Mnih V, Kavukcuoglu K, Silver D, Rusu AA, Veness J, Bellemare MG, Graves A, Riedmiller M, Fidjeland AK, Ostrovski G (2015) Human-level control through deep reinforcement learning. Nature 518(7540):529-533
20. Sun Z, Ke Q, Rahmani H, Bennamoun M, Wang G, Liu J (2022) Human action recognition from various data modalities: a review. IEEE Trans Pattern Anal Mach Intell 45(3):3200-3225
21. Fazzari E, Romano D, Falchi F, Stefanini C (2024) Animal behavior analysis methods using deep learning: a survey. arXiv:2405. 14002
22. Pereira TD, Tabris N, Matsliah A, Turner DM, Li J, Ravindranath S, Papadoyannis ES, Normand E, Deutsch DS, Wang ZY (2022) SLEAP: a deep learning system for multi-animal pose tracking. Nat Methods 19(4):486-495
23. Bohnslav JP, Wimalasena NK, Clausing KJ, Dai YY, Yarmolinsky DA, Cruz T, Kashlan AD, Chiappe ME, Orefice LL, Woolf CJ (2021) DeepEthogram, a machine learning pipeline for supervised behavior classification from raw pixels. Elife 10:63377
24. Odo A, Muns R, Boyle L, Kyriazakis I (2023) Video analysis using deep learning for automated quantification of ear biting in pigs. IEEE Access 11:59744-59757
25. Blau A, Gebhardt C, Bendesky A, Paninski L, Wu A (2022) Semimultipose: a semi-supervised multi-animal pose estimation framework. arXiv preprint arXiv:2204.07072
26. Fazzari E, Carrara F, Falchi F, Stefanini C, Romano D (2024) Using AI to decode the behavioral responses of an insect to chemical stimuli: towards machine-animal computational technologies. Int J Mach Learn Cybern 15(5):1985-1994
27. Fujimori S, Ishikawa T, Watanabe H (2020) Animal behavior classification using deeplabcut. In: 2020 IEEE 9th global conference on consumer electronics (GCCE). IEEE, pp 254-257
28. Arablouei R, Wang L, Currie L, Yates J, Alvarenga FA, BishopHurley GJ (2023) Animal behavior classicfiation via deep learning on embedded systems. Comput Electron Agric 207:107707
29. Arablouei R, Wang Z, Bishop-Hurley GJ, Liu J (2023) Multimodal sensor data fusion for in-situ classicfiation of animal behav - ior using accelerometry and GNSS data. Smart Agric Technol 4:100163
30. Zhao L, Stephany RG, Han Y, Ahmmed P, Huang T-P, Bozkurt A, Jia Y (2024) A wireless multimodal physiological monitoring ASIC for animal health monitoring injectable devices. IEEE Trans Biomed Circuits Syst
31. Anderson G, Johnson A, Arguelles-Ramos M, Ali A (2023) Impact of body-worn sensors on broiler chicken behavior and agonistic interactions. J Appl Anim Welf Sci 25:1-10
32. Zhao D, Chang Z, Guo S (2019) A multimodal fusion approach for image captioning. Neurocomputing 329:476-485
33. Gong X, Mohan S, Dhingra N, Bazin J-C, Li Y, Wang Z, Ranjan R (2023) MMG-EGO4D: multimodal generalization in egocentric action recognition. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 6481-6491
34. Xia W, Yang Y, Xue J-H, Wu B (2021) Tedigan: text-guided diverse face image generation and manipulation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 2256-2265
35. Fazzari E, Romano D, Falchi F, Stefanini C (2024) Selective state models are what you need for animal action recognition. Ecol Inf 25:102955
36. Duporge I, Kholiavchenko M, Harel R, Wolf S, Rubenstein D, Crofoot M, Berger-Wolf T, Lee S, Barreau J, Kline J et al (2024) BaboonLand dataset: tracking primates in the wild and automating behaviour recognition from drone videos. arXiv preprint arXiv: 2405.17698
37. Chen J, Hu M, Coker DJ, Berumen ML, Costelloe B, Beery S, Rohrbach A, Elhoseiny M (2023) Mammalnet: a large-scale video benchmark for mammal recognition and behavior understanding. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 13052-13061
38. Bertasius G, Wang H, Torresani L (2021) Is space-time attention all you need for video understanding? In: ICML, vol 2, p 4
39. Radford A, Kim JW, Hallacy C, Ramesh A, Goh G, Agarwal S, Sastry G, Askell A, Mishkin P, Clark J (2021) Learning transferable visual models from natural language supervision. In: International conference on machine learning. PMLR, pp 8748-8763
40. Vaswani A (2017) Attention is all you need. Adv Neural Inf Process Syst
41. Li J, Li D, Savarese S, Hoi S (2023) Blip-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In: International conference on machine learning. PMLR, pp 19730-19742
42. Dubey A, Jauhri A, Pandey A, Kadian A, Al-Dahle A, Letman A, Mathur A, Schelten A, Yang A, Fan A et al (2024) The llama 3 herd of models. arXiv preprint arXiv:2407.21783
43. Akbari H, Yuan L, Qian R, Chuang W-H, Chang S-F, Cui Y, Gong B (2021) VATT: transformers for multimodal self-supervised learning from raw video, audio and text. Adv Neural Inf Process Syst 34:24206-24221
44. Kipf T, Pol E, Welling M (2019) Contrastive learning of structured world models. In: International conference on learning representations
45. Wang H, Wang Y, Zhou Z, Ji X, Gong D, Zhou J, Li Z, Liu W (2018) COSFACE: large margin cosine loss for deep face recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5265-5274
46. Sun Q-S, Zeng S-G, Liu Y, Heng P-A, Xia D-S (2005) A new method of feature fusion and its application in image recognition. Pattern Recognit 38(12):2437-2448
47. Zhiyuan L (2024) Investigating an ensemble classifier based on multi-objective genetic algorithm for machine learning applications. Int J Adv Comput Sci Appl 15(5):756
48. Rudnick EM, Patel JH, Greenstein GS, Niermann TM (1997) A genetic algorithm framework for test generation. IEEE Trans Comput Aided Des Integr Circuits Syst 16(9):1034-1044
49. Takahashi M, Kita H (2001) A crossover operator using independent component analysis for real-coded genetic algorithms. In: Proceedings of the 2001 congress on evolutionary computation (IEEE Cat. No. 01th8546). IEEE, vol 1, pp 643-649
50. Schulman J, Wolski F, Dhariwal P, Radford A, Klimov O (2017) Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06347
51. Rafin A, Hill A, Gleave A, Kanervisto A, Ernestus M, Dormann N (2021) Stable-baselines3: reliable reinforcement learning implementations. J Mach Learn Res 22(268):1-8
52. Kingma DP (2014) ADAM: a method for stochastic optimization. arXiv preprint arXiv:1412.6980
53. Loshchilov I, Hutter F (2017) SGDR: stochastic gradient descent with warm restarts. In: International conference on learning representations. https://openreview.net/forum?id=Skq89Scxx
54. Student (1908) The probable error of a mean. Biometrika 1-25
55. Wilcoxon F (1992) Individual comparisons by ranking methods. In: Breakthroughs in statistics: methodology and distribution. Springer, New York, pp 196-202
Rao, Y, Jiang, M, Wang, W, Zhang, W, Wang, R. On-farm welfare monitoring system for goats based on internet of things and machine learning. Int J Distrib Sens Netw. 2020; 16 (7): 786-985
Kostarev, S, Sereda, T, Tatarnikova, N. Building a model for recognition of morphostructure pathologies in animal tissues. J Phys Conf Ser. 2020; 1515
Thenmozhi, M, Saravanan, M, Kumar, KPM, Suseela, S, Deepan, S. Improving the prediction rate of unusual behaviors of animal in a poultry using deep learning technique. Soft Comput. 2020; 24: 14491-14502
Jiang, M, Rao, Y, Zhang, J, Shen, Y. Automatic behavior recognition of group-housed goats using deep learning. Comput Electron Agric. 2020; 177
Feng, L, Zhao, Y, Sun, Y, Zhao, W, Tang, J. Action recognition using a spatial-temporal network for wild felines. Animals. 2021; 11 (2): 485
Mathis, A, Mamidanna, P, Cury, KM, Abe, T, Murthy, VN, Mathis, MW, Bethge, M. DeepLabCut: markerless pose estimation of user-defined body parts with deep learning. Nat Neurosci. 2018; 21 (9): 1281-1289
Segalin, C, Williams, J, Karigo, T, Hui, M, Zelikowsky, M, Sun, JJ, Perona, P, Anderson, DJ, Kennedy, A. The mouse action recognition system (MARS) software pipeline for automated analysis of social behaviors in mice. Elife. 2021; 10: 63720
Labuguen, R, Matsumoto, J, Negrete, SB, Nishimaru, H, Nishijo, H, Takada, M, Go, Y, Inoue, K-I, Shibata, T. Macaquepose: a novel “in the wild” macaque monkey pose dataset for markerless motion capture. Front Behav Neurosci. 2021; 14
10.Yu H, Xu Y, Zhang J, Zhao W, Guan Z, Tao D (2021) Ap-10k. A benchmark for animal pose estimation in the wild. arXiv preprint arXiv:2108.12617
11.Ng XL, Ong KE, Zheng Q, Ni Y, Yeo SY, Liu J (2022) Animal kingdom: a large and diverse dataset for animal behavior understanding. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 19023–19034
Terrasson, G, Llaria, A, Marra, A, Voaden, S. Accelerometer based solution for precision livestock farming: geolocation enhancement and animal activity identification. IOP Conf Ser Mater Sci Eng. 2016; 138
13.O’Shea K (2015) An introduction to convolutional neural networks. arXiv preprint arXiv:1511.08458
14.Mondal A, Nag S, Prada JM, Zhu X, Dutta A (2023) Actor-agnostic multi-label action recognition with multi-modal query. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 784–794
15.Soares S, Antunes C, Araújo R (2012) A genetic algorithm for designing neural network ensembles. In: Proceedings of the 14th annual conference on genetic and evolutionary computation, pp 681–688
16.Sivanandam S, Deepa S, Sivanandam S, Deepa S (2008) Genetic algorithm optimization problems. In: Introduction to genetic algorithms, pp 165–209
Mou, L, Saha, S, Hua, Y, Bovolo, F, Bruzzone, L, Zhu, XX. Deep reinforcement learning for band selection in hyperspectral image classification. IEEE Trans Geosci Remote Sens. 2021; 60: 1-14
Fazzari, E, Loughlin, HA, Stoughton, C. Controlling optical-cavity locking using reinforcement learning. Mach Learn Sci Technol. 2024; 5 (3)
Mnih, V, Kavukcuoglu, K, Silver, D, Rusu, AA, Veness, J, Bellemare, MG, Graves, A, Riedmiller, M, Fidjeland, AK, Ostrovski, G. Human-level control through deep reinforcement learning. Nature. 2015; 518 (7540): 529-533
Sun, Z, Ke, Q, Rahmani, H, Bennamoun, M, Wang, G, Liu, J. Human action recognition from various data modalities: a review. IEEE Trans Pattern Anal Mach Intell. 2022; 45 (3): 3200-3225
21.Fazzari E, Romano D, Falchi F, Stefanini C (2024) Animal behavior analysis methods using deep learning: a survey. arXiv:2405.14002
Pereira, TD, Tabris, N, Matsliah, A, Turner, DM, Li, J, Ravindranath, S, Papadoyannis, ES, Normand, E, Deutsch, DS, Wang, ZY. SLEAP: a deep learning system for multi-animal pose tracking. Nat Methods. 2022; 19 (4): 486-495
Bohnslav, JP, Wimalasena, NK, Clausing, KJ, Dai, YY, Yarmolinsky, DA, Cruz, T, Kashlan, AD, Chiappe, ME, Orefice, LL, Woolf, CJ. DeepEthogram, a machine learning pipeline for supervised behavior classification from raw pixels. Elife. 2021; 10: 63377
Odo, A, Muns, R, Boyle, L, Kyriazakis, I. Video analysis using deep learning for automated quantification of ear biting in pigs. IEEE Access. 2023; 11: 59744-59757
25.Blau A, Gebhardt C, Bendesky A, Paninski L, Wu A (2022) Semimultipose: a semi-supervised multi-animal pose estimation framework. arXiv preprint arXiv:2204.07072
Fazzari, E, Carrara, F, Falchi, F, Stefanini, C, Romano, D. Using AI to decode the behavioral responses of an insect to chemical stimuli: towards machine-animal computational technologies. Int J Mach Learn Cybern. 2024; 15 (5): 1985-1994
27.Fujimori S, Ishikawa T, Watanabe H (2020) Animal behavior classification using deeplabcut. In: 2020 IEEE 9th global conference on consumer electronics (GCCE). IEEE, pp 254–257
Arablouei, R, Wang, L, Currie, L, Yates, J, Alvarenga, FA, Bishop-Hurley, GJ. Animal behavior classification via deep learning on embedded systems. Comput Electron Agric. 2023; 207
Arablouei, R, Wang, Z, Bishop-Hurley, GJ, Liu, J. Multimodal sensor data fusion for in-situ classification of animal behavior using accelerometry and GNSS data. Smart Agric Technol. 2023; 4
30.Zhao L, Stephany RG, Han Y, Ahmmed P, Huang T-P, Bozkurt A, Jia Y (2024) A wireless multimodal physiological monitoring ASIC for animal health monitoring injectable devices. IEEE Trans Biomed Circuits Syst
Anderson, G, Johnson, A, Arguelles-Ramos, M, Ali, A. Impact of body-worn sensors on broiler chicken behavior and agonistic interactions. J Appl Anim Welf Sci. 2023; 25: 1-10
Zhao, D, Chang, Z, Guo, S. A multimodal fusion approach for image captioning. Neurocomputing. 2019; 329: 476-485
33.Gong X, Mohan S, Dhingra N, Bazin J-C, Li Y, Wang Z, Ranjan R (2023) MMG-EGO4D: multimodal generalization in egocentric action recognition. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 6481–6491
34.Xia W, Yang Y, Xue J-H, Wu B (2021) Tedigan: text-guided diverse face image generation and manipulation. In: Proceedings of the IEEE/CVF conference on computer vi sion and pattern recognition, pp 2256–2265
Fazzari, E, Romano, D, Falchi, F, Stefanini, C. Selective state models are what you need for animal action recognition. Ecol Inf. 2024; 25
36.Duporge I, Kholiavchenko M, Harel R, Wolf S, Rubenstein D, Crofoot M, Berger-Wolf T, Lee S, Barreau J, Kline J et al (2024) BaboonLand dataset: tracking primates in the wild and automating behaviour recognition from drone videos. arXiv preprint arXiv:2405.17698
37.Chen J, Hu M, Coker DJ, Berumen ML, Costelloe B, Beery S, Rohrbach A, Elhoseiny M (2023) Mammalnet: a large-scale video benchmark for mammal recognition and behavior understanding. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 13052–13061
38.Bertasius G, Wang H, Torresani L (2021) Is space-time attention all you need for video understanding? In: ICML, vol 2, p 4
39.Radford A, Kim JW, Hallacy C, Ramesh A, Goh G, Agarwal S, Sast ry G, Askell A, Mishkin P, Clark J (2021) Learning transferable visual models from natural language supervision. In: International conference on machine learning. PMLR, pp 8748–8763
40.Vaswani A (2017) Attention is all you need. Adv Neural Inf Process Syst
41.Li J, Li D, Savarese S, Hoi S (2023) Blip-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In: International conference on machine learning. PMLR, pp 19730–19742
42.Dubey A, Jauhri A, Pandey A, Kadian A, Al-Dahle A, Letman A, Mathur A, Schelten A, Yang A, Fan A et al (2024) The llama 3 herd of models. arXiv preprint arXiv:2407.21783
Akbari, H, Yuan, L, Qian, R, Chuang, W-H, Chang, S-F, Cui, Y, Gong, B. VATT: transformers for multimodal self-supervised learning from raw video, audio and text. Adv Neural Inf Process Syst. 2021; 34: 24206-24221
44.Kipf T, Pol E, Welling M (2019) Contrastive learning of structured world models. In: International conference on learning representations
45.Wang H, Wang Y, Zhou Z, Ji X, Gong D, Zhou J, Li Z, Liu W (2018) COSFACE: large margin cosine loss for deep face recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5265–5274
Sun, Q-S, Zeng, S-G, Liu, Y, Heng, P-A, Xia, D-S. A new method of feature fusion and its application in image recognition. Pattern Recognit. 2005; 38 (12): 2437-2448
Zhiyuan, L. Investigating an ensemble classifier based on multi-objective genetic algorithm for machine learning applications. Int J Adv Comput Sci Appl. 2024; 15 (5): 756
Rudnick, EM, Patel, JH, Greenstein, GS, Niermann, TM. A genetic algorithm framework for test generation. IEEE Trans Comput Aided Des Integr Circuits Syst. 1997; 16 (9): 1034-1044
49.Takahashi M, Kita H (2001) A crossover operator using independent component analysis for real-coded genetic algorithms. In: Proceedings of the 2001 congress on evolutionary computation (IEEE Cat. No. 01th8546). IEEE, vol 1, pp 643–649
50.Schulman J, Wolski F, Dhariwal P, Radford A, Klimov O (2017) Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347
Raffin, A, Hill, A, Gleave, A, Kanervisto, A, Ernestus, M, Dormann, N. Stable-baselines3: reliable reinforcement learning implementations. J Mach Learn Res. 2021; 22 (268): 1-8
52.Kingma DP (2014) ADAM: a method for stochastic optimization. arXiv preprint arXiv:1412.6980
53.Loshchilov I, Hutter F (2017) SGDR: stochastic gradient descent with warm restarts. In: International conference on learning representations. https://openreview.net/forum?id=Skq89Scxx
54.Student (1908) The probable error of a mean. Biometrika 1–25
55.Wilcoxon F (1992) Individual comparisons by ranking methods. In: Breakthroughs in statistics: methodology and distribution. Springer, New York, pp 196–202
1. Heyes C, Dickinson A (1990) The intentionality of animal action. Mind Lang 5(1):87-103
2. Rao Y, Jiang M, Wang W, Zhang W, Wang R (2020) On-farm welfare monitoring system for goats based on internet of things and machine learning. Int J Distrib Sens Netw 16(7):786-985
3. Kostarev S, Sereda T, Tatarnikova N (2020) Building a model for recognition of morphostructure pathologies in animal tissues. J Phys Conf Ser 1515:052005
4. Thenmozhi M, Saravanan M, Kumar KPM, Suseela S, Deepan S (2020) Improving the prediction rate of unusual behaviors of animal in a poultry using deep learning technique. Soft Comput 24:14491-14502
5. Jiang M, Rao Y, Zhang J, Shen Y (2020) Automatic behavior recognition of group-housed goats using deep learning. Comput Electron Agric 177:105706
6. Feng L, Zhao Y, Sun Y, Zhao W, Tang J (2021) Action recognition using a spatial-temporal network for wild felines. Animals 11(2):485
7. Mathis A, Mamidanna P, Cury KM, Abe T, Murthy VN, Mathis MW, Bethge M (2018) DeepLabCut: markerless pose estimation of user-defined body parts with deep learning. Nat Neurosci 21(9):1281-1289
8. Segalin C, Williams J, Karigo T, Hui M, Zelikowsky M, Sun JJ, Perona P, Anderson DJ, Kennedy A (2021) The mouse action recognition system (MARS) software pipeline for automated analysis of social behaviors in mice. Elife 10:63720
9. Labuguen R, Matsumoto J, Negrete SB, Nishimaru H, Nishijo H, Takada M, Go Y, Inoue K-I, Shibata T (2021) Macaquepose: a novel “in the wild'' macaque monkey pose dataset for markerless motion capture. Front Behav Neurosci 14:581154
10. Yu H, Xu Y, Zhang J, Zhao W, Guan Z, Tao D (2021) Ap-10k. A benchmark for animal pose estimation in the wild. arXiv preprint arXiv:2108.12617
11. Ng XL, Ong KE, Zheng Q, Ni Y, Yeo SY, Liu J (2022) Animal kingdom: a large and diverse dataset for animal behavior understanding. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 19023-19034
12. Terrasson G, Llaria A, Marra A, Voaden S (2016) Accelerometer based solution for precision livestock farming: geolocation enhancement and animal activity identification. IOP Conf Ser Mater Sci Eng 138:012004
13. O'Shea K (2015) An introduction to convolutional neural networks. arXiv preprint arXiv:1511.08458
14. Mondal A, Nag S, Prada JM, Zhu X, Dutta A (2023) Actor-agnostic multi-label action recognition with multi-modal query. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 784-794
15. Soares S, Antunes C, Araújo R (2012) A genetic algorithm for designing neural network ensembles. In: Proceedings of the 14th annual conference on genetic and evolutionary computation, pp 681-688
16. Sivanandam S, Deepa S, Sivanandam S, Deepa S (2008) Genetic algorithm optimization problems. In: Introduction to genetic algorithms, pp 165-209
17. Mou L, Saha S, Hua Y, Bovolo F, Bruzzone L, Zhu XX (2021) Deep reinforcement learning for band selection in hyperspectral image classification. IEEE Trans Geosci Remote Sens 60:1-14
18. Fazzari E, Loughlin HA, Stoughton C (2024) Controlling optical-cavity locking using reinforcement learning. Mach Learn Sci Technol 5(3):035027
19. Mnih V, Kavukcuoglu K, Silver D, Rusu AA, Veness J, Bellemare MG, Graves A, Riedmiller M, Fidjeland AK, Ostrovski G (2015) Human-level control through deep reinforcement learning. Nature 518(7540):529-533
20. Sun Z, Ke Q, Rahmani H, Bennamoun M, Wang G, Liu J (2022) Human action recognition from various data modalities: a review. IEEE Trans Pattern Anal Mach Intell 45(3):3200-3225
21. Fazzari E, Romano D, Falchi F, Stefanini C (2024) Animal behavior analysis methods using deep learning: a survey. arXiv:2405. 14002
22. Pereira TD, Tabris N, Matsliah A, Turner DM, Li J, Ravindranath S, Papadoyannis ES, Normand E, Deutsch DS, Wang ZY (2022) SLEAP: a deep learning system for multi-animal pose tracking. Nat Methods 19(4):486-495
23. Bohnslav JP, Wimalasena NK, Clausing KJ, Dai YY, Yarmolinsky DA, Cruz T, Kashlan AD, Chiappe ME, Orefice LL, Woolf CJ (2021) DeepEthogram, a machine learning pipeline for supervised behavior classification from raw pixels. Elife 10:63377
24. Odo A, Muns R, Boyle L, Kyriazakis I (2023) Video analysis using deep learning for automated quantification of ear biting in pigs. IEEE Access 11:59744-59757
25. Blau A, Gebhardt C, Bendesky A, Paninski L, Wu A (2022) Semimultipose: a semi-supervised multi-animal pose estimation framework. arXiv preprint arXiv:2204.07072
26. Fazzari E, Carrara F, Falchi F, Stefanini C, Romano D (2024) Using AI to decode the behavioral responses of an insect to chemical stimuli: towards machine-animal computational technologies. Int J Mach Learn Cybern 15(5):1985-1994
27. Fujimori S, Ishikawa T, Watanabe H (2020) Animal behavior classification using deeplabcut. In: 2020 IEEE 9th global conference on consumer electronics (GCCE). IEEE, pp 254-257
28. Arablouei R, Wang L, Currie L, Yates J, Alvarenga FA, BishopHurley GJ (2023) Animal behavior classicfiation via deep learning on embedded systems. Comput Electron Agric 207:107707
29. Arablouei R, Wang Z, Bishop-Hurley GJ, Liu J (2023) Multimodal sensor data fusion for in-situ classicfiation of animal behav - ior using accelerometry and GNSS data. Smart Agric Technol 4:100163
30. Zhao L, Stephany RG, Han Y, Ahmmed P, Huang T-P, Bozkurt A, Jia Y (2024) A wireless multimodal physiological monitoring ASIC for animal health monitoring injectable devices. IEEE Trans Biomed Circuits Syst
31. Anderson G, Johnson A, Arguelles-Ramos M, Ali A (2023) Impact of body-worn sensors on broiler chicken behavior and agonistic interactions. J Appl Anim Welf Sci 25:1-10
32. Zhao D, Chang Z, Guo S (2019) A multimodal fusion approach for image captioning. Neurocomputing 329:476-485
33. Gong X, Mohan S, Dhingra N, Bazin J-C, Li Y, Wang Z, Ranjan R (2023) MMG-EGO4D: multimodal generalization in egocentric action recognition. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 6481-6491
34. Xia W, Yang Y, Xue J-H, Wu B (2021) Tedigan: text-guided diverse face image generation and manipulation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 2256-2265
35. Fazzari E, Romano D, Falchi F, Stefanini C (2024) Selective state models are what you need for animal action recognition. Ecol Inf 25:102955
36. Duporge I, Kholiavchenko M, Harel R, Wolf S, Rubenstein D, Crofoot M, Berger-Wolf T, Lee S, Barreau J, Kline J et al (2024) BaboonLand dataset: tracking primates in the wild and automating behaviour recognition from drone videos. arXiv preprint arXiv: 2405.17698
37. Chen J, Hu M, Coker DJ, Berumen ML, Costelloe B, Beery S, Rohrbach A, Elhoseiny M (2023) Mammalnet: a large-scale video benchmark for mammal recognition and behavior understanding. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 13052-13061
38. Bertasius G, Wang H, Torresani L (2021) Is space-time attention all you need for video understanding? In: ICML, vol 2, p 4
39. Radford A, Kim JW, Hallacy C, Ramesh A, Goh G, Agarwal S, Sastry G, Askell A, Mishkin P, Clark J (2021) Learning transferable visual models from natural language supervision. In: International conference on machine learning. PMLR, pp 8748-8763
40. Vaswani A (2017) Attention is all you need. Adv Neural Inf Process Syst
41. Li J, Li D, Savarese S, Hoi S (2023) Blip-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In: International conference on machine learning. PMLR, pp 19730-19742
42. Dubey A, Jauhri A, Pandey A, Kadian A, Al-Dahle A, Letman A, Mathur A, Schelten A, Yang A, Fan A et al (2024) The llama 3 herd of models. arXiv preprint arXiv:2407.21783
43. Akbari H, Yuan L, Qian R, Chuang W-H, Chang S-F, Cui Y, Gong B (2021) VATT: transformers for multimodal self-supervised learning from raw video, audio and text. Adv Neural Inf Process Syst 34:24206-24221
44. Kipf T, Pol E, Welling M (2019) Contrastive learning of structured world models. In: International conference on learning representations
45. Wang H, Wang Y, Zhou Z, Ji X, Gong D, Zhou J, Li Z, Liu W (2018) COSFACE: large margin cosine loss for deep face recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5265-5274
46. Sun Q-S, Zeng S-G, Liu Y, Heng P-A, Xia D-S (2005) A new method of feature fusion and its application in image recognition. Pattern Recognit 38(12):2437-2448
47. Zhiyuan L (2024) Investigating an ensemble classifier based on multi-objective genetic algorithm for machine learning applications. Int J Adv Comput Sci Appl 15(5):756
48. Rudnick EM, Patel JH, Greenstein GS, Niermann TM (1997) A genetic algorithm framework for test generation. IEEE Trans Comput Aided Des Integr Circuits Syst 16(9):1034-1044
49. Takahashi M, Kita H (2001) A crossover operator using independent component analysis for real-coded genetic algorithms. In: Proceedings of the 2001 congress on evolutionary computation (IEEE Cat. No. 01th8546). IEEE, vol 1, pp 643-649
50. Schulman J, Wolski F, Dhariwal P, Radford A, Klimov O (2017) Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06347
51. Rafin A, Hill A, Gleave A, Kanervisto A, Ernestus M, Dormann N (2021) Stable-baselines3: reliable reinforcement learning implementations. J Mach Learn Res 22(268):1-8
52. Kingma DP (2014) ADAM: a method for stochastic optimization. arXiv preprint arXiv:1412.6980
53. Loshchilov I, Hutter F (2017) SGDR: stochastic gradient descent with warm restarts. In: International conference on learning representations. https://openreview.net/forum?id=Skq89Scxx
54. Student (1908) The probable error of a mean. Biometrika 1-25
55. Wilcoxon F (1992) Individual comparisons by ranking methods. In: Breakthroughs in statistics: methodology and distribution. Springer, New York, pp 196-202
Metrics
Back to previous page
BibTeX entry
@article{oai:iris.cnr.it:20.500.14243/552085,
title = {ARTEMIS: animal recognition through enhanced multimodal integration system},
author = {Fazzari E. and Romano D. and Falchi F. and Stefanini C.},
doi = {10.1007/s13042-025-02602-3},
year = {2025}
}
0000-0001-6258-5313
