Typology
operator:
and / or
Language
operator:
and / or
Date
operator:
and / or
Rights
operator:
and / or
2025
Journal article
Open Access
ARTEMIS: animal recognition through enhanced multimodal integration system
Fazzari E., Romano D., Falchi F., Stefanini C.
This paper introduces Animal Recognition Through Enhanced Multimodal Integration System (ARTEMIS), a transformer-based framework designed for multilabel animal action recognition by fusing video, image, and textual modalities. ARTEMIS utilizes state-of-the-art captioning and language models, such as BLIP2 and Llama 3, to generate textual descriptions from video frames, which are input to the model, significantly enhancing its performance unlikely previous results that do not consider this modality. Through comprehensive ablation studies, we explore the contribution of various model components and propose optimization strategies, including genetic algorithms and reinforcement learning, to dynamically adjust ensemble weights. Our feature alignment techniques-using contrastive and cosine similarity losses-further improve multimodal integration. Evaluations on the Animal Kingdom dataset, which includes 30,100 clips across 140 action classes, demonstrate that ARTEMIS achieves a new state-of-the-art mAP of 79.82, outperforming existing methods. The combination of multimodal fusion and ensemble strategies makes ARTEMIS a robust solution for complex animal action recognition tasks. The code of our fusion method is available at https://github.com/edofazza/ARTEMIS.Source: INTERNATIONAL JOURNAL OF MACHINE LEARNING AND CYBERNETICS
DOI: 10.1007/s13042-025-02602-3
Project(s): Robocoenosis
Metrics:
Fazzari E., Romano D., Falchi F., Stefanini C.
This paper introduces Animal Recognition Through Enhanced Multimodal Integration System (ARTEMIS), a transformer-based framework designed for multilabel animal action recognition by fusing video, image, and textual modalities. ARTEMIS utilizes state-of-the-art captioning and language models, such as BLIP2 and Llama 3, to generate textual descriptions from video frames, which are input to the model, significantly enhancing its performance unlikely previous results that do not consider this modality. Through comprehensive ablation studies, we explore the contribution of various model components and propose optimization strategies, including genetic algorithms and reinforcement learning, to dynamically adjust ensemble weights. Our feature alignment techniques-using contrastive and cosine similarity losses-further improve multimodal integration. Evaluations on the Animal Kingdom dataset, which includes 30,100 clips across 140 action classes, demonstrate that ARTEMIS achieves a new state-of-the-art mAP of 79.82, outperforming existing methods. The combination of multimodal fusion and ensemble strategies makes ARTEMIS a robust solution for complex animal action recognition tasks. The code of our fusion method is available at https://github.com/edofazza/ARTEMIS.Source: INTERNATIONAL JOURNAL OF MACHINE LEARNING AND CYBERNETICS
DOI: 10.1007/s13042-025-02602-3
Project(s): Robocoenosis
Metrics:
See at:
Archivio della ricerca della Scuola Superiore Sant'Anna 
| Archivio della ricerca della Scuola Superiore Sant'Anna | CNR IRIS | link.springer.com | GitHub 
| CNR IRIS
2025
Journal article
Open Access
Animal behavior analysis methods using deep learning: a survey
Fazzari E., Romano D., Falchi F., Stefanini C.
Animal behavior serves as a reliable indicator of the adaptation of organisms to their environment and their overall well-being. Through rigorous observation of animal actions and interactions, researchers and observers can glean valuable insights into diverse facets of their lives, encompassing health, social dynamics, ecological relationships, and neuroethological dimensions. Although state-of-the-art deep learning models have demonstrated remarkable accuracy in classifying various forms of animal data, their adoption in animal behavior studies remains limited. This survey article endeavors to comprehensively explore deep learning architectures and strategies applied to the identification of animal behavior, spanning auditory, visual, and audiovisual methodologies. The survey categorizes techniques into pose estimation-based and non-pose estimation-based methods, analyzing their applications, effectiveness, and limitations. Furthermore, the manuscript scrutinizes extant animal behavior datasets, offering a detailed examination of the principal challenges confronting this research domain. The article culminates in a comprehensive discussion of key research directions within deep learning that hold potential for advancing the field of animal behavior studies.Source: EXPERT SYSTEMS WITH APPLICATIONS, vol. 289 (issue 128330)
DOI: 10.1016/j.eswa.2025.128330
DOI: 10.48550/arxiv.2405.14002
Metrics:
Fazzari E., Romano D., Falchi F., Stefanini C.
Animal behavior serves as a reliable indicator of the adaptation of organisms to their environment and their overall well-being. Through rigorous observation of animal actions and interactions, researchers and observers can glean valuable insights into diverse facets of their lives, encompassing health, social dynamics, ecological relationships, and neuroethological dimensions. Although state-of-the-art deep learning models have demonstrated remarkable accuracy in classifying various forms of animal data, their adoption in animal behavior studies remains limited. This survey article endeavors to comprehensively explore deep learning architectures and strategies applied to the identification of animal behavior, spanning auditory, visual, and audiovisual methodologies. The survey categorizes techniques into pose estimation-based and non-pose estimation-based methods, analyzing their applications, effectiveness, and limitations. Furthermore, the manuscript scrutinizes extant animal behavior datasets, offering a detailed examination of the principal challenges confronting this research domain. The article culminates in a comprehensive discussion of key research directions within deep learning that hold potential for advancing the field of animal behavior studies.Source: EXPERT SYSTEMS WITH APPLICATIONS, vol. 289 (issue 128330)
DOI: 10.1016/j.eswa.2025.128330
DOI: 10.48550/arxiv.2405.14002
Metrics:
See at:
arXiv.org e-Print Archive | Expert Systems with Applications | Archivio della ricerca della Scuola Superiore Sant'Anna | CNR IRIS | www.sciencedirect.com | doi.org | CNR IRIS
2025
Journal article
Open Access
The Game Boy learning environment
Fazzari E., Donato R., Falchi F., Stefanini C.
In this article, we introduce the Game Boy Learning Environment (GLE), an innovative suite based on Nintendo Game Boy games, crafted to advance and evaluate deep reinforcement learning algorithms on rich and varied gameplay tasks. GLE offers a comprehensive selection of eleven Game Boy environments, spanning nine distinct titles. These environments represent a significant leap in complexity compared to previous endeavors, like the Arcade Learning Environment, presenting challenges for reinforcement learning like intricate long-term planning, strategic foresight, and hierarchical decision-making, posing substantial difficulties even for proficient human players. We delineate the spectrum of available environments and furnish initial baseline results obtained through the development and assessment of intelligent agents, employing established AI methodologies to address individual levels or subtasks within these environments.Source: IEEE TRANSACTIONS ON GAMES
DOI: 10.1109/tg.2025.3575527
Metrics:
Fazzari E., Donato R., Falchi F., Stefanini C.
In this article, we introduce the Game Boy Learning Environment (GLE), an innovative suite based on Nintendo Game Boy games, crafted to advance and evaluate deep reinforcement learning algorithms on rich and varied gameplay tasks. GLE offers a comprehensive selection of eleven Game Boy environments, spanning nine distinct titles. These environments represent a significant leap in complexity compared to previous endeavors, like the Arcade Learning Environment, presenting challenges for reinforcement learning like intricate long-term planning, strategic foresight, and hierarchical decision-making, posing substantial difficulties even for proficient human players. We delineate the spectrum of available environments and furnish initial baseline results obtained through the development and assessment of intelligent agents, employing established AI methodologies to address individual levels or subtasks within these environments.Source: IEEE TRANSACTIONS ON GAMES
DOI: 10.1109/tg.2025.3575527
Metrics:
See at:
CNR IRIS | ieeexplore.ieee.org | IEEE Transactions on Games | CNR IRIS | CNR IRIS
2025
Journal article
Open Access
Real-time behavior recognition using a legged robot for animal-robot interaction
Fazzari E., Romano D., Falchi F., Stefanini C.
Animal-robot interaction is an emerging interdisciplinary field that explores the dynamics between animals and robotic systems, as well as the design principles for effective engagement. While previous approaches have investigated animal responses to robotic stimuli, they have yet to integrate artificial intelligence (AI) for real-time behavioral analysis during the interaction. This paper addresses this gap by introducing an AI-driven framework that enables a robotic dog to autonomously monitor and analyze livestock behavior, specifically in cows and chickens. Our system processes real-time camera observations using deep-learning models to detect animal presence and recognize actions. It integrates three neural networks: YOLO-Chicken and YOLO-Cows, for accurate detection of chickens and cows, respectively, and DARTEMIS, a novel, distilled unimodal variant of a state-of-the-art Animal Action Recognition model. The networks communicate efficiently via Redis in a lightweight manner, with all processing conducted onboard the robot. We trained YOLO-Cow and YOLO-Chicken on a subset of the COCO data set for cows and a public data set for chickens, achieving mAP@50-95 scores of 0.67 and 0.56, respectively. DARTEMIS, trained on the Animal Kingdom data set like ARTEMIS, reached an mAP of 77.3. With these models, we tested our system in real-world conditions through field trials, evaluating its ability to accurately detect animals and classify their behaviors. This study presents the first successful integration of efficient deep- learning models into a robotic platform for real-time animal behavior analysis. The proposed framework paves the way for continuous automated livestock monitoring, with potential applications in improving animal welfare and farm management. The full implementation is publicly available and designed to be adaptable to various robotic platforms and related challenges.Source: JOURNAL OF FIELD ROBOTICS
DOI: 10.1002/rob.70123
Metrics:
Fazzari E., Romano D., Falchi F., Stefanini C.
Animal-robot interaction is an emerging interdisciplinary field that explores the dynamics between animals and robotic systems, as well as the design principles for effective engagement. While previous approaches have investigated animal responses to robotic stimuli, they have yet to integrate artificial intelligence (AI) for real-time behavioral analysis during the interaction. This paper addresses this gap by introducing an AI-driven framework that enables a robotic dog to autonomously monitor and analyze livestock behavior, specifically in cows and chickens. Our system processes real-time camera observations using deep-learning models to detect animal presence and recognize actions. It integrates three neural networks: YOLO-Chicken and YOLO-Cows, for accurate detection of chickens and cows, respectively, and DARTEMIS, a novel, distilled unimodal variant of a state-of-the-art Animal Action Recognition model. The networks communicate efficiently via Redis in a lightweight manner, with all processing conducted onboard the robot. We trained YOLO-Cow and YOLO-Chicken on a subset of the COCO data set for cows and a public data set for chickens, achieving mAP@50-95 scores of 0.67 and 0.56, respectively. DARTEMIS, trained on the Animal Kingdom data set like ARTEMIS, reached an mAP of 77.3. With these models, we tested our system in real-world conditions through field trials, evaluating its ability to accurately detect animals and classify their behaviors. This study presents the first successful integration of efficient deep- learning models into a robotic platform for real-time animal behavior analysis. The proposed framework paves the way for continuous automated livestock monitoring, with potential applications in improving animal welfare and farm management. The full implementation is publicly available and designed to be adaptable to various robotic platforms and related challenges.Source: JOURNAL OF FIELD ROBOTICS
DOI: 10.1002/rob.70123
Metrics:
See at:
CNR IRIS | onlinelibrary.wiley.com | CNR IRIS
2024
Other
Open Access
AIMH Research Activities 2024
Aloia N., Amato G., Bartalesi Lenzi V., Bianchi L., Bolettieri P., Bosio C., Carraglia M., Carrara F., Casarosa V., Cassese M., Ciampi L., Coccomini D. A., Concordia C., Connor R., Corbara S., De Martino C., Di Benedetto M., Esuli A., Falchi F., Fazzari E., Gennaro C., Iannello L., Negi K., Lagani G., Lenzi E., Leocata M., Malvaldi M., Meghini C., Messina N., Moreo Fernandez A., Nardi A., Pacini G., Pedrotti A., Pratelli N., Puccetti G., Rabitti F., Savino P., Scotti F., Sebastiani F., Sperduti G., Thanos C., Trupiano L., Vadicamo L., Vairo C., Versienti L., Volpi L.
The AIMH (Artificial Intelligence for Media and Humanities) laboratory is committed to advancing the field of Artificial Intelligence, with a special emphasis on its applications in digital media and the humanities. The lab aims to improve AI technologies, particularly in areas such as deep learning, text analysis, computer vision, multimedia information retrieval, content analysis, recognition, and retrieval. This report summarizes the laboratory’s achievements and activities over the course of 2024.DOI: 10.32079/isti-ar-2024/001
Metrics:
Aloia N., Amato G., Bartalesi Lenzi V., Bianchi L., Bolettieri P., Bosio C., Carraglia M., Carrara F., Casarosa V., Cassese M., Ciampi L., Coccomini D. A., Concordia C., Connor R., Corbara S., De Martino C., Di Benedetto M., Esuli A., Falchi F., Fazzari E., Gennaro C., Iannello L., Negi K., Lagani G., Lenzi E., Leocata M., Malvaldi M., Meghini C., Messina N., Moreo Fernandez A., Nardi A., Pacini G., Pedrotti A., Pratelli N., Puccetti G., Rabitti F., Savino P., Scotti F., Sebastiani F., Sperduti G., Thanos C., Trupiano L., Vadicamo L., Vairo C., Versienti L., Volpi L.
The AIMH (Artificial Intelligence for Media and Humanities) laboratory is committed to advancing the field of Artificial Intelligence, with a special emphasis on its applications in digital media and the humanities. The lab aims to improve AI technologies, particularly in areas such as deep learning, text analysis, computer vision, multimedia information retrieval, content analysis, recognition, and retrieval. This report summarizes the laboratory’s achievements and activities over the course of 2024.DOI: 10.32079/isti-ar-2024/001
Metrics:
2024
Journal article
Open Access
Selective state models are what you need for animal action recognition
Fazzari E., Romano D., Falchi F., Stefanini C.
Recognizing animal actions provides valuable insights into animal welfare, yielding crucial information for agricultural, ethological, and neuroscientific research. While video-based action recognition models have been applied to this task, current approaches often rely on computationally intensive Transformer layers, limiting their practical application in field settings such as farms and wildlife reserves. This study introduces Mamba-MSQNet, a novel architecture family for multilabel Animal Action Recognition using Selective Space Models. By transforming the state-of-the-art MSQNet model with Mamba blocks, we achieve significant reductions in computational requirements: up to 90% fewer Floating point OPerations and 78% fewer parameters compared to MSQNet. These optimizations not only make the model more efficient but also enable it to outperform Transformer-based counterparts on the Animal Kingdom dataset, achieving a mean Average Precision of 74.6, marking an improvement over previous architectures. This combination of enhanced efficiency and improved performance represents a significant advancement in the field of animal action recognition. The dramatic reduction in computational demands, coupled with a performance boost, opens new possibilities for real-time animal behavior monitoring in resource-constrained environments. This enhanced efficiency could revolutionize how we observe and analyze animal behavior, potentially leading to breakthroughs in animal welfare assessment, behavioral studies, and conservation efforts.Source: ECOLOGICAL INFORMATICS
DOI: 10.1016/j.ecoinf.2024.102955
Metrics:
Fazzari E., Romano D., Falchi F., Stefanini C.
Recognizing animal actions provides valuable insights into animal welfare, yielding crucial information for agricultural, ethological, and neuroscientific research. While video-based action recognition models have been applied to this task, current approaches often rely on computationally intensive Transformer layers, limiting their practical application in field settings such as farms and wildlife reserves. This study introduces Mamba-MSQNet, a novel architecture family for multilabel Animal Action Recognition using Selective Space Models. By transforming the state-of-the-art MSQNet model with Mamba blocks, we achieve significant reductions in computational requirements: up to 90% fewer Floating point OPerations and 78% fewer parameters compared to MSQNet. These optimizations not only make the model more efficient but also enable it to outperform Transformer-based counterparts on the Animal Kingdom dataset, achieving a mean Average Precision of 74.6, marking an improvement over previous architectures. This combination of enhanced efficiency and improved performance represents a significant advancement in the field of animal action recognition. The dramatic reduction in computational demands, coupled with a performance boost, opens new possibilities for real-time animal behavior monitoring in resource-constrained environments. This enhanced efficiency could revolutionize how we observe and analyze animal behavior, potentially leading to breakthroughs in animal welfare assessment, behavioral studies, and conservation efforts.Source: ECOLOGICAL INFORMATICS
DOI: 10.1016/j.ecoinf.2024.102955
Metrics:
See at:
Ecological Informatics | CNR IRIS | www.sciencedirect.com | CNR IRIS | CNR IRIS
2024
Conference article
Open Access
Mamba-MSQNet: a fast and efficient model for animal action recognition
Fazzari E., Romano D., Falchi F., Stefanini C.
Animal action recognition is crucial for assessing animal well-being in agriculture and environmental monitoring. Recent advancements in this field rely on computer vision technologies. However, many current applications are restricted to recognizing actions within a single animal species or a limited set of actions, resulting in highly specific models and lacking generality. When addressing a broader range of actions and species, transformer models are typically required, which demand significant computational and processing power, potentially limiting their practical use. In this work, we introduce a deep learning model based on selective state spaces designed to reduce the computational cost of MSQNet, the current state-of-the-art model for action recognition in the Animal Kingdom dataset. Our approach achieves superior results with fewer parameters and lower FLOPs, thereby enhancing efficiency without compromising performance. Code available on https://github.com/edofazza/mamba-msqnet.DOI: 10.1109/metroagrifor63043.2024.10948802
Metrics:
Fazzari E., Romano D., Falchi F., Stefanini C.
Animal action recognition is crucial for assessing animal well-being in agriculture and environmental monitoring. Recent advancements in this field rely on computer vision technologies. However, many current applications are restricted to recognizing actions within a single animal species or a limited set of actions, resulting in highly specific models and lacking generality. When addressing a broader range of actions and species, transformer models are typically required, which demand significant computational and processing power, potentially limiting their practical use. In this work, we introduce a deep learning model based on selective state spaces designed to reduce the computational cost of MSQNet, the current state-of-the-art model for action recognition in the Animal Kingdom dataset. Our approach achieves superior results with fewer parameters and lower FLOPs, thereby enhancing efficiency without compromising performance. Code available on https://github.com/edofazza/mamba-msqnet.DOI: 10.1109/metroagrifor63043.2024.10948802
Metrics:
See at:
CNR IRIS | ieeexplore.ieee.org | doi.org | Archivio della ricerca della Scuola Superiore Sant'Anna | CNR IRIS | CNR IRIS
2023
Other
Open Access
AIMH Research Activities 2023
Aloia N., Amato G., Bartalesi Lenzi V., Bianchi L., Bolettieri P., Bosio C., Carraglia M., Carrara F., Casarosa V., Ciampi L., Coccomini D. A., Concordia C., Corbara S., De Martino C., Di Benedetto M., Esuli A., Falchi F., Fazzari E., Gennaro C., Lagani G., Lenzi E., Meghini C., Messina N., Molinari A., Moreo Fernandez A., Nardi A., Pedrotti A., Pratelli N., Puccetti G., Rabitti F., Savino P., Sebastiani F., Sperduti G., Thanos C., Trupiano L., Vadicamo L., Vairo C., Versienti L.
The AIMH (Artificial Intelligence for Media and Humanities) laboratory is dedicated to exploring and pushing the boundaries in the field of Artificial Intelligence, with a particular focus on its application in digital media and humanities. This lab's objective is to enhance the current state of AI technology particularly on deep learning, text analysis, computer vision, multimedia information retrieval, multimedia content analysis, recognition, and retrieval. This report encapsulates the laboratory's progress and activities throughout the year 2023.DOI: https://doi.org/10.32079/isti-ar-2023/001
DOI: 10.32079/isti-ar-2023/001
Metrics:
Aloia N., Amato G., Bartalesi Lenzi V., Bianchi L., Bolettieri P., Bosio C., Carraglia M., Carrara F., Casarosa V., Ciampi L., Coccomini D. A., Concordia C., Corbara S., De Martino C., Di Benedetto M., Esuli A., Falchi F., Fazzari E., Gennaro C., Lagani G., Lenzi E., Meghini C., Messina N., Molinari A., Moreo Fernandez A., Nardi A., Pedrotti A., Pratelli N., Puccetti G., Rabitti F., Savino P., Sebastiani F., Sperduti G., Thanos C., Trupiano L., Vadicamo L., Vairo C., Versienti L.
The AIMH (Artificial Intelligence for Media and Humanities) laboratory is dedicated to exploring and pushing the boundaries in the field of Artificial Intelligence, with a particular focus on its application in digital media and humanities. This lab's objective is to enhance the current state of AI technology particularly on deep learning, text analysis, computer vision, multimedia information retrieval, multimedia content analysis, recognition, and retrieval. This report encapsulates the laboratory's progress and activities throughout the year 2023.DOI: https://doi.org/10.32079/isti-ar-2023/001
DOI: 10.32079/isti-ar-2023/001
Metrics:
See at:
CNR IRIS | ISTI Repository | CNR IRIS