2025
Book
Open Access
SUN: Social and hUman ceNtered XR - A Horizon Europe Project Paving the Way for the Widespread Adoption of Extended and Virtual Worlds
Vairo C., Caracciolo G., Giorgi D., Leonardis D., Vadicamo L.
Extended Reality (XR) is a rapidly growing technology that bridges physical and virtual worlds, opening up new possibilities in healthcare, communications, and security. The European project SUN – Social and hUman ceNtered XR, funded by the Horizon Europe program, addresses the ongoing challenges of making XR more accessible, usable, and realistic. SUN develops technologies and models that enhance social interaction and immersive perception, while keeping an ethical and human-centered design, by introducing new wearable sensors, haptic interfaces, and high-performance streaming solutions. Through new 3D acquisition techniques and the use of artificial intelligence, SUN explores innovative ways to connect physical objects and digital counterparts, creating coherent and immersive environments. The project’s innovations were validated in three real-world piloting scenarios: rehabilitation therapy, workplace safety and social interaction, and assistive technologies for individuals with severe mobility or communication impairments. This volume presents the results of three years of research and development, offering a solid vision of how XR can evolve in a sustainable, ethical, and human-centered way.DOI: 10.32079/isti-book-2025/001
Project(s): SUN
Metrics:
Vairo C., Caracciolo G., Giorgi D., Leonardis D., Vadicamo L.
Extended Reality (XR) is a rapidly growing technology that bridges physical and virtual worlds, opening up new possibilities in healthcare, communications, and security. The European project SUN – Social and hUman ceNtered XR, funded by the Horizon Europe program, addresses the ongoing challenges of making XR more accessible, usable, and realistic. SUN develops technologies and models that enhance social interaction and immersive perception, while keeping an ethical and human-centered design, by introducing new wearable sensors, haptic interfaces, and high-performance streaming solutions. Through new 3D acquisition techniques and the use of artificial intelligence, SUN explores innovative ways to connect physical objects and digital counterparts, creating coherent and immersive environments. The project’s innovations were validated in three real-world piloting scenarios: rehabilitation therapy, workplace safety and social interaction, and assistive technologies for individuals with severe mobility or communication impairments. This volume presents the results of three years of research and development, offering a solid vision of how XR can evolve in a sustainable, ethical, and human-centered way.DOI: 10.32079/isti-book-2025/001
Project(s): SUN
Metrics:
See at:
CNR IRIS 
| www.cnr.it | CNR IRIS 
2024
Journal article
Open Access
Interactive multimodal video search: an extended post-evaluation for the VBS 2022 competition
Schall K., Bailer W., Barthel K. -U., Carrara F., Lokoč J., Peška L., Schoeffmann K., Vadicamo L., Vairo C.
CLIP-based text-to-image retrieval has proven to be very effective at the interactive video retrieval competition Video Browser Showdown 2022, where all three top-scoring teams had implemented a variant of a CLIP model in their system. Since the performance of these three systems was quite close, this post-evaluation was designed to get better insights on the differences of the systems and compare the CLIP-based text-query retrieval engines by introducing slight modifications to the original competition settings. An extended analysis of the overall results and the retrieval performance of all systems’ functionalities shows that a strong text retrieval model certainly helps, but has to be coupled with extensive browsing capabilities and other query-modalities to consistently solve known-item-search tasks in a large-scale video database.Source: INTERNATIONAL JOURNAL OF MULTIMEDIA INFORMATION RETRIEVAL, vol. 13 (issue 15)
DOI: 10.1007/s13735-024-00325-9
Project(s): AI4Media, XReco
Metrics:
Schall K., Bailer W., Barthel K. -U., Carrara F., Lokoč J., Peška L., Schoeffmann K., Vadicamo L., Vairo C.
CLIP-based text-to-image retrieval has proven to be very effective at the interactive video retrieval competition Video Browser Showdown 2022, where all three top-scoring teams had implemented a variant of a CLIP model in their system. Since the performance of these three systems was quite close, this post-evaluation was designed to get better insights on the differences of the systems and compare the CLIP-based text-query retrieval engines by introducing slight modifications to the original competition settings. An extended analysis of the overall results and the retrieval performance of all systems’ functionalities shows that a strong text retrieval model certainly helps, but has to be coupled with extensive browsing capabilities and other query-modalities to consistently solve known-item-search tasks in a large-scale video database.Source: INTERNATIONAL JOURNAL OF MULTIMEDIA INFORMATION RETRIEVAL, vol. 13 (issue 15)
DOI: 10.1007/s13735-024-00325-9
Project(s): AI4Media
, XReco Metrics:
See at:
CNR IRIS | link.springer.com | CNR IRIS | CNR IRIS
2024
Conference article
Open Access
Will VISIONE remain competitive in lifelog image search?
Amato G., Bolettieri P., Carrara F., Falchi F., Gennaro C., Messina N., Vadicamo L., Vairo C.
VISIONE is a versatile video retrieval system supporting diverse search functionalities, including free-text, similarity, and temporal searches. Its recent success in securing first place in the 2024 Video Browser Showdown (VBS) highlights its effectiveness. Originally designed for analyzing, indexing, and searching diverse video content, VISIONE can also be adapted to images from lifelog cameras thanks to its reliance on frame-based representations and retrieval mechanisms. In this paper, we present an overview of VISIONE's core characteristics and the adjustments made to accommodate lifelog images. These adjustments primarily focus on enhancing result visualization within the GUI, such as grouping images by date or hour to align with lifelog dataset imagery. It's important to note that while the GUI has been updated, the core search engine and visual content analysis components remain unchanged from the version presented at VBS 2024. Specifically, metadata such as local time, GPS coordinates, and concepts associated with images are not indexed or utilized in the system. Instead, the system relies solely on the visual content of the images, with date and time information extracted from their filenames, which are utilized exclusively within the GUI for visualization purposes. Our objective is to evaluate the system's performance within the Lifelog Search Challenge, emphasizing reliance on visual content analysis without additional metadata.DOI: 10.1145/3643489.3661122
Project(s): AI4Media
Metrics:
Amato G., Bolettieri P., Carrara F., Falchi F., Gennaro C., Messina N., Vadicamo L., Vairo C.
VISIONE is a versatile video retrieval system supporting diverse search functionalities, including free-text, similarity, and temporal searches. Its recent success in securing first place in the 2024 Video Browser Showdown (VBS) highlights its effectiveness. Originally designed for analyzing, indexing, and searching diverse video content, VISIONE can also be adapted to images from lifelog cameras thanks to its reliance on frame-based representations and retrieval mechanisms. In this paper, we present an overview of VISIONE's core characteristics and the adjustments made to accommodate lifelog images. These adjustments primarily focus on enhancing result visualization within the GUI, such as grouping images by date or hour to align with lifelog dataset imagery. It's important to note that while the GUI has been updated, the core search engine and visual content analysis components remain unchanged from the version presented at VBS 2024. Specifically, metadata such as local time, GPS coordinates, and concepts associated with images are not indexed or utilized in the system. Instead, the system relies solely on the visual content of the images, with date and time information extracted from their filenames, which are utilized exclusively within the GUI for visualization purposes. Our objective is to evaluate the system's performance within the Lifelog Search Challenge, emphasizing reliance on visual content analysis without additional metadata.DOI: 10.1145/3643489.3661122
Project(s): AI4Media
Metrics:
2024
Conference article
Open Access
Visione 5.0: toward evaluation with novice users
Amato G., Bolettieri P., Carrara F., Falchi F., Gennaro C., Messina N., Vadicamo L., Vairo C.
VISIONE is a video search system that integrates multiple search functionalities, allowing users to search for video segments using textual and visual queries, complemented by temporal search capabilities. It exploits state-of-the-art Artificial Intelligence approaches for visual content analysis and highly efficient indexing techniques to ensure fast response and scalability. In the recently concluded Video Browser Showdown (VBS2024) - a well-established international competition in interactive video retrieval - VISIONE ranked first and scored as the best interactive video search system in four out of seven tasks carried out in the competition.This paper provides an overview of the VISIONE system, emphasizing the improvements made to the system in the last year to improve its usability for novice users. A demonstration video showcasing the system's capabilities across 2,300 hours of diverse video content is available online, as well as a simplified demo of VISIONE.DOI: 10.1109/cbmi62980.2024.10859203
Project(s): AI4Media, National Centre for HPC, Big Data and Quantum Computing, a MUltimedia platform for Content Enrichment and Search in audiovisual archives
Metrics:
Amato G., Bolettieri P., Carrara F., Falchi F., Gennaro C., Messina N., Vadicamo L., Vairo C.
VISIONE is a video search system that integrates multiple search functionalities, allowing users to search for video segments using textual and visual queries, complemented by temporal search capabilities. It exploits state-of-the-art Artificial Intelligence approaches for visual content analysis and highly efficient indexing techniques to ensure fast response and scalability. In the recently concluded Video Browser Showdown (VBS2024) - a well-established international competition in interactive video retrieval - VISIONE ranked first and scored as the best interactive video search system in four out of seven tasks carried out in the competition.This paper provides an overview of the VISIONE system, emphasizing the improvements made to the system in the last year to improve its usability for novice users. A demonstration video showcasing the system's capabilities across 2,300 hours of diverse video content is available online, as well as a simplified demo of VISIONE.DOI: 10.1109/cbmi62980.2024.10859203
Project(s): AI4Media
, National Centre for HPC, Big Data and Quantum Computing, a MUltimedia platform for Content Enrichment and Search in audiovisual archivesMetrics:
See at:
CNR IRIS | ieeexplore.ieee.org | CNR IRIS | CNR IRIS
2024
Other
Open Access
AIMH Research Activities 2024
Aloia N., Amato G., Bartalesi Lenzi V., Bianchi L., Bolettieri P., Bosio C., Carraglia M., Carrara F., Casarosa V., Cassese M., Ciampi L., Coccomini D. A., Concordia C., Connor R., Corbara S., De Martino C., Di Benedetto M., Esuli A., Falchi F., Fazzari E., Gennaro C., Iannello L., Negi K., Lagani G., Lenzi E., Leocata M., Malvaldi M., Meghini C., Messina N., Moreo Fernandez A., Nardi A., Pacini G., Pedrotti A., Pratelli N., Puccetti G., Rabitti F., Savino P., Scotti F., Sebastiani F., Sperduti G., Thanos C., Trupiano L., Vadicamo L., Vairo C., Versienti L., Volpi L.
The AIMH (Artificial Intelligence for Media and Humanities) laboratory is committed to advancing the field of Artificial Intelligence, with a special emphasis on its applications in digital media and the humanities. The lab aims to improve AI technologies, particularly in areas such as deep learning, text analysis, computer vision, multimedia information retrieval, content analysis, recognition, and retrieval. This report summarizes the laboratory’s achievements and activities over the course of 2024.DOI: 10.32079/isti-ar-2024/001
Metrics:
Aloia N., Amato G., Bartalesi Lenzi V., Bianchi L., Bolettieri P., Bosio C., Carraglia M., Carrara F., Casarosa V., Cassese M., Ciampi L., Coccomini D. A., Concordia C., Connor R., Corbara S., De Martino C., Di Benedetto M., Esuli A., Falchi F., Fazzari E., Gennaro C., Iannello L., Negi K., Lagani G., Lenzi E., Leocata M., Malvaldi M., Meghini C., Messina N., Moreo Fernandez A., Nardi A., Pacini G., Pedrotti A., Pratelli N., Puccetti G., Rabitti F., Savino P., Scotti F., Sebastiani F., Sperduti G., Thanos C., Trupiano L., Vadicamo L., Vairo C., Versienti L., Volpi L.
The AIMH (Artificial Intelligence for Media and Humanities) laboratory is committed to advancing the field of Artificial Intelligence, with a special emphasis on its applications in digital media and the humanities. The lab aims to improve AI technologies, particularly in areas such as deep learning, text analysis, computer vision, multimedia information retrieval, content analysis, recognition, and retrieval. This report summarizes the laboratory’s achievements and activities over the course of 2024.DOI: 10.32079/isti-ar-2024/001
Metrics:
2024
Other
Open Access
SUN D5.1 - SUN Platform architecture
Ferdinando Bosco, Giovanni Di Marco, Alexandru Stan, George Loukas, Panagiotis Kasnesis, Lazaros Toumanidis, Ioannis Paraskevopoulos, Vincent Mendez, Leesa Joyce, Claudio Vairo, Spyridon Symeonidis, Sotiris Diplaris, Vasileios-Rafail Xefteris, Ilias Poulios, Panagiotis Vrachnos, Georgios Loupas, Orestis Sarakatsanos
This document reports the SUN Architecture design and technical specifications for implementing the SUN XR Platform.Project(s): SUN
Ferdinando Bosco, Giovanni Di Marco, Alexandru Stan, George Loukas, Panagiotis Kasnesis, Lazaros Toumanidis, Ioannis Paraskevopoulos, Vincent Mendez, Leesa Joyce, Claudio Vairo, Spyridon Symeonidis, Sotiris Diplaris, Vasileios-Rafail Xefteris, Ilias Poulios, Panagiotis Vrachnos, Georgios Loupas, Orestis Sarakatsanos
This document reports the SUN Architecture design and technical specifications for implementing the SUN XR Platform.Project(s): SUN
2024
Conference article
Open Access
VISIONE 5.0: enhanced user interface and AI models for VBS2024
Giuseppe Amato, Paolo Bolettieri, Fabio Carrara, Fabrizio Falchi, Claudio Gennaro, Nicola Messina, Lucia Vadicamo, Claudio Vairo
In this paper, we introduce the fifth release of VISIONE, an advanced video retrieval system offering diverse search functionalities. The user can search for a target video using textual prompts, drawing objects and colors appearing in the target scenes in a canvas, or images as query examples to search for video keyframes with similar content. Compared to the previous version of our system, which was runner-up at VBS 2023, the forthcoming release, set to participate in VBS 2024, showcases a refined user interface that enhances its usability and updated AI models for more effective video content analysis.Source: LECTURE NOTES IN COMPUTER SCIENCE, vol. 14557, pp. 332-339. Amsterdam, NL, 29/01-2/02/2024
DOI: 10.1007/978-3-031-53302-0_29
DOI: https://doi.org/10.1007/978-3-031-53302-0_29
Project(s): AI4Media, SUN
Metrics:
Giuseppe Amato, Paolo Bolettieri, Fabio Carrara, Fabrizio Falchi, Claudio Gennaro, Nicola Messina, Lucia Vadicamo, Claudio Vairo
In this paper, we introduce the fifth release of VISIONE, an advanced video retrieval system offering diverse search functionalities. The user can search for a target video using textual prompts, drawing objects and colors appearing in the target scenes in a canvas, or images as query examples to search for video keyframes with similar content. Compared to the previous version of our system, which was runner-up at VBS 2023, the forthcoming release, set to participate in VBS 2024, showcases a refined user interface that enhances its usability and updated AI models for more effective video content analysis.Source: LECTURE NOTES IN COMPUTER SCIENCE, vol. 14557, pp. 332-339. Amsterdam, NL, 29/01-2/02/2024
DOI: 10.1007/978-3-031-53302-0_29
DOI: https://doi.org/10.1007/978-3-031-53302-0_29
Project(s): AI4Media
, SUN Metrics:
2023
Conference article
Open Access
Social and hUman ceNtered XR
Vairo C, Callieri M, Carrara F, Cignoni P, Di Benedetto M, Gennaro C, Giorgi D, Palma G, Vadicamo L, Amato G
The Social and hUman ceNtered XR (SUN) project is focused on developing eXtended Reality (XR) solutions that integrate the physical and virtual world in a way that is convincing from a human and social perspective. In this paper, we outline the limitations that the SUN project aims to overcome, including the lack of scalable and cost-effective solutions for developing XR applications, limited solutions for mixing the virtual and physical environment, and barriers related to resource limitations of end-user devices. We also propose solutions to these limitations, including using artificial intelligence, computer vision, and sensor analysis to incrementally learn the visual and physical properties of real objects and generate convincing digital twins in the virtual environment. Additionally, the SUN project aims to provide wearable sensors and haptic interfaces to enhance natural interaction with the virtual environment and advanced solutions for user interaction. Finally, we describe three real-life scenarios in which we aim to demonstrate the proposed solutions.Source: CEUR WORKSHOP PROCEEDINGS. Pisa, Italy, 29-31/05/2023
Vairo C, Callieri M, Carrara F, Cignoni P, Di Benedetto M, Gennaro C, Giorgi D, Palma G, Vadicamo L, Amato G
The Social and hUman ceNtered XR (SUN) project is focused on developing eXtended Reality (XR) solutions that integrate the physical and virtual world in a way that is convincing from a human and social perspective. In this paper, we outline the limitations that the SUN project aims to overcome, including the lack of scalable and cost-effective solutions for developing XR applications, limited solutions for mixing the virtual and physical environment, and barriers related to resource limitations of end-user devices. We also propose solutions to these limitations, including using artificial intelligence, computer vision, and sensor analysis to incrementally learn the visual and physical properties of real objects and generate convincing digital twins in the virtual environment. Additionally, the SUN project aims to provide wearable sensors and haptic interfaces to enhance natural interaction with the virtual environment and advanced solutions for user interaction. Finally, we describe three real-life scenarios in which we aim to demonstrate the proposed solutions.Source: CEUR WORKSHOP PROCEEDINGS. Pisa, Italy, 29-31/05/2023
See at:
ceur-ws.org | CNR IRIS | ISTI Repository | ISTI Repository | CNR IRIS | CNR IRIS
2023
Other
Restricted
SUN D1.1 - Management Website
Amato G, Bolettieri P, Gennaro C, Vadicamo L, Vairo C
Report describing the online web accessible repository for all project-related documentation, which serves as the primary means for project partners to manage and share documents of the project. https://wiki.sun-xr-project.eu
Amato G, Bolettieri P, Gennaro C, Vadicamo L, Vairo C
Report describing the online web accessible repository for all project-related documentation, which serves as the primary means for project partners to manage and share documents of the project. https://wiki.sun-xr-project.eu
2023
Other
Restricted
THE D.3.2.1 - AA@THE User needs, technical requirements and specifications
Pratali L, Campana M G, Delmastro F, Di Martino F, Pescosolido L, Barsocchi P, Broccia G, Ciancia V, Gennaro C, Girolami M, Lagani G, La Rosa D, Latella D, Magrini M, Manca M, Massink M, Mattioli A, Moroni D, Palumbo F, Paradisi P, Paternò F, Santoro C, Sebastiani L, Vairo C
Deliverable D3.2.1 del progetto PNRR Ecosistemi ed innovazione - THE
Pratali L, Campana M G, Delmastro F, Di Martino F, Pescosolido L, Barsocchi P, Broccia G, Ciancia V, Gennaro C, Girolami M, Lagani G, La Rosa D, Latella D, Magrini M, Manca M, Massink M, Mattioli A, Moroni D, Palumbo F, Paradisi P, Paternò F, Santoro C, Sebastiani L, Vairo C
Deliverable D3.2.1 del progetto PNRR Ecosistemi ed innovazione - THE
2023
Other
Restricted
SUN D1.3 - Data Management and IPR issues
Boi S, Amato G, Vairo C, Casarosa V
This document presents the Data Management Plan (DMP) for the SUN project, outlining the methodology adopted to effectively manage all data collected, generated, or acquired during the project's lifecycle. The DMP encompasses the management of research and non-research data, covering aspects such as collection, storage, sharing, preservation, privacy, ethics, and data interoperability. The DMP also defines rules on intellectual property ownership, access rights to background and results, and the protection of intellectual property rights (IPRs).
Boi S, Amato G, Vairo C, Casarosa V
This document presents the Data Management Plan (DMP) for the SUN project, outlining the methodology adopted to effectively manage all data collected, generated, or acquired during the project's lifecycle. The DMP encompasses the management of research and non-research data, covering aspects such as collection, storage, sharing, preservation, privacy, ethics, and data interoperability. The DMP also defines rules on intellectual property ownership, access rights to background and results, and the protection of intellectual property rights (IPRs).
2023
Conference article
Open Access
VISIONE: a large-scale video retrieval system with advanced search functionalities
Amato G, Bolettieri P, Carrara F, Falchi F, Gennaro C, Messina N, Vadicamo L, Vairo C
VISIONE is a large-scale video retrieval system that integrates multiple search functionalities, including free text search, spatial color and object search, visual and semantic similarity search, and temporal search. The system leverages cutting-edge AI technology for visual analysis and advanced indexing techniques to ensure scalability. As demonstrated by its runner-up position in the 2023 Video Browser Showdown competition, VISIONE effectively integrates these capabilities to provide a comprehensive video retrieval solution. A system demo is available online, showcasing its capabilities on over 2300 hours of diverse video content (V3C1+V3C2 dataset) and 12 hours of highly redundant content (Marine dataset). The demo can be accessed at https://visione.isti.cnr.itDOI: 10.1145/3591106.3592226
DOI: https://doi.org/10.1145/3591106.3592226
Project(s): AI4Media
Metrics:
Amato G, Bolettieri P, Carrara F, Falchi F, Gennaro C, Messina N, Vadicamo L, Vairo C
VISIONE is a large-scale video retrieval system that integrates multiple search functionalities, including free text search, spatial color and object search, visual and semantic similarity search, and temporal search. The system leverages cutting-edge AI technology for visual analysis and advanced indexing techniques to ensure scalability. As demonstrated by its runner-up position in the 2023 Video Browser Showdown competition, VISIONE effectively integrates these capabilities to provide a comprehensive video retrieval solution. A system demo is available online, showcasing its capabilities on over 2300 hours of diverse video content (V3C1+V3C2 dataset) and 12 hours of highly redundant content (Marine dataset). The demo can be accessed at https://visione.isti.cnr.itDOI: 10.1145/3591106.3592226
DOI: https://doi.org/10.1145/3591106.3592226
Project(s): AI4Media
Metrics:
See at:
CNR IRIS | ISTI Repository | CNR IRIS | CNR IRIS
2023
Conference article
Open Access
VISIONE at Video Browser Showdown 2023
Amato G, Bolettieri P, Carrara F, Falchi F, Gennaro C, Messina N, Vadicamo L, Vairo C
In this paper, we present the fourth release of VISIONE, a tool for fast and effective video search on a large-scale dataset. It includes several search functionalities like text search, object and color-based search, semantic and visual similarity search, and temporal search. VISIONE uses ad-hoc textual encoding for indexing and searching video content, and it exploits a full-text search engine as search backend. In this new version of the system, we introduced some changes both to the current search techniques and to the user interface.DOI: 10.1007/978-3-031-27077-2_48
DOI: https://doi.org/10.1007/978-3-031-27077-2_48
Project(s): AI4Media
Metrics:
Amato G, Bolettieri P, Carrara F, Falchi F, Gennaro C, Messina N, Vadicamo L, Vairo C
In this paper, we present the fourth release of VISIONE, a tool for fast and effective video search on a large-scale dataset. It includes several search functionalities like text search, object and color-based search, semantic and visual similarity search, and temporal search. VISIONE uses ad-hoc textual encoding for indexing and searching video content, and it exploits a full-text search engine as search backend. In this new version of the system, we introduced some changes both to the current search techniques and to the user interface.DOI: 10.1007/978-3-031-27077-2_48
DOI: https://doi.org/10.1007/978-3-031-27077-2_48
Project(s): AI4Media
Metrics:
See at:
CNR IRIS | ISTI Repository | ZENODO | CNR IRIS | CNR IRIS
2023
Other
Open Access
CNR activity in the ESA Extension project
Vairo C, Bolettieri P, Gennaro C, Amato G
The CNR activity within the ESA "EXTENSION" project aims to develop an advanced visual recognition system for cultural heritage objects in L'Aquila, using AI techniques such as classifiers. However, this task requires substantial computational resources due to the large amount of data and deep learning-based AI techniques involved. To overcome these challenges, a centralized approach has been adopted, with a central server providing the necessary computational power and storage capacity.DOI: https://doi.org/10.32079/isti-tr-2023/010
DOI: 10.32079/isti-tr-2023/010
Metrics:
Vairo C, Bolettieri P, Gennaro C, Amato G
The CNR activity within the ESA "EXTENSION" project aims to develop an advanced visual recognition system for cultural heritage objects in L'Aquila, using AI techniques such as classifiers. However, this task requires substantial computational resources due to the large amount of data and deep learning-based AI techniques involved. To overcome these challenges, a centralized approach has been adopted, with a central server providing the necessary computational power and storage capacity.DOI: https://doi.org/10.32079/isti-tr-2023/010
DOI: 10.32079/isti-tr-2023/010
Metrics:
See at:
CNR IRIS | ISTI Repository | CNR IRIS
2023
Conference article
Open Access
AIMH Lab 2022 activities for Vision
Ciampi L, Amato G, Bolettieri P, Carrara F, Di Benedetto M, Falchi F, Gennaro C, Messina N, Vadicamo L, Vairo C
The explosion of smartphones and cameras has led to a vast production of multimedia data. Consequently, Artificial Intelligence-based tools for automatically understanding and exploring these data have recently gained much attention. In this short paper, we report some activities of the Artificial Intelligence for Media and Humanities (AIMH) laboratory of the ISTI-CNR, tackling some challenges in the field of Computer Vision for the automatic understanding of visual data and for novel interactive tools aimed at multimedia data exploration. Specifically, we provide innovative solutions based on Deep Learning techniques carrying out typical vision tasks such as object detection and visual counting, with particular emphasis on scenarios characterized by scarcity of labeled data needed for the supervised training and on environments with limited power resources imposing miniaturization of the models. Furthermore, we describe VISIONE, our large-scale video search system designed to search extensive multimedia databases in an interactive and user-friendly manner.Source: CEUR WORKSHOP PROCEEDINGS, pp. 538-543. Pisa, Italy, 29-31/05/2023
Project(s): AI4Media, Future Artificial Intelligence Research
Ciampi L, Amato G, Bolettieri P, Carrara F, Di Benedetto M, Falchi F, Gennaro C, Messina N, Vadicamo L, Vairo C
The explosion of smartphones and cameras has led to a vast production of multimedia data. Consequently, Artificial Intelligence-based tools for automatically understanding and exploring these data have recently gained much attention. In this short paper, we report some activities of the Artificial Intelligence for Media and Humanities (AIMH) laboratory of the ISTI-CNR, tackling some challenges in the field of Computer Vision for the automatic understanding of visual data and for novel interactive tools aimed at multimedia data exploration. Specifically, we provide innovative solutions based on Deep Learning techniques carrying out typical vision tasks such as object detection and visual counting, with particular emphasis on scenarios characterized by scarcity of labeled data needed for the supervised training and on environments with limited power resources imposing miniaturization of the models. Furthermore, we describe VISIONE, our large-scale video search system designed to search extensive multimedia databases in an interactive and user-friendly manner.Source: CEUR WORKSHOP PROCEEDINGS, pp. 538-543. Pisa, Italy, 29-31/05/2023
Project(s): AI4Media
, Future Artificial Intelligence Research
See at:
ceur-ws.org | CNR IRIS | ISTI Repository | CNR IRIS
2023
Other
Open Access
AIMH Research Activities 2023
Aloia N., Amato G., Bartalesi Lenzi V., Bianchi L., Bolettieri P., Bosio C., Carraglia M., Carrara F., Casarosa V., Ciampi L., Coccomini D. A., Concordia C., Corbara S., De Martino C., Di Benedetto M., Esuli A., Falchi F., Fazzari E., Gennaro C., Lagani G., Lenzi E., Meghini C., Messina N., Molinari A., Moreo Fernandez A., Nardi A., Pedrotti A., Pratelli N., Puccetti G., Rabitti F., Savino P., Sebastiani F., Sperduti G., Thanos C., Trupiano L., Vadicamo L., Vairo C., Versienti L.
The AIMH (Artificial Intelligence for Media and Humanities) laboratory is dedicated to exploring and pushing the boundaries in the field of Artificial Intelligence, with a particular focus on its application in digital media and humanities. This lab's objective is to enhance the current state of AI technology particularly on deep learning, text analysis, computer vision, multimedia information retrieval, multimedia content analysis, recognition, and retrieval. This report encapsulates the laboratory's progress and activities throughout the year 2023.DOI: https://doi.org/10.32079/isti-ar-2023/001
DOI: 10.32079/isti-ar-2023/001
Metrics:
Aloia N., Amato G., Bartalesi Lenzi V., Bianchi L., Bolettieri P., Bosio C., Carraglia M., Carrara F., Casarosa V., Ciampi L., Coccomini D. A., Concordia C., Corbara S., De Martino C., Di Benedetto M., Esuli A., Falchi F., Fazzari E., Gennaro C., Lagani G., Lenzi E., Meghini C., Messina N., Molinari A., Moreo Fernandez A., Nardi A., Pedrotti A., Pratelli N., Puccetti G., Rabitti F., Savino P., Sebastiani F., Sperduti G., Thanos C., Trupiano L., Vadicamo L., Vairo C., Versienti L.
The AIMH (Artificial Intelligence for Media and Humanities) laboratory is dedicated to exploring and pushing the boundaries in the field of Artificial Intelligence, with a particular focus on its application in digital media and humanities. This lab's objective is to enhance the current state of AI technology particularly on deep learning, text analysis, computer vision, multimedia information retrieval, multimedia content analysis, recognition, and retrieval. This report encapsulates the laboratory's progress and activities throughout the year 2023.DOI: https://doi.org/10.32079/isti-ar-2023/001
DOI: 10.32079/isti-ar-2023/001
Metrics:
See at:
CNR IRIS | ISTI Repository | CNR IRIS
2023
Conference article
Open Access
VISIONE for newbies: an easier-to-use video retrieval system
Amato G, Bolettieri P, Carrara F, Falchi F, Gennaro C, Messina N, Vadicamo L, Vairo C
This paper presents a revised version of the VISIONE video retrieval system, which offers a wide range of search functionalities, including free text search, spatial color and object search, visual and semantic similarity search, and temporal search. The system is designed to ensure scalability using advanced indexing techniques and effectiveness using cutting-edge Artificial Intelligence technology for visual content analysis. VISIONE was the runner-up in the 2023 Video Browser Showdown competition, demonstrating its comprehensive video retrieval capabilities. In this paper, we detail the improvements made to the search and browsing interface to enhance its usability for non-expert users. A demonstration video of our system with the restyled interface, showcasing its capabilities on over 2,300 hours of diverse video content, is available online at https://youtu.be/srD3TCUkMSg.DOI: 10.1145/3617233.3617261
DOI: https://doi.org/10.1145/3617233.3617261
Project(s): AI4Media
Metrics:
Amato G, Bolettieri P, Carrara F, Falchi F, Gennaro C, Messina N, Vadicamo L, Vairo C
This paper presents a revised version of the VISIONE video retrieval system, which offers a wide range of search functionalities, including free text search, spatial color and object search, visual and semantic similarity search, and temporal search. The system is designed to ensure scalability using advanced indexing techniques and effectiveness using cutting-edge Artificial Intelligence technology for visual content analysis. VISIONE was the runner-up in the 2023 Video Browser Showdown competition, demonstrating its comprehensive video retrieval capabilities. In this paper, we detail the improvements made to the search and browsing interface to enhance its usability for non-expert users. A demonstration video of our system with the restyled interface, showcasing its capabilities on over 2,300 hours of diverse video content, is available online at https://youtu.be/srD3TCUkMSg.DOI: 10.1145/3617233.3617261
DOI: https://doi.org/10.1145/3617233.3617261
Project(s): AI4Media
Metrics:
See at:
ISTI Repository | CNR IRIS | CNR IRIS
2023
Other
Open Access
D3.2.1: AA@THE User needs, technical requirements and specifications
Lorenza Pratali, Franca Delmastro, Mattia Campana, Flavio Di Martino, Loreto Pescosolido, Paolo Barsocchi, Giovanna Broccia, Vincenzo Ciancia, Claudio Gennaro, Michele Girolami, Gabriele Lagani, Diego Latella, Massimo Magrini, Marco Manca, Mieke Massink, Andrea Mattioli, Davide Moroni, Filippo Palumbo, Paolo Paradisi, Fabio Paternò, Laura Sebastiani, Claudio Vairo, Carmelina Santoro, Davide La Rosa
The objective of this deliverable is to compile a comprehensive report that describes the user needs, requirements, and technical specifications necessary to successfully implement the pilot study. To achieve this, it is crucial to establish contacts with specific associations and medical experts, which, collaboratively, will help to establish exclusion and inclusion criteria for the target population of healthy adults. Furthermore, another related objective is to define the different categories of users that will interact with the system and their specific needs. This holistic approach will ensure that the system is designed and developed to satisfy the diverse needs of the users and be aligned with the goals of the project. To achieve the milestone M3.2.1, we made significant progresses in the definition of the pilot study for the AA@THE subproject. One of our key achievements is the successful description of users’ needs, requirements, and technical specifications necessary for the study. We worked closely with both a specialized association of personal trainers for Adapted Physical Activity (APA) for older adults, already active in the area of Pisa, and the medical partner who played a crucial role in providing valuable insights and expertise to establish exclusion and inclusion criteria for the target population of healthy adults. In this milestone, we also defined the activities and services that we intend to offer. Specifically, we plan to provide technological systems aimed at monitoring physical and cognitive training processes, as well as stability evaluations, by instrumenting a gym dedicated to active and healthy ageing, which is located within the CNR research area in Pisa. Additionally, we will conduct sleep, nutrition, and sedentary assessments at the volunteers' homes. Furthermore, we successfully defined the different user categories involved in the study. To facilitate the recruitment process and people engagement, on January 17th 2023, we organized an open day in collaboration with the gym association where we presented the overall objectives of the project and we collected feedbacks from a group of healthy adults over 65, already involved in APA training. This allowed us to gain a comprehensive understanding of the users' specific needs in terms of system interactions, thus establishing the system requirements and technical specifications of the AA@THE ecosystem. In parallel, a specific action on “Automatic Support of Medical Image Analysis” has been initiated by members of the “Formal Methods and Tools” group at CNR-ISTI. Such an action aims at leveraging Formal Methods in Computer Science, Logic and Model Checking to augment state-of-the-art machine learning techniques for automatic medical image analysis, enabling end-users to make specific assumptions on the level of accountability and affordability of the system. The methodology is based on a strict intertwining between theory and experimentation, with the development of new theoretical foundations for model reduction and efficient model checking, and experimentation and finalization of a graphical user interface that is being evaluated from the points of view of usability and of cognitive load. Moreover, the design and implementation of a suitable GUI able to support the analysis of medical images has been conducted and tested with small groups of people derived from the hospital in Lucca. The proposed GUI prototype has been evaluated from a cognitive point of view in order to allow easy employment with little training, for general practitioners and caregivers who may lack the technical skills required to use fully-fledged medical imaging programs.Project(s): Tuscany Health Ecosystem
Lorenza Pratali, Franca Delmastro, Mattia Campana, Flavio Di Martino, Loreto Pescosolido, Paolo Barsocchi, Giovanna Broccia, Vincenzo Ciancia, Claudio Gennaro, Michele Girolami, Gabriele Lagani, Diego Latella, Massimo Magrini, Marco Manca, Mieke Massink, Andrea Mattioli, Davide Moroni, Filippo Palumbo, Paolo Paradisi, Fabio Paternò, Laura Sebastiani, Claudio Vairo, Carmelina Santoro, Davide La Rosa
The objective of this deliverable is to compile a comprehensive report that describes the user needs, requirements, and technical specifications necessary to successfully implement the pilot study. To achieve this, it is crucial to establish contacts with specific associations and medical experts, which, collaboratively, will help to establish exclusion and inclusion criteria for the target population of healthy adults. Furthermore, another related objective is to define the different categories of users that will interact with the system and their specific needs. This holistic approach will ensure that the system is designed and developed to satisfy the diverse needs of the users and be aligned with the goals of the project. To achieve the milestone M3.2.1, we made significant progresses in the definition of the pilot study for the AA@THE subproject. One of our key achievements is the successful description of users’ needs, requirements, and technical specifications necessary for the study. We worked closely with both a specialized association of personal trainers for Adapted Physical Activity (APA) for older adults, already active in the area of Pisa, and the medical partner who played a crucial role in providing valuable insights and expertise to establish exclusion and inclusion criteria for the target population of healthy adults. In this milestone, we also defined the activities and services that we intend to offer. Specifically, we plan to provide technological systems aimed at monitoring physical and cognitive training processes, as well as stability evaluations, by instrumenting a gym dedicated to active and healthy ageing, which is located within the CNR research area in Pisa. Additionally, we will conduct sleep, nutrition, and sedentary assessments at the volunteers' homes. Furthermore, we successfully defined the different user categories involved in the study. To facilitate the recruitment process and people engagement, on January 17th 2023, we organized an open day in collaboration with the gym association where we presented the overall objectives of the project and we collected feedbacks from a group of healthy adults over 65, already involved in APA training. This allowed us to gain a comprehensive understanding of the users' specific needs in terms of system interactions, thus establishing the system requirements and technical specifications of the AA@THE ecosystem. In parallel, a specific action on “Automatic Support of Medical Image Analysis” has been initiated by members of the “Formal Methods and Tools” group at CNR-ISTI. Such an action aims at leveraging Formal Methods in Computer Science, Logic and Model Checking to augment state-of-the-art machine learning techniques for automatic medical image analysis, enabling end-users to make specific assumptions on the level of accountability and affordability of the system. The methodology is based on a strict intertwining between theory and experimentation, with the development of new theoretical foundations for model reduction and efficient model checking, and experimentation and finalization of a graphical user interface that is being evaluated from the points of view of usability and of cognitive load. Moreover, the design and implementation of a suitable GUI able to support the analysis of medical images has been conducted and tested with small groups of people derived from the hospital in Lucca. The proposed GUI prototype has been evaluated from a cognitive point of view in order to allow easy employment with little training, for general practitioners and caregivers who may lack the technical skills required to use fully-fledged medical imaging programs.Project(s): Tuscany Health Ecosystem
2022
Conference article
Open Access
AIMH Lab for Cybersecurity
Vairo C, Coccomini Da, Falchi F, Gennaro C, Massoli Fv, Messina N, Amato G
In this short paper, we report the activities of the Artificial Intelligence for Media and Humanities (AIMH) laboratory of the ISTI-CNR related to Cy-bersecurity. We discuss about our active research fields, their applications and challenges. We focus on face recognition and detection of adversarial examples and deep fakes. We also present our activities on the detection of persuasion techniques combining image and text analysis.
Vairo C, Coccomini Da, Falchi F, Gennaro C, Massoli Fv, Messina N, Amato G
In this short paper, we report the activities of the Artificial Intelligence for Media and Humanities (AIMH) laboratory of the ISTI-CNR related to Cy-bersecurity. We discuss about our active research fields, their applications and challenges. We focus on face recognition and detection of adversarial examples and deep fakes. We also present our activities on the detection of persuasion techniques combining image and text analysis.
See at:
CNR IRIS | ISTI Repository | www.ital-ia2022.it | CNR IRIS
2022
Journal article
Open Access
Multi-camera vehicle counting using edge-AI
Ciampi L, Gennaro C, Carrara F, Falchi F, Vairo C, Amato G
This paper presents a novel solution to automatically count vehicles in a parking lot using images captured by smart cameras. Unlike most of the literature on this task, which focuses on the analysis of single images, this paper proposes the use of multiple visual sources to monitor a wider parking area from different perspectives. The proposed multi-camera system is capable of automatically estimating the number of cars present in the entire parking lot directly on board the edge devices. It comprises an on-device deep learning-based detector that locates and counts the vehicles from the captured images and a decentralized geometric-based approach that can analyze the inter-camera shared areas and merge the data acquired by all the devices. We conducted the experimental evaluation on an extended version of the CNRPark-EXT dataset, a collection of images taken from the parking lot on the campus of the National Research Council (CNR) in Pisa, Italy. We show that our system is robust and takes advantage of the redundant information deriving from the different cameras, improving the overall performance without requiring any extra geometrical information of the monitored scene.Source: EXPERT SYSTEMS WITH APPLICATIONS
DOI: 10.1016/j.eswa.2022.117929
Project(s): AI4EU, AI4Media
Metrics:
Ciampi L, Gennaro C, Carrara F, Falchi F, Vairo C, Amato G
This paper presents a novel solution to automatically count vehicles in a parking lot using images captured by smart cameras. Unlike most of the literature on this task, which focuses on the analysis of single images, this paper proposes the use of multiple visual sources to monitor a wider parking area from different perspectives. The proposed multi-camera system is capable of automatically estimating the number of cars present in the entire parking lot directly on board the edge devices. It comprises an on-device deep learning-based detector that locates and counts the vehicles from the captured images and a decentralized geometric-based approach that can analyze the inter-camera shared areas and merge the data acquired by all the devices. We conducted the experimental evaluation on an extended version of the CNRPark-EXT dataset, a collection of images taken from the parking lot on the campus of the National Research Council (CNR) in Pisa, Italy. We show that our system is robust and takes advantage of the redundant information deriving from the different cameras, improving the overall performance without requiring any extra geometrical information of the monitored scene.Source: EXPERT SYSTEMS WITH APPLICATIONS
DOI: 10.1016/j.eswa.2022.117929
Project(s): AI4EU
, AI4Media Metrics:
See at:
CNR IRIS | ISTI Repository | www.sciencedirect.com | CNR IRIS | CNR IRIS | CNR IRIS