2025 Conference article Open Access

Stress-testing machine generated text detection: shifting language models writing style to fool detectors
Pedrotti A., Papucci M., Ciaccio C., Miaschi A., Puccetti G., Dell'Orletta F., Esuli A.
Recent advancements in Generative AI and Large Language Models (LLMs) have enabled the creation of highly realistic synthetic content, raising concerns about the potential for malicious use, such as misinformation and manipulation. Moreover, detecting Machine-Generated Text (MGT) remains challenging due to the lack of robust benchmarks that assess generalization to real-world scenarios. In this work, we evaluate the resilience of state-of-the-art MGT detectors (e.g., Mage, Radar, LLM-DetectAIve) to linguistically informed adversarial attacks. We develop a pipeline that fine-tunes language models using Direct Preference Optimization (DPO) to shift the MGT style toward human-written text (HWT), obtaining generations more challenging to detect by current models. Additionally, we analyze the linguistic shifts induced by the alignment and how detectors rely on “linguistic shortcuts” to detect texts. Our results show that detectors can be easily fooled with relatively few examples, resulting in a significant drop in detecting performances. This highlights the importance of improving detection methods and making them robust to unseen in-domain texts. We release code, models, and data to support future research on more robust MGT detection benchmarks.DOI: 10.18653/v1/2025.findings-acl.156
Project(s): SoBigData
Metrics:

See at: aclanthology.org | CNR IRIS | CNR IRIS

2025 Other Open Access

See at: CNR IRIS | www.isti.cnr.it | CNR IRIS

2025 Conference article Open Access

How humans and LLMs organize conceptual knowledge: exploring subordinate categories in Italian
Pedrotti A., Rambelli G., Villani C., Bolognesi M.
People can categorize the same entity at multiple taxonomic levels, such as basic (bear), superordinate (animal), and subordinate (grizzly bear). While prior research has focused on basic-level categories, this study is the first attempt to examine the organization of categories by analyzing exemplars produced at the subordinate level. We present a new Italian psycholinguistic dataset of human-generated exemplars for 187 concrete words. We then leverage these data to evaluate whether textual and vision LLMs produce meaningful exemplars that align with human category organization across three key tasks: exemplar generation, category induction, and typicality judgment. Our findings show a low alignment between humans and LLMs, consistent with previous studies. However, their performance varies notably across different semantic domains. Ultimately, this study highlights both the promises and the constraints of using AI-generated exemplars to support psychological and linguistic research.DOI: 10.18653/v1/2025.acl-long.224
Project(s): SoBigData
Metrics:

See at: aclanthology.org | CNR IRIS | CNR IRIS

2024 Conference article Open Access

Multimodal heterogeneous transfer learning for multilingual image-text classification
Pedrotti A., Moreo Fernandez A., Sebastiani F.
The Multilingual Image-Text Classification (MITC) task is a specific instance of the Image-Text Classification (ITC) task, where each item to be classified consists of a visual representation and a textual description written in one of several possible languages. In this paper we propose MM-gFun, an extension of the gFun learning architecture originally developed for cross-lingual text classification. We extend its original text-only implementation to handle perceptual modalities.Source: CEUR WORKSHOP PROCEEDINGS, vol. 3928. Pisa, Italy, 14-16/10/2024
Project(s): SoBigData

See at: ceur-ws.org | CNR IRIS | CNR IRIS

2024 Other Open Access

See at: CNR IRIS | CNR IRIS

2024 Conference article Open Access

ViLMA: a zero-shot benchmark for linguistic and temporal grounding in video-language models
Kesen I., Pedrotti A., Dogan M., Cafagna M., Can Acikgoz E., Parcalabescu L., Calixto I., Frank A., Gatt A., Erdem A., Erdem E.
With the ever-increasing popularity of pretrained Video-Language Models (VidLMs), there is a pressing need to develop robust evaluation methodologies that delve deeper into their visio-linguistic capabilities. To address this challenge, we present ViLMA (Video Language Model Assessment), a task-agnostic benchmark that places the assessment of fine-grained capabilities of these models on a firm footing. Task-based evaluations, while valuable, fail to capture the complexities and specific temporal aspects of moving images that VidLMs need to process. Through carefully curated counterfactuals, ViLMA offers a controlled evaluation suite that sheds light on the true potential of these models, as well as their performance gaps compared to human-level understanding. ViLMA also includes proficiency tests, which assess basic capabilities deemed essential to solving the main counterfactual tests. We show that current VidLMs' grounding abilities are no better than those of vision-language models which use static images. This is especially striking once the performance on proficiency tests is factored in. Our benchmark serves as a catalyst for future research on VidLMs, helping to highlight areas that still need to be explored.Project(s): AI4Media

See at: CNR IRIS | openreview.net | CNR IRIS

2023 Other Open Access

See at: CNR IRIS | ISTI Repository | CNR IRIS

2022 Journal article Open Access

Generalized funnelling: ensemble learning and heterogeneous document embeddings for cross-lingual text classification
Moreo A, Pedrotti A, Sebastiani F
Funnelling (Fun) is a recently proposed method for cross-lingual text classification (CLTC) based on a two-tier learning ensemble for heterogeneous transfer learning (HTL). In this ensemble method, 1st-tier classifiers, each working on a different and language-dependent feature space, return a vector of calibrated posterior probabilities (with one dimension for each class) for each document, and the final classification decision is taken by a meta-classifier that uses this vector as its input. The meta-classifier can thus exploit class-class correlations, and this (among other things) gives Fun an edge over CLTC systems in which these correlations cannot be brought to bear. In this paper we describe Generalized Funnelling (gFun), a generalisation of Fun consisting of an HTL architecture in which 1st-tier components can be arbitrary view-generating functions, i.e., language-dependent functions that each produce a language-independent representation ("view") of the (monolingual) document. We describe an instance of gFun in which the meta-classifier receives as input a vector of calibrated posterior probabilities (as in Fun) aggregated to other embedded representations that embody other types of correlations, such as word-class correlations (as encoded by Word-Class Embeddings), word-word correlations (as encoded by Multilingual Unsupervised or Supervised Embeddings), and word-context correlations (as encoded by multilingual BERT ). We show that this instance of gFun substantially improves over Fun and over state-of-the-art baselines, by reporting experimental results obtained on two large, standard datasets for multilingual multilabel text classification. Our code that implements gFun is publicly available.Source: ACM TRANSACTIONS ON INFORMATION SYSTEMS, vol. 41 (issue 2)
DOI: 10.1145/3544104
Project(s): AI4Media , ARIADNEplus , SoBigData-PlusPlus
Metrics:

See at: dl.acm.org | CNR IRIS | ISTI Repository | CNR IRIS | CNR IRIS

2022 Other Open Access

See at: CNR IRIS | ISTI Repository | CNR IRIS

2021 Conference article Open Access

Heterogeneous document embeddings for cross-lingual text classification
Moreo A, Pedrotti A, Sebastiani F
Funnelling (Fun) is a method for cross-lingual text classification (CLC) based on a two-tier ensemble for heterogeneous transfer learning. In Fun, 1st-tier classifiers, each working on a different, language-dependent feature space, return a vector of calibrated posterior probabilities (with one dimension for each class) for each document, and the final classification decision is taken by a meta-classifier that uses this vector as its input. The metaclassifier can thus exploit class-class correlations, and this (among other things) gives Fun an edge over CLC systems where these correlations cannot be leveraged. We here describe Generalized Funnelling (gFun), a learning ensemble where the metaclassifier receives as input the above vector of calibrated posterior probabilities, concatenated with document embeddings (aligned across languages) that embody other types of correlations, such as word-class correlations (as encoded by Word-Class Embeddings) and word-word correlations (as encoded by Multilingual Unsupervised or Supervised Embeddings). We show that gFun improves on Fun by describing experiments on two large, standard multilingual datasets for multi-label text classification.DOI: 10.1145/3412841.3442093
Project(s): AI4Media , ARIADNEplus , SoBigData-PlusPlus
Metrics:

See at: dl.acm.org | CNR IRIS | ISTI Repository | ZENODO | dl.acm.org | CNR IRIS | CNR IRIS

2021 Conference article Open Access

See at: ceur-ws.org | ISTI Repository | CNR ExploRA

2021 Other Open Access

See at: ISTI Repository | CNR ExploRA

2020 Other Open Access

See at: CNR IRIS | ISTI Repository | CNR IRIS