2004
Contribution to conference
Unknown
Scheduling and load balancing
Luque E., Castaños J. G., Markatos E. P., Perego R.
Scheduling and Load Balancing techniques are key issues for the performance of applications executed in parallel and distributed environments, and for the efficient utilization of these computational resources. Research in this field has a long history and is well consolidated. Nevertheless, the evolution of parallel and distributed systems toward clusters, computational grids, and global computing environments, introduces new challenging problems that require a new generation of scheduling and load balancing algorithms. Topic 3 in Euro-Par 2004 covers all aspects related to scheduling and load balancing from application and system levels, to theoretical foundations and practical tools. All these aspects are addressed by contributed papers.Source: Euro-Par 2010 - Parallel Processing, 16th International Euro-Par Conference, pp. 220–221, 2004
Luque E., Castaños J. G., Markatos E. P., Perego R.
Scheduling and Load Balancing techniques are key issues for the performance of applications executed in parallel and distributed environments, and for the efficient utilization of these computational resources. Research in this field has a long history and is well consolidated. Nevertheless, the evolution of parallel and distributed systems toward clusters, computational grids, and global computing environments, introduces new challenging problems that require a new generation of scheduling and load balancing algorithms. Topic 3 in Euro-Par 2004 covers all aspects related to scheduling and load balancing from application and system levels, to theoretical foundations and practical tools. All these aspects are addressed by contributed papers.Source: Euro-Par 2010 - Parallel Processing, 16th International Euro-Par Conference, pp. 220–221, 2004
See at: CNR ExploRA
2010
Contribution to conference
Open Access
Scheduling and load balancing. Introduction to topic
Yahyapour R., Perego R., Desprez F., Epstein L., Bernat F. G.
Scheduling and load balancing techniques are crucial for implementing efficient parallel and distributed applications and for making best use of parallel and distributed systems. This includes planning and optimization of resource allocation as well as coping with the dynamics of the systems.Source: Euro-Par 2010 - Parallel Processing - 16th International Euro-Par Conference, pp. 156–156, Ischia, Italy, August 31 - September 3 2010
DOI: 10.1007/978-3-642-15277-1_15
Metrics:
Yahyapour R., Perego R., Desprez F., Epstein L., Bernat F. G.
Scheduling and load balancing techniques are crucial for implementing efficient parallel and distributed applications and for making best use of parallel and distributed systems. This includes planning and optimization of resource allocation as well as coping with the dynamics of the systems.Source: Euro-Par 2010 - Parallel Processing - 16th International Euro-Par Conference, pp. 156–156, Ischia, Italy, August 31 - September 3 2010
DOI: 10.1007/978-3-642-15277-1_15
Metrics:
See at:
link.springer.com 
| doi.org 
| link.springer.com | CNR ExploRA
2006
Conference article
Restricted
Mining frequent closed itemsets out-of-core
Lucchese C., Orlando S., Perego R.
Extracting frequent itemsets is an important task in many data mining applications. When data are very large, it becomes mandatory to perform the mining task by using an external memory algorithm, but only a few of these algorithms have been proposed so far. Since also the result set of all the frequent itemsets is likely to be undesirably large, condensed representations, such as closed itemsets, have recently gained a lot of attention. In this paper we discuss the limitations of the partitioning techniques adopted by external memory algorithms for extracting all the frequent itemsets, when applied to closed itemsets mining. The main issue is that the closedness of an itemset cannot be evaluated only using the local knowledge available in a single partition of the input dataset. A further step is thus needed to correctly merge the partial results. We introduce the first algorithm for mining closed itemsets out of core. The algorithm exploits a divide-et-impera approach, where the input dataset is split into smaller partitions, such that not only they can be loaded, but also they can be mined entirely into the main memory. Moreover, we devised a simple technique based on a new theoretical result that allows us to reduce the problem of merging partial solutions to an external memory sorting problem.Source: SIAM International Conference on Data Mining, pp. 417–427, Bethesda, Maryland, 20-22/04/2006
Lucchese C., Orlando S., Perego R.
Extracting frequent itemsets is an important task in many data mining applications. When data are very large, it becomes mandatory to perform the mining task by using an external memory algorithm, but only a few of these algorithms have been proposed so far. Since also the result set of all the frequent itemsets is likely to be undesirably large, condensed representations, such as closed itemsets, have recently gained a lot of attention. In this paper we discuss the limitations of the partitioning techniques adopted by external memory algorithms for extracting all the frequent itemsets, when applied to closed itemsets mining. The main issue is that the closedness of an itemset cannot be evaluated only using the local knowledge available in a single partition of the input dataset. A further step is thus needed to correctly merge the partial results. We introduce the first algorithm for mining closed itemsets out of core. The algorithm exploits a divide-et-impera approach, where the input dataset is split into smaller partitions, such that not only they can be loaded, but also they can be mined entirely into the main memory. Moreover, we devised a simple technique based on a new theoretical result that allows us to reduce the problem of merging partial solutions to an external memory sorting problem.Source: SIAM International Conference on Data Mining, pp. 417–427, Bethesda, Maryland, 20-22/04/2006
See at:
www.siam.org | CNR ExploRA
2017
Contribution to conference
Open Access
Proceedings of the 8th Italian Information Retrieval Workshop
Crestani F., Di Noia T., Perego R.
This volume contains the papers presented at IIR'17: 8th Italian Information Retrieval Workshop held on June 05-07, 2017 in Lugano, Switzerland. The purpose of the Italian Information Retrieval (IIR) workshop series is to provide a forum for stimulating and disseminating research in information retrieval, where Italian researchers (especially young ones) and researchers a liated with Italian institutions can network and discuss their research results in an informal way. Previously IIR workshops took place in Venice (2016), Cagliari (2015), Rome (2014), Pisa (2013), Bari (2012), Milan (2011) and Padua (2010).
Crestani F., Di Noia T., Perego R.
This volume contains the papers presented at IIR'17: 8th Italian Information Retrieval Workshop held on June 05-07, 2017 in Lugano, Switzerland. The purpose of the Italian Information Retrieval (IIR) workshop series is to provide a forum for stimulating and disseminating research in information retrieval, where Italian researchers (especially young ones) and researchers a liated with Italian institutions can network and discuss their research results in an informal way. Previously IIR workshops took place in Venice (2016), Cagliari (2015), Rome (2014), Pisa (2013), Bari (2012), Milan (2011) and Padua (2010).
See at:
ceur-ws.org | ISTI Repository | CNR ExploRA
1994
Report
Unknown
Ottimizzazione parallela con simulated annealing e calcolo speculativo
Perego R.
Simulated Annealing (SA) is a Monte Carlo search technique for obtaining approximate solutions to combinatorial problems depending from independent variables with many degrees of freedom. In this paper a new parallel SA algorithm which exploits the technique of speculative computation is presented. The proposal sensibly shortens the high computational costs of traditional SA implementations without violating its serial decision sequence. Our implementation is similar to that proposed by Witte, Chamberlain e Franklin [9], althoug characterized by a higher flexibility and by an enhanced dynamic behavior which allows the efficient exploitation of more parallelism. Results achieved on a nCUBE 2 multicomputer running our parallel SA algorithm applied to a VLSI placement and routing problem are reported and analyzed.Source: ISTI Technical reports, pp.1–18, 1994
Perego R.
Simulated Annealing (SA) is a Monte Carlo search technique for obtaining approximate solutions to combinatorial problems depending from independent variables with many degrees of freedom. In this paper a new parallel SA algorithm which exploits the technique of speculative computation is presented. The proposal sensibly shortens the high computational costs of traditional SA implementations without violating its serial decision sequence. Our implementation is similar to that proposed by Witte, Chamberlain e Franklin [9], althoug characterized by a higher flexibility and by an enhanced dynamic behavior which allows the efficient exploitation of more parallelism. Results achieved on a nCUBE 2 multicomputer running our parallel SA algorithm applied to a VLSI placement and routing problem are reported and analyzed.Source: ISTI Technical reports, pp.1–18, 1994
See at: CNR ExploRA
1994
Report
Unknown
Ottimizzazione parallela con simulated annealing e calcolo speculativo
Perego R.
Simulated Annealing (SA) is a Monte Carlo search technique for obtaining approximate solutions to combinatorial problems depending from independent variables with many degrees of freedom. In this paper a new parallel SA algorithm which exploits the technique of speculative computation is presented. The proposal sensibly shortens the high computational costs of traditional SA implementations without violating its serial decision sequence. Our implementation is similar to that proposed by Witte, Chamberlain e Franklin [9], althoug characterized by a higher flexibility and by an enhanced dynamic behavior which allows the efficient exploitation of more parallelism. Results achieved on a nCUBE 2 multicomputer running our parallel SA algorithm applied to a VLSI placement and routing problem are reported and analyzed.Source: ISTI Technical reports, pp.1–18, 1994
Perego R.
Simulated Annealing (SA) is a Monte Carlo search technique for obtaining approximate solutions to combinatorial problems depending from independent variables with many degrees of freedom. In this paper a new parallel SA algorithm which exploits the technique of speculative computation is presented. The proposal sensibly shortens the high computational costs of traditional SA implementations without violating its serial decision sequence. Our implementation is similar to that proposed by Witte, Chamberlain e Franklin [9], althoug characterized by a higher flexibility and by an enhanced dynamic behavior which allows the efficient exploitation of more parallelism. Results achieved on a nCUBE 2 multicomputer running our parallel SA algorithm applied to a VLSI placement and routing problem are reported and analyzed.Source: ISTI Technical reports, pp.1–18, 1994
See at: CNR ExploRA
1994
Report
Unknown
A new mapping heuristic for last generation multicomputers
Perego R., De Petris G.
An abstract is not available.Source: ISTI Technical reports, pp.4–30, 1994
Perego R., De Petris G.
An abstract is not available.Source: ISTI Technical reports, pp.4–30, 1994
See at: CNR ExploRA
1994
Report
Unknown
Locality-based programming on a reconfigurable transputer network
Perego R.
An abstract is not available.Source: ISTI Technical reports, 1994
Perego R.
An abstract is not available.Source: ISTI Technical reports, 1994
See at: CNR ExploRA
2018
Conference article
Open Access
Continuation methods and curriculum learning for learning to rank
Ferro N., Lucchese C., Maistro M., Perego R.
In this paper we explore the use of Continuation Methods and Curriculum Learning techniques in the area of Learning to Rank. The basic idea is to design the training process as a learning path across increasingly complex training instances and objective functions. We propose to instantiate continuation methods in Learning to Rank by changing the IR measure to optimize during training, and we present two different curriculum learning strategies to identify easy training examples. Experimental results show that simple continuation methods are more promising than curriculum learning ones since they allow for slightly improving the performance of state-of-the-art ?-MART models and provide a faster convergence speed.Source: ACM CIKM, pp. 1523–1526, Torino, 22-26/10/2018
DOI: 10.1145/3269206.3269239
Metrics:
Ferro N., Lucchese C., Maistro M., Perego R.
In this paper we explore the use of Continuation Methods and Curriculum Learning techniques in the area of Learning to Rank. The basic idea is to design the training process as a learning path across increasingly complex training instances and objective functions. We propose to instantiate continuation methods in Learning to Rank by changing the IR measure to optimize during training, and we present two different curriculum learning strategies to identify easy training examples. Experimental results show that simple continuation methods are more promising than curriculum learning ones since they allow for slightly improving the performance of state-of-the-art ?-MART models and provide a faster convergence speed.Source: ACM CIKM, pp. 1523–1526, Torino, 22-26/10/2018
DOI: 10.1145/3269206.3269239
Metrics:
See at:
ISTI Repository | dl.acm.org | doi.org | CNR ExploRA
2019
Journal article
Open Access
Boosting learning to rank with user dynamics and continuation methods
Ferro N., Lucchese C., Maistro M., Perego R.
Learning to rank (LtR) techniques leverage assessed samples of query-document relevance to learn effective ranking functions able to exploit the noisy signals hidden in the features used to represent queries and documents. In this paper we explore how to enhance the state-of-the-art LambdaMart LtR algorithm by integrating in the training process an explicit knowledge of the underlying user-interaction model and the possibility of targeting different objective functions that can effectively drive the algorithm towards promising areas of the search space. We enrich the iterative process followed by the learning algorithm in two ways: (1) by considering complex query-based user dynamics instead than simply discounting the gain by the rank position; (2) by designing a learning path across different loss functions that can capture different signals in the training data. Our extensive experiments, conducted on publicly available datasets, show that the proposed solution permits to improve various ranking quality measures by statistically significant margins.Source: Information retrieval (Boston) 23 (2019): 528–554. doi:10.1007/s10791-019-09366-9
DOI: 10.1007/s10791-019-09366-9
Metrics:
Ferro N., Lucchese C., Maistro M., Perego R.
Learning to rank (LtR) techniques leverage assessed samples of query-document relevance to learn effective ranking functions able to exploit the noisy signals hidden in the features used to represent queries and documents. In this paper we explore how to enhance the state-of-the-art LambdaMart LtR algorithm by integrating in the training process an explicit knowledge of the underlying user-interaction model and the possibility of targeting different objective functions that can effectively drive the algorithm towards promising areas of the search space. We enrich the iterative process followed by the learning algorithm in two ways: (1) by considering complex query-based user dynamics instead than simply discounting the gain by the rank position; (2) by designing a learning path across different loss functions that can capture different signals in the training data. Our extensive experiments, conducted on publicly available datasets, show that the proposed solution permits to improve various ranking quality measures by statistically significant margins.Source: Information retrieval (Boston) 23 (2019): 528–554. doi:10.1007/s10791-019-09366-9
DOI: 10.1007/s10791-019-09366-9
Metrics:
See at:
Copenhagen University Research Information System | Archivio istituzionale della ricerca - Università degli Studi di Venezia Ca' Foscari | Information Retrieval | ISTI Repository | Information Retrieval | link.springer.com | CNR ExploRA
2018
Journal article
Open Access
From Evaluating to Forecasting Performance: How to Turn Information Retrieval, Natural Language Processing and Recommender Systems into Predictive Sciences (Dagstuhl Perspectives Workshop 17442)
Ferro N., Fuhr N., Grefenstette G., Konstan J. A., Castells P., Daly E. M., Declerck T., Ekstrand M. D., Geyer W., Gonzalo J., Kuflik T., Lind'En K., Magnini B., Nie J. Y., Perego R., Shapira B., Soboroff I., Tintarev N., Verspoor K., Willemsen M. C., Zobel J.
We describe the state-of-the-art in performance modeling and prediction for Information Retrieval (IR), Natural Language Processing (NLP) and Recommender Systems (RecSys) along with its shortcomings and strengths. We present a framework for further research, identifying five major problem areas: understanding measures, performance analysis, making underlying assumptions explicit, identifying application features determining performance, and the development of predic- tion models describing the relationship between assumptions, features and resulting performanceSource: Dagstuhl manifestos 7 (2018): 96–139. doi:10.4230/DagMan.7.1.96
DOI: 10.4230/dagman.7.1.96
Metrics:
Ferro N., Fuhr N., Grefenstette G., Konstan J. A., Castells P., Daly E. M., Declerck T., Ekstrand M. D., Geyer W., Gonzalo J., Kuflik T., Lind'En K., Magnini B., Nie J. Y., Perego R., Shapira B., Soboroff I., Tintarev N., Verspoor K., Willemsen M. C., Zobel J.
We describe the state-of-the-art in performance modeling and prediction for Information Retrieval (IR), Natural Language Processing (NLP) and Recommender Systems (RecSys) along with its shortcomings and strengths. We present a framework for further research, identifying five major problem areas: understanding measures, performance analysis, making underlying assumptions explicit, identifying application features determining performance, and the development of predic- tion models describing the relationship between assumptions, features and resulting performanceSource: Dagstuhl manifestos 7 (2018): 96–139. doi:10.4230/DagMan.7.1.96
DOI: 10.4230/dagman.7.1.96
Metrics:
See at:
drops.dagstuhl.de | ISTI Repository | CNR ExploRA
1998
Conference article
Closed Access
An MPI-based Run-Time Support to Coordinate HPF Tasks
Orlando S., Perego R.
This paper describes COLTHPF, an MPI-based run-time support for the coordination of concurrent and communicating HPF tasks. COLTHPF is conceived for use by a compiler of a high-level coordination language to structure a set of data-parallel HPF tasks according to popular forms of task-parallelism. Since it requires only small changes to the run-time support of the HPF compiler used, COLTHPF is easily portable among different compilation systems. The paper outlines design and implementation issues, and reports the results of experiments conducted on an SGI/Cray T3E.Source: Recent Advances in Parallel Virtual Machine and Message Passing Interface, 5th European PVM/MPI Users' Group Meeting, Liverpool, UK, September 7-9, 1998, Proceedings., pp. 289–296, Liverpool, UK, 7-9/09/1998
DOI: 10.1007/bfb0056587
Metrics:
Orlando S., Perego R.
This paper describes COLTHPF, an MPI-based run-time support for the coordination of concurrent and communicating HPF tasks. COLTHPF is conceived for use by a compiler of a high-level coordination language to structure a set of data-parallel HPF tasks according to popular forms of task-parallelism. Since it requires only small changes to the run-time support of the HPF compiler used, COLTHPF is easily portable among different compilation systems. The paper outlines design and implementation issues, and reports the results of experiments conducted on an SGI/Cray T3E.Source: Recent Advances in Parallel Virtual Machine and Message Passing Interface, 5th European PVM/MPI Users' Group Meeting, Liverpool, UK, September 7-9, 1998, Proceedings., pp. 289–296, Liverpool, UK, 7-9/09/1998
DOI: 10.1007/bfb0056587
Metrics:
See at:
doi.org | link.springer.com | CNR ExploRA
1998
Conference article
Closed Access
A Coordination Layer for Exploiting Task Parallelism with HPF
Orlando S., Perego R.
This paper introduces COLThpf, a run-time support for exploiting task parallelism within HPF programs, which can be employed by a compiler of a high-level coordination language to structure a set of data-parallel HPF tasks according to popular paradigms of task-parallelism. We use COLThpf to program a computer vision application and report the results obtained by running the application on an SGI/Cray T3E.Source: Languages, Compilers, and Run-Time Systems for Scalable Computers, pp. 386–393, Pittsburgh, USA, 28-30/05/1998
DOI: 10.1007/3-540-49530-4_30
Metrics:
Orlando S., Perego R.
This paper introduces COLThpf, a run-time support for exploiting task parallelism within HPF programs, which can be employed by a compiler of a high-level coordination language to structure a set of data-parallel HPF tasks according to popular paradigms of task-parallelism. We use COLThpf to program a computer vision application and report the results obtained by running the application on an SGI/Cray T3E.Source: Languages, Compilers, and Run-Time Systems for Scalable Computers, pp. 386–393, Pittsburgh, USA, 28-30/05/1998
DOI: 10.1007/3-540-49530-4_30
Metrics:
See at:
doi.org | link.springer.com | CNR ExploRA
1998
Conference article
Open Access
Scheduling Data-Parallel Computations on Heterogeneous and Time-Shared Environments
Orlando S., Perego R.
This paper addresses the problem of load balancing data-parallel computations on heterogeneous and time-shared parallel computing environments, where load imbalance may be introduced by the different capacities of processors populating a computer, or by the sharing of the same computational resources among several users. To solve this problem we propose a run-time support for parallel loops based upon a hybrid (static + dynamic) scheduling strategy. The main features of our technique are the absence of centralization and synchronization points, the prefetching of work toward slower processors, and the overlapping of communication latencies with useful computation.Source: Euro-Par '98, 4th International Euro-Par Conference, pp. 356–366, Southampton, UK, 1-4/09/1998
DOI: 10.1007/bfb0057874
Metrics:
Orlando S., Perego R.
This paper addresses the problem of load balancing data-parallel computations on heterogeneous and time-shared parallel computing environments, where load imbalance may be introduced by the different capacities of processors populating a computer, or by the sharing of the same computational resources among several users. To solve this problem we propose a run-time support for parallel loops based upon a hybrid (static + dynamic) scheduling strategy. The main features of our technique are the absence of centralization and synchronization points, the prefetching of work toward slower processors, and the overlapping of communication latencies with useful computation.Source: Euro-Par '98, 4th International Euro-Par Conference, pp. 356–366, Southampton, UK, 1-4/09/1998
DOI: 10.1007/bfb0057874
Metrics:
See at:
link.springer.com | doi.org | link.springer.com | CNR ExploRA
1998
Journal article
Closed Access
A mapping heuristic for minimizing network contention
Raffaele Perego
The combinatorial optimization problem of assigning tasks of a parallel program to processing nodes (pn's) of a parallel system is a well-known NP-hard problem. In this paper a new greedy heuristic for compile-time mapping of tasks without precedence constraints is proposed. The solution is addressed to modern multicomputers based on k-ary n-cube direct interconnection networks exploiting the e-cube routing algorithm and the wormhole flow control strategy. The proposed algorithm takes into account communication delays due to network blocking of colliding messages. Results achieved on several program-derived graphs with up to 784 tasks demonstrate the effectiveness of the approach followed.Source: Journal of systems architecture 45 (1998): 65–82. doi:10.1016/S1383-7621(97)00073-8
DOI: 10.1016/s1383-7621(97)00073-8
Metrics:
Raffaele Perego
The combinatorial optimization problem of assigning tasks of a parallel program to processing nodes (pn's) of a parallel system is a well-known NP-hard problem. In this paper a new greedy heuristic for compile-time mapping of tasks without precedence constraints is proposed. The solution is addressed to modern multicomputers based on k-ary n-cube direct interconnection networks exploiting the e-cube routing algorithm and the wormhole flow control strategy. The proposed algorithm takes into account communication delays due to network blocking of colliding messages. Results achieved on several program-derived graphs with up to 784 tasks demonstrate the effectiveness of the approach followed.Source: Journal of systems architecture 45 (1998): 65–82. doi:10.1016/S1383-7621(97)00073-8
DOI: 10.1016/s1383-7621(97)00073-8
Metrics:
See at:
Journal of Systems Architecture | www.sciencedirect.com | www.scopus.com | CNR ExploRA
1992
Other
Unknown
OCTOPUS: architettura del sistema e modalita' di utilizzo
Perego R., Pedelini P.
Abstract non disponibile
Perego R., Pedelini P.
Abstract non disponibile
See at: CNR ExploRA
2021
Conference article
Unknown
Hierarchical dependence-aware evaluation measures for conversational search
Faggioli G., Ferrante M., Ferro N., Perego R., Tonellotto N.
Conversational agents are drawing a lot of attention in the information retrieval (IR) community also thanks to the advancements in language understanding enabled by large contextualized language models. IR researchers have long ago recognized the importance o fa sound evaluation of new approaches. Yet, the development of evaluation techniques for conversational search is still an underlooked problem. Currently, most evaluation approaches rely on procedures directly drawn from ad-hoc search evaluation, treating utterances in a conversation as independent events, as if they were just separate topics, instead of accounting for the conversation context. We overcome this issue by proposing a framework for defining evaluation measures that are aware of the conversation context and the utterance semantic dependencies. In particular, we model the conversations as Direct Acyclic Graphs (DAG), where self-explanatory utterances are root nodes, while anaphoric utterances are linked to sentences that contain their missing semantic information. Then,we propose a family of hierarchical dependence-aware aggregations of the evaluation metrics driven by the conversational graph. In our experiments, we show that utterances from the same conversation are 20% more correlated than utterances from different conversations. Thanks to the proposed framework, we are able to include such correlation in our aggregations, and be more accurate when determining which pairs of conversational systems are deemed significantly different.Source: SIGIR '21: 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1935–1939, Online Conference, 11-15/07/2021
DOI: 10.1145/3404835.3463090
Metrics:
Faggioli G., Ferrante M., Ferro N., Perego R., Tonellotto N.
Conversational agents are drawing a lot of attention in the information retrieval (IR) community also thanks to the advancements in language understanding enabled by large contextualized language models. IR researchers have long ago recognized the importance o fa sound evaluation of new approaches. Yet, the development of evaluation techniques for conversational search is still an underlooked problem. Currently, most evaluation approaches rely on procedures directly drawn from ad-hoc search evaluation, treating utterances in a conversation as independent events, as if they were just separate topics, instead of accounting for the conversation context. We overcome this issue by proposing a framework for defining evaluation measures that are aware of the conversation context and the utterance semantic dependencies. In particular, we model the conversations as Direct Acyclic Graphs (DAG), where self-explanatory utterances are root nodes, while anaphoric utterances are linked to sentences that contain their missing semantic information. Then,we propose a family of hierarchical dependence-aware aggregations of the evaluation metrics driven by the conversational graph. In our experiments, we show that utterances from the same conversation are 20% more correlated than utterances from different conversations. Thanks to the proposed framework, we are able to include such correlation in our aggregations, and be more accurate when determining which pairs of conversational systems are deemed significantly different.Source: SIGIR '21: 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1935–1939, Online Conference, 11-15/07/2021
DOI: 10.1145/3404835.3463090
Metrics:
See at:
dl.acm.org | CNR ExploRA
2022
Conference article
Open Access
A dependency-aware utterances permutation strategy to improve conversational evaluation
Faggioli G., Ferrante M., Ferro N., Perego R., Tonellotto N.
The rapid growth in the number and complexity of conversational agents has highlighted the need for suitable evaluation tools to describe their performance. The main evaluation paradigms move from analyzing conversations where the user explores information needs following a scripted dialogue with the agent. We argue that this is not a realistic setting: different users ask different questions (and in a diverse order), obtaining distinct answers and changing the conversation path. We analyze what happens to conversational systems performance when we change the order of the utterances in a scripted conversation while respecting temporal dependencies between them. Our results highlight that the performance of the system widely varies. Our experiments show that diverse orders of utterances determine completely different rankings of systems by performance. The current way of evaluating conversational systems is thus biased. Motivated by these observations, we propose a new evaluation approach based on dependency-aware utterance permutations to increase the power of our evaluation tools.Source: ECIR 2022 - 44th European Conference on IR Research, pp. 184–198, Stavanger, Norway, 10-14/04/2022
DOI: 10.1007/978-3-030-99736-6_13
Metrics:
Faggioli G., Ferrante M., Ferro N., Perego R., Tonellotto N.
The rapid growth in the number and complexity of conversational agents has highlighted the need for suitable evaluation tools to describe their performance. The main evaluation paradigms move from analyzing conversations where the user explores information needs following a scripted dialogue with the agent. We argue that this is not a realistic setting: different users ask different questions (and in a diverse order), obtaining distinct answers and changing the conversation path. We analyze what happens to conversational systems performance when we change the order of the utterances in a scripted conversation while respecting temporal dependencies between them. Our results highlight that the performance of the system widely varies. Our experiments show that diverse orders of utterances determine completely different rankings of systems by performance. The current way of evaluating conversational systems is thus biased. Motivated by these observations, we propose a new evaluation approach based on dependency-aware utterance permutations to increase the power of our evaluation tools.Source: ECIR 2022 - 44th European Conference on IR Research, pp. 184–198, Stavanger, Norway, 10-14/04/2022
DOI: 10.1007/978-3-030-99736-6_13
Metrics:
See at:
ISTI Repository | doi.org | link.springer.com | CNR ExploRA
2022
Conference article
Open Access
Improving conversational evaluation via a dependency-aware permutation strategy
Faggioli G., Ferrante M., Ferro N., Perego R., Tonellotto N.
The rapid growth in number and complexity of conversational agents has highlighted the need for suitable evaluation tools to describe their performance. Current offline conversational evaluation approaches rely on collections composed of multiturn conversations, each including a sequence of utterances. Such sequences represent a snapshot of reality: a single dialog between the user and a hypothetical system on a specific topic. We argue that this paradigm is not realistic enough: multiple users will ask diverse questions in variable order, even for a conversation on the same topic. In this work1 we propose a dependency-aware utterances sampling strategy to augment data available in conversational collections while maintaining temporal dependencies within conversations. Using the sampled conversations, we show that the current evaluation framework favours specific systems while penalizing others, leading to biased evaluation. We further show how to exploit dependency-aware utterances permutations in our current evaluation framework and increase the power of statistical evaluation tools such as ANOVA.Source: SEBD 2022 - Italian Symposium on Advanced Database Systems, Pisa, Italy, 19-22/07/2022
Faggioli G., Ferrante M., Ferro N., Perego R., Tonellotto N.
The rapid growth in number and complexity of conversational agents has highlighted the need for suitable evaluation tools to describe their performance. Current offline conversational evaluation approaches rely on collections composed of multiturn conversations, each including a sequence of utterances. Such sequences represent a snapshot of reality: a single dialog between the user and a hypothetical system on a specific topic. We argue that this paradigm is not realistic enough: multiple users will ask diverse questions in variable order, even for a conversation on the same topic. In this work1 we propose a dependency-aware utterances sampling strategy to augment data available in conversational collections while maintaining temporal dependencies within conversations. Using the sampled conversations, we show that the current evaluation framework favours specific systems while penalizing others, leading to biased evaluation. We further show how to exploit dependency-aware utterances permutations in our current evaluation framework and increase the power of statistical evaluation tools such as ANOVA.Source: SEBD 2022 - Italian Symposium on Advanced Database Systems, Pisa, Italy, 19-22/07/2022
See at:
ceur-ws.org | ISTI Repository | CNR ExploRA