2012
Book
Restricted
Multilingual information retrieval: from research to practice
Peters C, Braschler M, Clough P D
Peters, Braschler and Clough present a comprehensive description of the technologies involved in designing and developing systems for Multilingual Information Retrieval (MLIR). They provide readers with broad coverage of the various issues involved in creating systems to make accessible digitally stored materials regardless of the language(s) they are written in. Details on Cross-Language Information Retrieval (CLIR) are also covered that help readers to understand how to develop retrieval systems that cross language boundaries. Their work is divided into six chapters and accompanies the reader step-by-step through the various stages involved in building, using and evaluating MLIR systems. The book concludes with some examples of recent applications that utilise MLIR technologies. Some of the techniques described have recently started to appear in commercial search systems, while others have the potential to be part of future incarnations. The book is intended for graduate students, scholars, and practitioners with a basic understanding of classical text retrieval methods. It offers guidelines and information on all aspects that need to be taken into consideration when building MLIR systems, while avoiding too many 'hands-on details' that could rapidly become obsolete. Thus it bridges the gap between the material covered by most of the classical IR textbooks and the novel requirements related to the acquisition and dissemination of information in whatever language it is stored.DOI: 10.1007/978-3-642-23008-0
Metrics:
Peters C, Braschler M, Clough P D
Peters, Braschler and Clough present a comprehensive description of the technologies involved in designing and developing systems for Multilingual Information Retrieval (MLIR). They provide readers with broad coverage of the various issues involved in creating systems to make accessible digitally stored materials regardless of the language(s) they are written in. Details on Cross-Language Information Retrieval (CLIR) are also covered that help readers to understand how to develop retrieval systems that cross language boundaries. Their work is divided into six chapters and accompanies the reader step-by-step through the various stages involved in building, using and evaluating MLIR systems. The book concludes with some examples of recent applications that utilise MLIR technologies. Some of the techniques described have recently started to appear in commercial search systems, while others have the potential to be part of future incarnations. The book is intended for graduate students, scholars, and practitioners with a basic understanding of classical text retrieval methods. It offers guidelines and information on all aspects that need to be taken into consideration when building MLIR systems, while avoiding too many 'hands-on details' that could rapidly become obsolete. Thus it bridges the gap between the material covered by most of the classical IR textbooks and the novel requirements related to the acquisition and dissemination of information in whatever language it is stored.DOI: 10.1007/978-3-642-23008-0
Metrics:
See at:
CNR IRIS 
| link.springer.com | link.springer.com | www.springerlink.com
2010
Contribution to book
Restricted
What happened in CLEF 2009
Peters C
The organization of the CLEF 2009 evaluation campaign is described and details are provided concerning the tracks, test collections, evaluation infrastructure,and participation. The aim is to provide the reader of these proceedings with a complete picture of the entire campaign, covering both text and multimedia retrieval experiments. In the final section, the main results achieved by CLEF in the first ten years of activity are discussed and plans for the future of CLEF are presented.DOI: 10.1007/978-3-642-15754-7_1
DOI: 10.1007/978-3-642-15751-6_1
Metrics:
Peters C
The organization of the CLEF 2009 evaluation campaign is described and details are provided concerning the tracks, test collections, evaluation infrastructure,and participation. The aim is to provide the reader of these proceedings with a complete picture of the entire campaign, covering both text and multimedia retrieval experiments. In the final section, the main results achieved by CLEF in the first ten years of activity are discussed and plans for the future of CLEF are presented.DOI: 10.1007/978-3-642-15754-7_1
DOI: 10.1007/978-3-642-15751-6_1
Metrics:
See at:
doi.org | doi.org | CNR IRIS | CNR IRIS | www.springerlink.com
2010
Book
Restricted
Preface to CLEF 2010
Agosti M, Ferro N, Peters C, De Rijke M, Smeaton A
Editorial of the Proceedings of the CLEF 2010 ConferenceDOI: 10.1007/978-3-642-15998-5
Metrics:
Agosti M, Ferro N, Peters C, De Rijke M, Smeaton A
Editorial of the Proceedings of the CLEF 2010 ConferenceDOI: 10.1007/978-3-642-15998-5
Metrics:
See at:
NARCIS | NARCIS | CNR IRIS | CNR IRIS | www.springerlink.com
2010
Contribution to book
Restricted
CLEF 2009 ad hoc track overview: TEL and Persian Tasks
Peters C, Ferro N
The design of the 2009 Ad Hoc track was to a large extent a repetition of the previous year's track, with the same three tasks: Tel@CLEF, Persian@CLEF, and Robust-WSD. In this first of the two track overviews, we describe the objectives and results of the TEL and Persian tasks and provide some statistical analyses.DOI: 10.1007/978-3-642-15754-7_2
Metrics:
Peters C, Ferro N
The design of the 2009 Ad Hoc track was to a large extent a repetition of the previous year's track, with the same three tasks: Tel@CLEF, Persian@CLEF, and Robust-WSD. In this first of the two track overviews, we describe the objectives and results of the TEL and Persian tasks and provide some statistical analyses.DOI: 10.1007/978-3-642-15754-7_2
Metrics:
See at:
doi.org | CNR IRIS | CNR IRIS | www.springerlink.com
2010
Book
Open Access
Preface to Multilingual Information Access Evaluation I. Multimedia Experiments
Peters C, Caputo B, Gonzalo J, Jones G J, Kalpathycramer J, Mueller H, Tsikrika T
This book constitutes the thoroughly refereed proceedings of the 10th Workshop of the Cross Language Evaluation Forum, CLEF 2010, held in Corfu, Greece, in September/October 2009. The volume reports experiments on various types of multimedia collections. It is divided into three main sections presenting the results of the following tracks: Interactive Cross-Language Retrieval (iCLEF), Cross-Language Image Retrieval (ImageCLEF), and Cross-Language Video Retrieval (VideoCLEF).DOI: 10.1007/978-3-642-15751-6
Metrics:
Peters C, Caputo B, Gonzalo J, Jones G J, Kalpathycramer J, Mueller H, Tsikrika T
This book constitutes the thoroughly refereed proceedings of the 10th Workshop of the Cross Language Evaluation Forum, CLEF 2010, held in Corfu, Greece, in September/October 2009. The volume reports experiments on various types of multimedia collections. It is divided into three main sections presenting the results of the following tracks: Interactive Cross-Language Retrieval (iCLEF), Cross-Language Image Retrieval (ImageCLEF), and Cross-Language Video Retrieval (VideoCLEF).DOI: 10.1007/978-3-642-15751-6
Metrics:
See at:
hal.archives-ouvertes.fr 
| doi.org | CNR IRIS | CNR IRIS | www.springerlink.com
2010
Book
Restricted
Preface to Multilingual Information Access Evaluation I. Text Retrieval Experiments
Peters C, Di Nunzio G, Kurimo M, Mandl T, Mostefa D, Penas A, Roda G
This book constitutes the thoroughly refereed proceedings of the 10th Workshop of the Cross Language Evaluation Forum, CLEF 2010, held in Corfu, Greece, in September/October 2009. The volume reports experiments on various types of textual document collections. It is divided into six main sections presenting the results of the following tracks: Multilingual Document Retrieval (Ad-Hoc), Multiple Language Question Answering (QA@CLEF), Multilingual Information Filtering (INFILE@CLEF), Intellectual Property (CLEF-IP) and Log File Analysis (LogCLEF), plus the activities of the MorphoChallenge Program.DOI: 10.1007/978-3-642-15754-7
Project(s): PASCAL
Metrics:
Peters C, Di Nunzio G, Kurimo M, Mandl T, Mostefa D, Penas A, Roda G
This book constitutes the thoroughly refereed proceedings of the 10th Workshop of the Cross Language Evaluation Forum, CLEF 2010, held in Corfu, Greece, in September/October 2009. The volume reports experiments on various types of textual document collections. It is divided into six main sections presenting the results of the following tracks: Multilingual Document Retrieval (Ad-Hoc), Multiple Language Question Answering (QA@CLEF), Multilingual Information Filtering (INFILE@CLEF), Intellectual Property (CLEF-IP) and Log File Analysis (LogCLEF), plus the activities of the MorphoChallenge Program.DOI: 10.1007/978-3-642-15754-7
Project(s): PASCAL
Metrics:
See at:
doi.org | CNR IRIS | CNR IRIS | www.springerlink.com
2010
Contribution to book
Restricted
What happened in CLEF 2009
Peters C
The organization of the CLEF 2009 evaluation campaign is described and details are provided concerning the tracks, test collections, evaluation infrastructure,and participation. The aim is to provide the reader of these proceedings with a complete picture of the entire campaign, covering both text and multimedia retrieval experiments. In the final section, the main results achieved by CLEF in the first ten years of activity are discussed and plans for the future of CLEF are presented.DOI: 10.1007/978-3-642-15751-6_1
DOI: 10.1007/978-3-642-15754-7_1
Metrics:
Peters C
The organization of the CLEF 2009 evaluation campaign is described and details are provided concerning the tracks, test collections, evaluation infrastructure,and participation. The aim is to provide the reader of these proceedings with a complete picture of the entire campaign, covering both text and multimedia retrieval experiments. In the final section, the main results achieved by CLEF in the first ten years of activity are discussed and plans for the future of CLEF are presented.DOI: 10.1007/978-3-642-15751-6_1
DOI: 10.1007/978-3-642-15754-7_1
Metrics:
See at:
doi.org | doi.org | CNR IRIS | CNR IRIS | www.springerlink.com
2010
Book
Restricted
Introduction to the Special Issue on Indian Language Information Retrieval Part I
Harman D, Kando N, Majumder P, Mitra M, Peters C
[not available]DOI: 10.1145/1838745.1838746
Metrics:
Harman D, Kando N, Majumder P, Mitra M, Peters C
[not available]DOI: 10.1145/1838745.1838746
Metrics:
See at:
dl.acm.org | ACM Transactions on Asian Language Information Processing | CNR IRIS | CNR IRIS
2009
Contribution to book
Restricted
What Happened in CLEF 2008
Peters C
The organization of the CLEF 2008 evaluation campaign is described and details are provided concerning the tracks, test collections, evaluation infrastructure, and participation. The main results are commented and future evolutions in the organization of CLEF are discussed.DOI: 10.1007/978-3-642-04447-2_1
Metrics:
Peters C
The organization of the CLEF 2008 evaluation campaign is described and details are provided concerning the tracks, test collections, evaluation infrastructure, and participation. The main results are commented and future evolutions in the organization of CLEF are discussed.DOI: 10.1007/978-3-642-04447-2_1
Metrics:
See at:
doi.org | CNR IRIS | CNR IRIS | link.springer.com
2009
Contribution to book
Open Access
CLEF 2008: ad hoc track overview
Agirre E, Di Nunzio G, Ferro N, Mandl T, Peters C
We describe the objectives and organization of the CLEF 2008 Ad Hoc track and discuss the main characteristics of the tasks offered to test monolingual and cross-language textual document retrieval systems. The track was changed considerably this year with the introduction of tasks with new document collections consisting of (i) library catalog records derived from The European Library, and (ii) and non- European language data, plus a task offering the chance to test retrieval with word sense disambiguated data. The track was thus structured in three distinct streams denominated: TEL@CLEF, Persian@CLEF and Robust WSD. The results obtained for each task are presented and statistical analyses are given.DOI: 10.1007/978-3-642-04447-2_2
Metrics:
Agirre E, Di Nunzio G, Ferro N, Mandl T, Peters C
We describe the objectives and organization of the CLEF 2008 Ad Hoc track and discuss the main characteristics of the tasks offered to test monolingual and cross-language textual document retrieval systems. The track was changed considerably this year with the introduction of tasks with new document collections consisting of (i) library catalog records derived from The European Library, and (ii) and non- European language data, plus a task offering the chance to test retrieval with word sense disambiguated data. The track was thus structured in three distinct streams denominated: TEL@CLEF, Persian@CLEF and Robust WSD. The results obtained for each task are presented and statistical analyses are given.DOI: 10.1007/978-3-642-04447-2_2
Metrics:
See at:
doras.dcu.ie | doi.org | CNR IRIS | CNR IRIS | link.springer.com | www.scopus.com
2009
Book
Restricted
Preface to Evaluating Systems for Multilingual and Multimodal information Access
Peters C, Deselaers T, Ferro N, Gonzalo J, Jones G J, Kurimo M, Mandl T, Penas A, Petras V
This book constitutes the thoroughly refereed proceedings of the 9th Workshop of the Cross-Language Evaluation Forum, CLEF 2008, held in Aarhus, Denmark, in September 2008. The 130 revised and extended papers presented were carefully reviewed and selected for inclusion in the book. They are completed by an introduction on CLEF 2008. As usual, the seven main evaluation tracks in CLEF 2008 aimed to test the performance of a wide range of multilingual information access systems or system components. The papers are organized in topical main sections on Multilingual Textual Document Retrieval (Ad Hoc), Mono- and Cross-Language Scientific Data Retrieval (Domain-Specific), Interactive Cross-Language Retrieval (iCLEF), Multiple Language Question Answering (QA@CLEF), Cross-Language Retrieval in Image Collections (ImageCLEF), Multilingual Web Track (WebCLEF), Cross-Language Geographical Retrieval (GeoCLEF), Cross-Language Video Retrieval (VideoCLEF), Multilingual Information Filtering (INFILE@CLEF), and Morpho Challenge at CLEF 2008.DOI: 10.1007/978-3-642-04447-2
Metrics:
Peters C, Deselaers T, Ferro N, Gonzalo J, Jones G J, Kurimo M, Mandl T, Penas A, Petras V
This book constitutes the thoroughly refereed proceedings of the 9th Workshop of the Cross-Language Evaluation Forum, CLEF 2008, held in Aarhus, Denmark, in September 2008. The 130 revised and extended papers presented were carefully reviewed and selected for inclusion in the book. They are completed by an introduction on CLEF 2008. As usual, the seven main evaluation tracks in CLEF 2008 aimed to test the performance of a wide range of multilingual information access systems or system components. The papers are organized in topical main sections on Multilingual Textual Document Retrieval (Ad Hoc), Mono- and Cross-Language Scientific Data Retrieval (Domain-Specific), Interactive Cross-Language Retrieval (iCLEF), Multiple Language Question Answering (QA@CLEF), Cross-Language Retrieval in Image Collections (ImageCLEF), Multilingual Web Track (WebCLEF), Cross-Language Geographical Retrieval (GeoCLEF), Cross-Language Video Retrieval (VideoCLEF), Multilingual Information Filtering (INFILE@CLEF), and Morpho Challenge at CLEF 2008.DOI: 10.1007/978-3-642-04447-2
Metrics:
See at:
doi.org | CNR IRIS | CNR IRIS | link.springer.com
2009
Conference article
Open Access
MultiMatch - Multilingual / Multimedia Access to Cultural Heritage
Amato G, Debole F, Peters C, Savino P
Our shared cultural heritage (CH) is an essential part of our European identity, transcending cultural and language barriers. The aim of the MultiMatch project is to enable users to explore and interact with online internet-accessible CH content, across media types and language boundaries, in ways that do justice to the multitude of existing perspec- tives. This has been achieved through the development of a search engine targeted for the access, organisation and personalized presentation of cul- tural heritage information. MultiMatch aims at complex, heterogeneous digital object retrieval and presentation.
Amato G, Debole F, Peters C, Savino P
Our shared cultural heritage (CH) is an essential part of our European identity, transcending cultural and language barriers. The aim of the MultiMatch project is to enable users to explore and interact with online internet-accessible CH content, across media types and language boundaries, in ways that do justice to the multitude of existing perspec- tives. This has been achieved through the development of a search engine targeted for the access, organisation and personalized presentation of cul- tural heritage information. MultiMatch aims at complex, heterogeneous digital object retrieval and presentation.
2009
Conference article
Open Access
MultiMatch: Multiple Access to Cultural Heritage
Amato G, Debole F, Peters C, Savino P
Our shared cultural heritage (CH) is an essential part of our European identity, transcending cultural and language barriers. The aim of the MultiMatch project is to enable users to explore and interact with online internet-accessible CH content, across media types and language boundaries, in ways that do justice to the multitude of existing perspec- tives. This has been achieved through the development of a search en- gine targeted for the access, organisation and personalized presentation of large and complex cultural heritage information. MultiMatch aims at heterogeneous digital object retrieval and presentation.Project(s): Multilingual/Multimedia access to cultural heritage
Amato G, Debole F, Peters C, Savino P
Our shared cultural heritage (CH) is an essential part of our European identity, transcending cultural and language barriers. The aim of the MultiMatch project is to enable users to explore and interact with online internet-accessible CH content, across media types and language boundaries, in ways that do justice to the multitude of existing perspec- tives. This has been achieved through the development of a search en- gine targeted for the access, organisation and personalized presentation of large and complex cultural heritage information. MultiMatch aims at heterogeneous digital object retrieval and presentation.Project(s): Multilingual/Multimedia access to cultural heritage
2009
Book
Restricted
Results of CLEF 2009. Books of Abstracts.
Borri F, Nardi A, Peters C
These Working Notes contain descriptions of the experiments conducted within CLEF 2009 organised by the Cross-Language evaluation Forum.Project(s): Cross-Language Evaluation Forum
Borri F, Nardi A, Peters C
These Working Notes contain descriptions of the experiments conducted within CLEF 2009 organised by the Cross-Language evaluation Forum.Project(s): Cross-Language Evaluation Forum
See at:
CNR IRIS | CNR IRIS | www.clef-campaign.org
2008
Conference article
Open Access
CLEF 2007: ad hoc track overview
Di Nunzio G, Ferro N, Mandl T, Peters C
We describe the objectives and organization of the CLEF 2007 Ad Hoc track and discuss the main characteristics of the tasks offered to test monolingual and cross-language textual document retrieval systems. The track was divided into two streams. The main stream offered mono- and bilingual tasks on target collections for central European languages (Bulgarian, Czech and Hungarian). Similarly to last year, a bilingual task that encouraged system testing with non-European languages against English documents was also offered; this year, particular attention was given to Indian languages. The second stream, designed for more experienced participants, offered mono- and bilingual "robust" tasks with the objective of privileging experiments which achieve good stable performance over all queries rather than high average performance. These experiments re-used CLEF test collections from previous years in three languages (English, French, and Portuguese). The performance achieved for each task is presented and discussed.DOI: 10.1007/978-3-540-85760-0_2
Metrics:
Di Nunzio G, Ferro N, Mandl T, Peters C
We describe the objectives and organization of the CLEF 2007 Ad Hoc track and discuss the main characteristics of the tasks offered to test monolingual and cross-language textual document retrieval systems. The track was divided into two streams. The main stream offered mono- and bilingual tasks on target collections for central European languages (Bulgarian, Czech and Hungarian). Similarly to last year, a bilingual task that encouraged system testing with non-European languages against English documents was also offered; this year, particular attention was given to Indian languages. The second stream, designed for more experienced participants, offered mono- and bilingual "robust" tasks with the objective of privileging experiments which achieve good stable performance over all queries rather than high average performance. These experiments re-used CLEF test collections from previous years in three languages (English, French, and Portuguese). The performance achieved for each task is presented and discussed.DOI: 10.1007/978-3-540-85760-0_2
Metrics:
See at:
doras.dcu.ie | doi.org | CNR IRIS | CNR IRIS | link.springer.com | www.scopus.com
2008
Conference article
Restricted
The MultiMatch prototype: multilingual/multimedia search for cultural heritage objects
Amato G, Debole F, Peters C, Savino P
MultiMatch is a 30 month targeted research project under the Sixth Framework Programme, supported by the unit for Content, Learning and Cultural Heritage (Digicult) of the Information Society DG. MultiMatch is developing a multimedia/multilingual search engine designed specifically for the access, organization and personalized presentation of cultural heritage information. The demonstration will present the MultiMatch system prototype.Source: LECTURE NOTES IN COMPUTER SCIENCE, vol. 5173, pp. 385-387
DOI: 10.1007/978-3-540-87599-4_40
Metrics:
Amato G, Debole F, Peters C, Savino P
MultiMatch is a 30 month targeted research project under the Sixth Framework Programme, supported by the unit for Content, Learning and Cultural Heritage (Digicult) of the Information Society DG. MultiMatch is developing a multimedia/multilingual search engine designed specifically for the access, organization and personalized presentation of cultural heritage information. The demonstration will present the MultiMatch system prototype.Source: LECTURE NOTES IN COMPUTER SCIENCE, vol. 5173, pp. 385-387
DOI: 10.1007/978-3-540-87599-4_40
Metrics:
2008
Conference article
Restricted
What happened in CLEF 2007
Peters C
The organization of the CLEF 2007 evaluation campaign is described and details are provided concerning the tracks, test collections, evaluation infrastructure, and participation. The main results are commented and future evolutions in the organization of CLEF are discussed.DOI: 10.1007/978-3-540-85760-0_1
Metrics:
Peters C
The organization of the CLEF 2007 evaluation campaign is described and details are provided concerning the tracks, test collections, evaluation infrastructure, and participation. The main results are commented and future evolutions in the organization of CLEF are discussed.DOI: 10.1007/978-3-540-85760-0_1
Metrics:
See at:
doi.org | CNR IRIS | CNR IRIS | link.springer.com
2008
Book
Restricted
CLEF 2007
Peters C, Jijkoun V, Mandl T, Mueller H, Oard D W, Petras V, Santos D
This book constitutes the thoroughly refereed post-proceedings of the 8th Workshop of the Cross-Language Evaluation Forum, CLEF 2007, held in Budapest, Hungary, September 2007. The revised papers presented together with an introduction were carefully reviewed and selected for inclusion in the book. The papers are organized in topical sections on Multilingual Textual Document Retrieval, Domain-Specific Information Retrieval, Multiple Language Question Answering, Cross-language Retrieval in Image Collections, Cross-Language Speech retrieval, Multingual Web Retrieval, and Cross-Language Geographical retrieval.DOI: 10.1007/978-3-540-85760-0_1
Metrics:
Peters C, Jijkoun V, Mandl T, Mueller H, Oard D W, Petras V, Santos D
This book constitutes the thoroughly refereed post-proceedings of the 8th Workshop of the Cross-Language Evaluation Forum, CLEF 2007, held in Budapest, Hungary, September 2007. The revised papers presented together with an introduction were carefully reviewed and selected for inclusion in the book. The papers are organized in topical sections on Multilingual Textual Document Retrieval, Domain-Specific Information Retrieval, Multiple Language Question Answering, Cross-language Retrieval in Image Collections, Cross-Language Speech retrieval, Multingual Web Retrieval, and Cross-Language Geographical retrieval.DOI: 10.1007/978-3-540-85760-0_1
Metrics:
See at:
doi.org | CNR IRIS | CNR IRIS | link.springer.com
2008
Conference article
Open Access
From CLEF to TrebleCLEF: the Evolution of the Cross-Language Evaluation Forum
Ferro N, Peters C
The Cross-Language Evaluation Forum (CLEF) has been running for nearly ten years now; the aim of this paper is to provide a critical assessment of the results achieved so far. In the first part of the paper, we pro- vide a brief overview of the entire activity and sum- marise the main achievements; in the second part, we focus our attention on the Ad Hoc track with the aim of showing how the results of evaluation can be exploited to increase understanding of the many issues involved in multilingual retrieval system development. In the final part, we outline our main ideas for the future of CLEF.Project(s): TREBLE-CLEF
Ferro N, Peters C
The Cross-Language Evaluation Forum (CLEF) has been running for nearly ten years now; the aim of this paper is to provide a critical assessment of the results achieved so far. In the first part of the paper, we pro- vide a brief overview of the entire activity and sum- marise the main achievements; in the second part, we focus our attention on the Ad Hoc track with the aim of showing how the results of evaluation can be exploited to increase understanding of the many issues involved in multilingual retrieval system development. In the final part, we outline our main ideas for the future of CLEF.Project(s): TREBLE-CLEF
2008
Conference article
Restricted
From research to application in multilingual information access: the contribution of evaluation
Peters C, Braschler M, Di Nunzio G, Ferro N, Gonzalo J, Sanderson M
The importance of evaluation in promoting research and development in the information retrieval and natural language processing domains has long been recognised but is this sufficient? In many areas there is still a considerable gap between the results achieved by the research community and their implementation in commercial applications. This is particularly true for the cross-language or multilingual retrieval areas. Despite the strong demand for and interest in multilingual IR functionality, there are still very few operational systems on offer. The Cross Language Evaluation Forum (CLEF) is now taking steps aimed at changing this situation. The paper provides a critical assessment of the main results achieved by CLEF so far and discusses plans now underway to extend its activities in order to have a more direct impact on the application sector.
Peters C, Braschler M, Di Nunzio G, Ferro N, Gonzalo J, Sanderson M
The importance of evaluation in promoting research and development in the information retrieval and natural language processing domains has long been recognised but is this sufficient? In many areas there is still a considerable gap between the results achieved by the research community and their implementation in commercial applications. This is particularly true for the cross-language or multilingual retrieval areas. Despite the strong demand for and interest in multilingual IR functionality, there are still very few operational systems on offer. The Cross Language Evaluation Forum (CLEF) is now taking steps aimed at changing this situation. The paper provides a critical assessment of the main results achieved by CLEF so far and discusses plans now underway to extend its activities in order to have a more direct impact on the application sector.