short-paper

Exploring semi-automatic nugget extraction for Japanese one click access evaluation

Authors:

Matthew Ekstrand-Abueg,

Mayu IwataAuthors Info & Claims

SIGIR '13: Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval

Pages 749 - 752

https://doi.org/10.1145/2484028.2484153

Published: 28 July 2013 Publication History

Get Access

Abstract

Building test collections based on nuggets is useful evaluating systems that return documents, answers, or summaries. However, nugget construction requires a lot of manual work and is not feasible for large query sets. Towards an efficient and scalable nugget-based evaluation, we study the applicability of semi-automatic nugget extraction in the context of the ongoing NTCIR One Click Access (1CLICK) task. We compare manually-extracted and semi-automatically-extracted Japanese nuggets to demonstrate the coverage and efficiency of the semi-automatic nugget extraction. Our findings suggest that the manual nugget extraction can be replaced with a direct adaptation of the English semi-automatic nugget extraction system, especially for queries for which the user desires broad answers from free-form text.

References

[1]

C. L. Clarke, M. Kolla, G. V. Cormack, O. Vechtomova, A. Ashkan, S. Büttcher, and I. MacKinnon. Novelty and diversity in information retrieval evaluation. In ACM SIGIR 2008, pages 659--666, 2009.

Digital Library

Google Scholar

[2]

J. A. et al. Frontiers, challenges and opportunities for information retrieval: Report from SWIRL 2012. SIGIR Forum, 46(1):2--32, 2012.

Digital Library

Google Scholar

[3]

Y. Freund and R. E. Schapire. A decision-theoretic generalization of on-line learning and an application to boosting. COLT '95.

Digital Library

Google Scholar

[4]

J. Li, S. Huffman, and A. Tokuda. Good abandonment in mobile and PC internet search. In ACM SIGIR 2009, pages 43--50, 2009.

Digital Library

Google Scholar

[5]

C.-Y. Lin. ROUGE: A package for automatic evaluation of summaries. In ACL 2004 Workshop on Text Summarization Branches Out, 2004.

Google Scholar

[6]

J. Lin and D. Demner-Fushman. Methods for automatically evaluating answers to complex questions. Information Retrieval, 9(5):565--587, 2006.

Digital Library

Google Scholar

[7]

T. Mitamura, H. Shima, T. Sakai, N. Kando, T. Mori, K. Takeda, C.-Y. Lin, R. Song, C.-J. Lin, and C.-W. Lee. Overview of the NTCIR-8 ACLIA tasks: Advanced cross-lingual information access. In NTCIR-8, pages 15--24, 2010.

Google Scholar

[8]

S. Rajput, M. Ekstrand-Abueg, V. Pavlu, and J. A. Aslam. Constructing test collections by inferring document relevance via extracted relevant information. In ACM CIKM 2012, pages 145--154, 2012.

Digital Library

Google Scholar

[9]

T. Sakai, M. P. Kato, and Y.-I. Song. Click the search button and be happy: Evaluating direct and immediate information access. In ACM CIKM 2011, pages 621--630, 2011.

Digital Library

Google Scholar

Cited By

View all

Wu DLiang S(2018)Mobile Search Behavious: An In-depth Analysis based on Contexts, APPs, and DevicesSynthesis Lectures on Information Concepts, Retrieval, and Services10.2200/S00831ED1V01Y201802ICR06310:2(i-159)Online publication date: 19-Mar-2018
https://doi.org/10.2200/S00831ED1V01Y201802ICR063
Pareek JJhaveri M(2017)DLNEx: A Tool to Automatically Extract Desired Learning Nuggets from Various Learning MaterialsSmart Trends in Systems, Security and Sustainability10.1007/978-981-10-6916-1_30(319-330)Online publication date: 31-Dec-2017
https://doi.org/10.1007/978-981-10-6916-1_30
Kato MSakai TYamamoto TIwata MJones GSheridan PKelly Dde Rijke MSakai T(2013)Report from the NTCIR-10 1CLICK-2 Japanese subtaskProceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval10.1145/2484028.2484117(753-756)Online publication date: 28-Jul-2013
https://dl.acm.org/doi/10.1145/2484028.2484117

Index Terms

Exploring semi-automatic nugget extraction for Japanese one click access evaluation
1. Information systems
  1. Information retrieval

Recommendations

Creation and evaluation of large keyphrase extraction collections with multiple opinions

While several automatic keyphrase extraction (AKE) techniques have been developed and analyzed, there is little consensus on the definition of the task and a lack of overview of the effectiveness of different techniques. Proper evaluation of keyphrase ...
IR system evaluation using nugget-based test collections
WSDM '12: Proceedings of the fifth ACM international conference on Web search and data mining

The development of information retrieval systems such as search engines relies on good test collections, including assessments of retrieved content. The widely employed Cranfield paradigm dictates that the information relevant to a topic be encoded at ...
Automatic office document classification and information extraction

Comments

comments powered by Disqus.

Information & Contributors

Information

Published In

SIGIR '13: Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval

July 2013

1188 pages

ISBN:9781450320344

DOI:10.1145/2484028

General Chairs:
Gareth J.F. Jones
Dublin City University, Ireland
,
Páraic Sheridan
Dublin City University, Ireland
,
Program Chairs:
Diane Kelly
University of North Carolina, Chapel Hill, USA
,
Maarten de Rijke
University of Amsterdam, The Netherlands
,
Tetsuya Sakai
Microsoft Research Asia, China

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 28 July 2013

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Conference

SIGIR '13

Sponsor:

SIGIR

SIGIR '13: The 36th International ACM SIGIR conference on research and development in Information Retrieval

July 28 - August 1, 2013

Dublin, Ireland

Acceptance Rates

SIGIR '13 Paper Acceptance Rate 73 of 366 submissions, 20%;

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
197
Total Downloads

Downloads (Last 12 months)5
Downloads (Last 6 weeks)1

Reflects downloads up to 25 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Wu DLiang S(2018)Mobile Search Behavious: An In-depth Analysis based on Contexts, APPs, and DevicesSynthesis Lectures on Information Concepts, Retrieval, and Services10.2200/S00831ED1V01Y201802ICR06310:2(i-159)Online publication date: 19-Mar-2018
https://doi.org/10.2200/S00831ED1V01Y201802ICR063
Pareek JJhaveri M(2017)DLNEx: A Tool to Automatically Extract Desired Learning Nuggets from Various Learning MaterialsSmart Trends in Systems, Security and Sustainability10.1007/978-981-10-6916-1_30(319-330)Online publication date: 31-Dec-2017
https://doi.org/10.1007/978-981-10-6916-1_30
Kato MSakai TYamamoto TIwata MJones GSheridan PKelly Dde Rijke MSakai T(2013)Report from the NTCIR-10 1CLICK-2 Japanese subtaskProceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval10.1145/2484028.2484117(753-756)Online publication date: 28-Jul-2013
https://dl.acm.org/doi/10.1145/2484028.2484117

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Cited By

Index Terms

Recommendations

Creation and evaluation of large keyphrase extraction collections with multiple opinions

IR system evaluation using nugget-based test collections

Automatic office document classification and information extraction