Crowdsourced dataset to study the generation and impact of text highlighting in classification tasks

Jorge Ramírez, Marcos Baez, Fabio Casati, Boualem Benatallah

Результат исследований: Материалы для журналаСтатья

Аннотация

Objectives: Text classification is a recurrent goal in machine learning projects and a typical task in crowdsourcing platforms. Hybrid approaches, leveraging crowdsourcing and machine learning, work better than either in isolation and help to reduce crowdsourcing costs. One way to mix crowd and machine efforts is to have algorithms highlight passages from texts and feed these to the crowd for classification. In this paper, we present a dataset to study text highlighting generation and its impact on document classification. Data description: The dataset was created through two series of experiments where we first asked workers to (i) classify documents according to a relevance question and to highlight parts of the text that supported their decision, and on a second phase, (ii) to assess document relevance but supported by text highlighting of varying quality (six human-generated and six machine-generated highlighting conditions). The dataset features documents from two application domains: systematic literature reviews and product reviews, three document sizes, and three relevance questions of different levels of difficulty. We expect this dataset of 27,711 individual judgments from 1851 workers to benefit not only this specific problem domain, but the larger class of classification problems where crowdsourced datasets with individual judgments are scarce.

Язык оригиналаАнглийский
Номер статьи820
ЖурналBMC Research Notes
Том12
Номер выпуска1
DOI
СостояниеОпубликовано - 21 дек 2019

ASJC Scopus subject areas

  • Biochemistry, Genetics and Molecular Biology(all)

Fingerprint Подробные сведения о темах исследования «Crowdsourced dataset to study the generation and impact of text highlighting in classification tasks». Вместе они формируют уникальный семантический отпечаток (fingerprint).

  • Цитировать