On the Impact of Predicate Complexity in Crowdsourced Classification Tasks

Jorge Ramírez, Marcos Baez, Fabio Casati, Luca Cernuzzi, Boualem Benatallah, Ekaterina A. Taran, Veronika A. Malanina

Результат исследований: Материалы для книги/типы отчетовМатериалы для конференции

Аннотация

This paper explores and offers guidance on a specific and relevant problem in task design for crowdsourcing: how to formulate a complex question used to classify a set of items. In micro-task markets, classification is still among the most popular tasks. We situate our work in the context of information retrieval and multi-predicate classification, i.e., classifying a set of items based on a set of conditions. Our experiments cover a wide range of tasks and domains, and also consider crowd workers alone and in tandem with machine learning classifiers. We provide empirical evidence into how the resulting classification performance is affected by different predicate formulation strategies, emphasizing the importance of predicate formulation as a task design dimension in crowdsourcing.

Язык оригиналаАнглийский
Название основной публикацииWSDM 2021 - Proceedings of the 14th ACM International Conference on Web Search and Data Mining
ИздательAssociation for Computing Machinery, Inc
Страницы67-75
Число страниц9
ISBN (электронное издание)9781450382977
DOI
СостояниеОпубликовано - 3 авг 2021
Событие14th ACM International Conference on Web Search and Data Mining, WSDM 2021 - Virtual, Online, Израиль
Продолжительность: 8 мар 202112 мар 2021

Серия публикаций

НазваниеWSDM 2021 - Proceedings of the 14th ACM International Conference on Web Search and Data Mining

Конференция

Конференция14th ACM International Conference on Web Search and Data Mining, WSDM 2021
СтранаИзраиль
ГородVirtual, Online
Период8.3.2112.3.21

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Computer Science Applications
  • Software

Fingerprint Подробные сведения о темах исследования «On the Impact of Predicate Complexity in Crowdsourced Classification Tasks». Вместе они формируют уникальный семантический отпечаток (fingerprint).

Цитировать