On the Impact of Predicate Complexity in Crowdsourced Classification Tasks

Jorge Ramírez, Marcos Baez, Fabio Casati, Luca Cernuzzi, Boualem Benatallah, Ekaterina A. Taran, Veronika A. Malanina

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

This paper explores and offers guidance on a specific and relevant problem in task design for crowdsourcing: how to formulate a complex question used to classify a set of items. In micro-task markets, classification is still among the most popular tasks. We situate our work in the context of information retrieval and multi-predicate classification, i.e., classifying a set of items based on a set of conditions. Our experiments cover a wide range of tasks and domains, and also consider crowd workers alone and in tandem with machine learning classifiers. We provide empirical evidence into how the resulting classification performance is affected by different predicate formulation strategies, emphasizing the importance of predicate formulation as a task design dimension in crowdsourcing.

Original languageEnglish
Title of host publicationWSDM 2021 - Proceedings of the 14th ACM International Conference on Web Search and Data Mining
PublisherAssociation for Computing Machinery, Inc
Pages67-75
Number of pages9
ISBN (Electronic)9781450382977
DOIs
Publication statusPublished - 3 Aug 2021
Event14th ACM International Conference on Web Search and Data Mining, WSDM 2021 - Virtual, Online, Israel
Duration: 8 Mar 202112 Mar 2021

Publication series

NameWSDM 2021 - Proceedings of the 14th ACM International Conference on Web Search and Data Mining

Conference

Conference14th ACM International Conference on Web Search and Data Mining, WSDM 2021
CountryIsrael
CityVirtual, Online
Period8.3.2112.3.21

Keywords

  • crowdsourcing
  • predicate complexity
  • task design

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Computer Science Applications
  • Software

Fingerprint Dive into the research topics of 'On the Impact of Predicate Complexity in Crowdsourced Classification Tasks'. Together they form a unique fingerprint.

Cite this