TY - GEN
T1 - On the Impact of Predicate Complexity in Crowdsourced Classification Tasks
AU - Ramírez, Jorge
AU - Baez, Marcos
AU - Casati, Fabio
AU - Cernuzzi, Luca
AU - Benatallah, Boualem
AU - Taran, Ekaterina A.
AU - Malanina, Veronika A.
N1 - Funding Information:
Thiswork was supported by the Russian Science Foundation (Project No. 19-18-00282).
Publisher Copyright:
© 2021 ACM.
Copyright:
Copyright 2021 Elsevier B.V., All rights reserved.
PY - 2021/8/3
Y1 - 2021/8/3
N2 - This paper explores and offers guidance on a specific and relevant problem in task design for crowdsourcing: how to formulate a complex question used to classify a set of items. In micro-task markets, classification is still among the most popular tasks. We situate our work in the context of information retrieval and multi-predicate classification, i.e., classifying a set of items based on a set of conditions. Our experiments cover a wide range of tasks and domains, and also consider crowd workers alone and in tandem with machine learning classifiers. We provide empirical evidence into how the resulting classification performance is affected by different predicate formulation strategies, emphasizing the importance of predicate formulation as a task design dimension in crowdsourcing.
AB - This paper explores and offers guidance on a specific and relevant problem in task design for crowdsourcing: how to formulate a complex question used to classify a set of items. In micro-task markets, classification is still among the most popular tasks. We situate our work in the context of information retrieval and multi-predicate classification, i.e., classifying a set of items based on a set of conditions. Our experiments cover a wide range of tasks and domains, and also consider crowd workers alone and in tandem with machine learning classifiers. We provide empirical evidence into how the resulting classification performance is affected by different predicate formulation strategies, emphasizing the importance of predicate formulation as a task design dimension in crowdsourcing.
KW - crowdsourcing
KW - predicate complexity
KW - task design
UR - http://www.scopus.com/inward/record.url?scp=85103066199&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85103066199&partnerID=8YFLogxK
U2 - 10.1145/3437963.3441831
DO - 10.1145/3437963.3441831
M3 - Conference contribution
AN - SCOPUS:85103066199
T3 - WSDM 2021 - Proceedings of the 14th ACM International Conference on Web Search and Data Mining
SP - 67
EP - 75
BT - WSDM 2021 - Proceedings of the 14th ACM International Conference on Web Search and Data Mining
PB - Association for Computing Machinery, Inc
T2 - 14th ACM International Conference on Web Search and Data Mining, WSDM 2021
Y2 - 8 March 2021 through 12 March 2021
ER -