Crowd-based multi-predicate screening of papers in literature reviews

Evgeny Krivosheev, Fabio Casati, Boualem Benatallah

Результат исследований: Материалы для книги/типы отчетовМатериалы для конференции

6 Цитирования (Scopus)


Systematic literature reviews (SLRs) are one of the most common and useful form of scientific research and publication. Tens of thousands of SLRs are published each year, and this rate is growing across all fields of science. Performing an accurate, complete and unbiased SLR is however a difficult and expensive endeavor. This is true in general for all phases of a literature review, and in particular for the paper screening phase, where authors filter a set of potentially in-scope papers based on a number of exclusion criteria. To address the problem, in recent years the research community has began to explore the use of the crowd to allow for a faster, accurate, cheaper and unbiased screening of papers. Initial results show that crowdsourcing can be effective, even for relatively complex reviews. In this paper we derive and analyze a set of strategies for crowd-based screening, and show that an adaptive strategy, that continuously re-assesses the statistical properties of the problem to minimize the number of votes needed to take decisions for each paper, significantly outperforms a number of non-adaptive approaches in terms of cost and accuracy. We validate both applicability and results of the approach through a set of crowdsourcing experiments, and discuss properties of the problem and algorithms that we believe to be generally of interest for classification problems where items are classified via a series of successive tests (as it often happens in medicine).

Язык оригиналаАнглийский
Название основной публикацииThe Web Conference 2018 - Proceedings of the World Wide Web Conference, WWW 2018
ИздательAssociation for Computing Machinery, Inc
Число страниц10
ISBN (электронное издание)9781450356398
СостояниеОпубликовано - 10 апр 2018
Событие27th International World Wide Web, WWW 2018 - Lyon, Франция
Продолжительность: 23 апр 201827 апр 2018

Серия публикаций

НазваниеThe Web Conference 2018 - Proceedings of the World Wide Web Conference, WWW 2018


Конференция27th International World Wide Web, WWW 2018

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Software

Fingerprint Подробные сведения о темах исследования «Crowd-based multi-predicate screening of papers in literature reviews». Вместе они формируют уникальный семантический отпечаток (fingerprint).