A quantitative analysis of peer review

Abstract

In this paper we focus on the analysis of peer reviews and reviewers behaviour in a number of different review processes. More specifically, we report on the development, definition and rationale of a number of appropriate quantitative metrics to assess the processes main properties. We then apply the proposed metrics and analysis framework to data sets from conference evaluation processes and we discuss the results implications and their eventual use toward improving the analyzed peer review processes. A number of unexpected results were found, in particular: (1) the low correlation between peer review outcome and impact in time of the accepted contributions and (2) the presence of a high level of randomness in the analyzed peer review processes.

Original language	English
Title of host publication	Proceedings of ISSI 2011 - 13th Conference of the International Society for Scientometrics and Informetrics
Editors	Ed Noyons, Patrick Ngulube, Jacqueline Leta
Publisher	ISSI, Leiden University and University of Zululand
Pages	724-736
Number of pages	13
ISBN (Electronic)	9789081752701
Publication status	Published - 2011
Externally published	Yes
Event	13th Conference of the International Society for Scientometrics and Informetrics, ISSI 2011 - Durban, South Africa Duration: 4 Jul 2011 → 7 Jul 2011

Publication series

Name	Proceedings of ISSI 2011 - 13th Conference of the International Society for Scientometrics and Informetrics
Volume	2

Conference

Conference	13th Conference of the International Society for Scientometrics and Informetrics, ISSI 2011
Country	South Africa
City	Durban
Period	4.7.11 → 7.7.11

ASJC Scopus subject areas

Modelling and Simulation
Management Science and Operations Research
Computer Science Applications
Applied Mathematics
Statistics and Probability

Fingerprint Dive into the research topics of 'A quantitative analysis of peer review'. Together they form a unique fingerprint.

Cite this

Ragone, A., Mirylenka, K., Casati, F., & Marchese, M. (2011). A quantitative analysis of peer review. In E. Noyons, P. Ngulube, & J. Leta (Eds.), Proceedings of ISSI 2011 - 13th Conference of the International Society for Scientometrics and Informetrics (pp. 724-736). (Proceedings of ISSI 2011 - 13th Conference of the International Society for Scientometrics and Informetrics; Vol. 2). ISSI, Leiden University and University of Zululand.

A quantitative analysis of peer review. / Ragone, Azzurra; Mirylenka, Katsiaryna; Casati, Fabio; Marchese, Maurizio.

Proceedings of ISSI 2011 - 13th Conference of the International Society for Scientometrics and Informetrics. ed. / Ed Noyons; Patrick Ngulube; Jacqueline Leta. ISSI, Leiden University and University of Zululand, 2011. p. 724-736 (Proceedings of ISSI 2011 - 13th Conference of the International Society for Scientometrics and Informetrics; Vol. 2).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Ragone, A, Mirylenka, K, Casati, F & Marchese, M 2011, A quantitative analysis of peer review. in E Noyons, P Ngulube & J Leta (eds), Proceedings of ISSI 2011 - 13th Conference of the International Society for Scientometrics and Informetrics. Proceedings of ISSI 2011 - 13th Conference of the International Society for Scientometrics and Informetrics, vol. 2, ISSI, Leiden University and University of Zululand, pp. 724-736, 13th Conference of the International Society for Scientometrics and Informetrics, ISSI 2011, Durban, South Africa, 4.7.11.

Ragone A, Mirylenka K, Casati F, Marchese M. A quantitative analysis of peer review. In Noyons E, Ngulube P, Leta J, editors, Proceedings of ISSI 2011 - 13th Conference of the International Society for Scientometrics and Informetrics. ISSI, Leiden University and University of Zululand. 2011. p. 724-736. (Proceedings of ISSI 2011 - 13th Conference of the International Society for Scientometrics and Informetrics).

Ragone, Azzurra ; Mirylenka, Katsiaryna ; Casati, Fabio ; Marchese, Maurizio. / A quantitative analysis of peer review. Proceedings of ISSI 2011 - 13th Conference of the International Society for Scientometrics and Informetrics. editor / Ed Noyons ; Patrick Ngulube ; Jacqueline Leta. ISSI, Leiden University and University of Zululand, 2011. pp. 724-736 (Proceedings of ISSI 2011 - 13th Conference of the International Society for Scientometrics and Informetrics).

@inproceedings{d12feb6b39564ac2b3f4001f8a2a2676,

title = "A quantitative analysis of peer review",

abstract = "In this paper we focus on the analysis of peer reviews and reviewers behaviour in a number of different review processes. More specifically, we report on the development, definition and rationale of a number of appropriate quantitative metrics to assess the processes main properties. We then apply the proposed metrics and analysis framework to data sets from conference evaluation processes and we discuss the results implications and their eventual use toward improving the analyzed peer review processes. A number of unexpected results were found, in particular: (1) the low correlation between peer review outcome and impact in time of the accepted contributions and (2) the presence of a high level of randomness in the analyzed peer review processes.",

author = "Azzurra Ragone and Katsiaryna Mirylenka and Fabio Casati and Maurizio Marchese",

year = "2011",

language = "English",

series = "Proceedings of ISSI 2011 - 13th Conference of the International Society for Scientometrics and Informetrics",

publisher = "ISSI, Leiden University and University of Zululand",

pages = "724--736",

editor = "Ed Noyons and Patrick Ngulube and Jacqueline Leta",

booktitle = "Proceedings of ISSI 2011 - 13th Conference of the International Society for Scientometrics and Informetrics",

note = "13th Conference of the International Society for Scientometrics and Informetrics, ISSI 2011 ; Conference date: 04-07-2011 Through 07-07-2011",

}

TY - GEN

T1 - A quantitative analysis of peer review

AU - Ragone, Azzurra

AU - Mirylenka, Katsiaryna

AU - Casati, Fabio

AU - Marchese, Maurizio

PY - 2011

Y1 - 2011

N2 - In this paper we focus on the analysis of peer reviews and reviewers behaviour in a number of different review processes. More specifically, we report on the development, definition and rationale of a number of appropriate quantitative metrics to assess the processes main properties. We then apply the proposed metrics and analysis framework to data sets from conference evaluation processes and we discuss the results implications and their eventual use toward improving the analyzed peer review processes. A number of unexpected results were found, in particular: (1) the low correlation between peer review outcome and impact in time of the accepted contributions and (2) the presence of a high level of randomness in the analyzed peer review processes.

AB - In this paper we focus on the analysis of peer reviews and reviewers behaviour in a number of different review processes. More specifically, we report on the development, definition and rationale of a number of appropriate quantitative metrics to assess the processes main properties. We then apply the proposed metrics and analysis framework to data sets from conference evaluation processes and we discuss the results implications and their eventual use toward improving the analyzed peer review processes. A number of unexpected results were found, in particular: (1) the low correlation between peer review outcome and impact in time of the accepted contributions and (2) the presence of a high level of randomness in the analyzed peer review processes.

UR - http://www.scopus.com/inward/record.url?scp=85083494681&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85083494681&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:85083494681

T3 - Proceedings of ISSI 2011 - 13th Conference of the International Society for Scientometrics and Informetrics

SP - 724

EP - 736

BT - Proceedings of ISSI 2011 - 13th Conference of the International Society for Scientometrics and Informetrics

A2 - Noyons, Ed

A2 - Ngulube, Patrick

A2 - Leta, Jacqueline

PB - ISSI, Leiden University and University of Zululand

T2 - 13th Conference of the International Society for Scientometrics and Informetrics, ISSI 2011

Y2 - 4 July 2011 through 7 July 2011