Notes on the Reviewing of Papers

Roger Clarke **

Notes of 20 June 2006, rev. 21 June 2006

Prepared for the 19th Bled eCommerce Conference 5-7 June 2006, when presenting the Outstanding Paper Award

© Xamax Consultancy Pty Ltd, 2006

This document is at http://www.anu.edu.au/people/Roger.Clarke/SOS/PaperRev.html

2 slides to support a brief presentation on the topic are at http://www.anu.edu.au/people/Roger.Clarke/SOS/PaperRev.ppt

Introduction

Academic communities are heavily dependent upon the efforts of their members in relation to the editing of journals, the management of conference programs, the refereeing of journal articles and the review of conference papers. Despite the importance of these roles, few journals or conferences provide guidance about what is expected of a referee or reviewer. These Notes are intended to contribute to the filling of that gap. I'd appreciate your thoughts about how to improve this document.

Although there are differences in the level of expectation for a refereed journal article in comparison with the reviewing of a conference paper, the two activities are sufficiently similar that these Notes treat them as though they were the same activity. The term 'review' is used throughout to encompass 'refereeing' as well.

The Function of Reviewing

The reviewing of papers is an act of quality assurance. The dimensions of quality that a journal or conference is concerned about vary, particularly in the weighting given to them. The quality factors that are considered by the Bled Outstanding Paper Award Committee are as follows:

• Academic Quality (which corresponds to Rigour)
• Real World Quality (or Relevance)
• Presentation Quality (or Accessibility)
• Contribution Quality (or Ambition)

Regrettably, some journals have sacrificed Relevance in recent years and prioritised Rigour for its own sake. Some journals, and most conferences, adopt a more balanced approach. The reviewer needs to take into account the the particular venue's attitude towards these quality factors.

Who is the Reviewer Working For?

There are several customers or stakeholders whose interests need to be taken into account.

Clearly, the immediate purpose of a review is to provide the Editor or Program Chair with information about the paper's acceptability or otherwise. But, important as those people are, they operate as a proxy for the journal's readers or the conference delegates.

The next purpose of a review is to inform the Author(s) about what needs to be done to improve the paper, possibly to the level needed to get across the threshhold and achieve acceptance, possibly not.

But it's advantageous to think about the Community with whom the reviewer is engaging. This may be formed around the nucleus of a discipline, a research domain, a geographical region, or intersections among two or more of them. All members of that community stand to gain from a professional approach to reviewing.

Indicators of Quality in a Review

A treatise could be written on what constitutes a good review; but there are a couple of key characteristics that enable a review to serve the needs of the editor, the author and the community alike:

1. demonstrated understanding of the paper. A lengthy recapitulation is unnecessary, but the reviewer needs to either summarise the paper in a couple of sentences, or to convey by other means that they've grasped what the author was trying to do. The reason this matters is that if the readers of a review aren't satisfied that the reviewer understood the paper, they will de-value the comments made
2. positive features of the paper, identified briefly. This should not be overdone, but it should be present. It's very rare that a paper reaches a reviewer without at least some redeeming features. Especially where the list of criticisms is substantial, it's important to convey to the reader that they and their effort aren't worthless. Apart from the morality of the matter, criticism is more likely to be effective if it's cushioned by some recognition of worth
3. critique, expressed constructively. This is the central feature: a good review must lead somewhere. The editor or program chair needs to make a decision; and the author needs suggestions about what they can do about the features the reviewer doesn't like. If the reviewer's opinion is that the paper isn't appropriate for this venue and never will be, where does it belong? Or, at the very least, what next steps should the author take in order to improve their work?
4. value-add. It's a cliché, but it's applicable. It's linked to another cliché: performing a review is an act of collaboration not competition. The self-confident reviewer doesn't limit themselves to identifying and helping to eradicate the inadequacies; they also offer additional perspectives, references and inferences

What Not To Do

There are also some traps that it's important for reviewers to avoid. Three are ethical:

• don't fail to declare lack of expertise. Reviewers should decline invitations to review papers in areas they are not sufficiently familiar with. Where the lack of expertise is partial (e.g. in relation to a particular research technique), a simple declaration of the reviewer's limitations, and hence deference to other reviewers on that aspect, is sufficient
• don't fail to declare a conflict of interest. Examples include an association with the research reported on, or with one of the authors of the paper
• don't take premature advantage of content. Examples include providing copies to a colleague or student prior to publication. This doesn't preclude approaching the editor to seek the author's consent for a targeted and limited distribution

There are also a couple of corollaries of the four quality indicators discussed above:

• don't be emotive, aggressive, sarcastic or demeaning. Not only do such stylistic problems shake the confidence of junior or otherwise insecure authors, they also reduce the effectiveness of the advice provided
• don't be vague about the paper's weaknesses. It helps neither editor nor author to say that 'this paper makes too limited a contribution to be considered for this venue', 'the method is weak', 'the literature search is too narrow', or 'the implications are insufficiently developed', unless the theme is supported and developed
• don't criticise without 'actionable advice'. This term is Allen Lee's. He goes further than merely urging that criticism be expressed constructively. Criticism shouldn't even be made unless it culminates in a suggestion about what might be done to address the problem

Bibliography

Directly relevant articles and notes from within the IS discipline are:

Bieber M. (1997?) 'How to Review' at http://www-ec.njit.edu/~bieber/review.html

Davison R.M., de Vreede G-J. & Briggs R.O. (2006) 'On Peer Review Standards For The Information Systems Literature' Commun. AIS 16, 49 (2005) 967-980, at http://cais.isworld.org/articles/default.asp?vol=16&art=49

• Drawing on the literature, Davison et al. (2006, p. 15) identifies attributes of a good reviewer as being humaneness, competence, openness, unbiassedness and absence of prejudice, ethicality, timeliness, persuasiveness and diligence

Koh C. (2003) 'IS journal review process: a survey on IS research practices and journal review issues' Infor. & Mngt 40 (2003) 743-756

Lee A.S. (1995) 'Reviewing a Manuscript for Publication' Invited Note in J. Ops Mngt 13, 1 (July 1995) 87-92, at http://www.people.vcu.edu/~aslee/referee.htm

Zmud R. (1998) 'A Personal Perspective on the State of Journal Refereeing' MIS Qtly 22, 2 (September 1998), at http://www.misq.org/archivist/vol/no22/issue3/edstat.html

A couple of relevant items from other disciplines are:

Black N., van Rooyen S., Godlee F., Smith R. & Evans S. (1998) 'What Makes a Good Reviewer and a Good Review for a General Medical Journal' J. Am. Med. Assoc. 280, 3 (1998) 231-233

Finney D.J. (1997) 'The Responsible Referee' Biometrics 53, 2 (June 1997) 715-719

Smith, A.J. (1990)  'The task of the referee' IEEE Computer 23, 4 (April 1990) 65-71

Author Affiliations

Roger Clarke is Principal of Xamax Consultancy Pty Ltd, Canberra. He is also a Visiting Professor in the E-Commerce Programme at the University of Hong Kong, a Visiting Professor in the Cyberspace Law & Policy Centre at the University of N.S.W., and a Visiting Professor in the Department of Computer Science at the Australian National University.

Personalia

Photographs

Access Statistics

These community service pages are a joint offering of the Australian National University (which provides the infrastructure), and Roger Clarke (who provides the content).

Created: 20 June 2006 - Last Amended: 21 June 2006 by Roger Clarke - Site Last Verified: 15 February 2005
This document is at www.anu.edu.au/people/Roger.Clarke/SOS/PaperRev.html
Mail to Webmaster   -    © Xamax Consultancy Pty Ltd, 1995-2006   -    Privacy Policy