Journal peer review: comparing the perceptions and quality judgments of experienced Australian reviewers in education and the physical sciences

Journal peer review has often been the subject of research especially its practices, but most studies focus on isolated aspects and isolated contexts, and rarely have reviewer perceptions and understandings of what is involved in judging quality been explored and compared by discipline. This study used a holistic approach and a mixed methods design, drawing on journal editorials and policies, surveys and an interview with experienced Australian reviewers, and their review reports, to gain insight into peer review across and within contrasting disciplines of education and physics and chemistry. The study addresses a number of questions: How are the principles and practices of peer review defined within education and the physical sciences? How do reviewers perceive and address their role and activities in their respective disciplines? What judgments are made about quality and why, and do either differ by discipline? The study challenged the perception that there are major differences between disciplines in terms of expectations and judgments of research. There were more similarities than differences found across disciplines not only in practices, but in the specific considerations of quality. Reviewer considerations of quality can be conceptualised into Group-A features related to remediable flaws and Group-B features related to irremediable flaws, and a Quality Continuum that stratifies papers by lines of ‘Remedy’, ‘Excellence’, and ‘Impact’. Each of the features can also be further interpreted by reviewer emphases, as well as their respective ‘power’ in determining recommendation. The broad grouping of quality features identified from the contrasting disciplines was the same, namely, argument, writing, literature review, fit for journal, research method, contribution, and novelty. However, reviewer emphases in distinguishing quality differed slightly in the interpretations of ‘novelty’ and ‘contribution’. There was no sustained disciplinary difference in the language of review even though it was exceedingly rich and nuanced. One key finding was that ‘poor argument’ is the feature most likely to lead to rejection if other flaws are detected in a marginal paper. There was a stronger emphasis from education respondents about the ‘instruction’ role of peer review and ‘collegial support’ as a motivation for them to contribute as reviewers. The participants expressed a strong sense of faith in peer review as well as a tolerance of its deficiencies, and generally conceded that little could be changed in the process. They indicated that learning to review was a continuous cycle where formal training would not work, and there was also a pressing need to recognise reviewers’ contribution formally and appropriately, especially by their employing institutions. This thesis contributes to the emerging literature on how judgments occur in the context of research assessment and offers a framework to evaluate the effectiveness of journal peer review, to detect imbalances in the system, and to inform reviewer development.