Using examiner reports to identify quality PhD theses

Research reported in PhD theses represents a major component of all educational research undertaken in Australia. As many candidates are also recently or currently practising teachers and administrators, PhD research often has an immediate impact on schools and schooling. Thus the quality of theses is important as a contribution to our national research effort. A national study in progress has collected candidature information and examiner reports for 804 PhD candidates across all discipline areas who submitted a thesis for examination - of these, 84 theses were in the field of Education. The text of examiner reports is being analysed on the basis of 30 categories. Among the evaluative elements are summative comments about the thesis as a whole or major sections of the thesis. The summative comment categories, accounting on average for 21% of each examiner report, have been analysed to date for 117 reports on 39 Education theses across four universities, and are being used to create a scale of 'research quality' to grade the theses. The comments are also compared with the examiner recommendations made on each thesis. The results and validity of such a process are discussed.