RESEARCH SUPERVISION @ ANU

Ideas and Tools ANU Information

Overview and research

Examples of summative evaluation are generally surveys of some kind that provide individual supervisors with a of rating or score on performance, e.g. on a scale of 1-5 a supervisor might be rated 4.5 by students. Summative evaluations can also be used for developmental purposes when the recipient is prepared to understand what the ratings might mean and how changes might be made. As summative evaluations do not always contain suggestions for improvement this can sometimes be quite difficult.

However, the major difficulty with this form of evaluation at the Higher Degree by Research level is the concern that students have with regard to lack of anonymity which is critical given the sensitive nature of the relationship between supervisor and candidate. Unless a supervisor has five or more students at any one time it is unlikely that the results of some form of rating survey will be anonymous plus there are serious doubts about the validity of the results with such small numbers. This means that such rating scales are not a reasonable alternative for most supervisors with only one or two students.

There are a number of alternatives which might be used, particularly for promotion and performance purposes

The above text was based on the following research:

Aspland, T., Edwards, H., O'Leary, J., & Ryan, Y. (1999). Tracking new directions in the evaluation of postgraduate supervision. Innovative Higher Education, 24(2), 127.

Marsh, H. W. (2002). PhD students' evaluations of research supervision. The Journal of Higher Education, 73(3), 313-348.

Ideas and tools

Case Study of Peer Evaluation Research Supervision at Otago University

What is it?

Peer evaluation (or review) is a term used to describe a wide range of evaluative practices characterised by being undertaken with colleagues. It can provide evidence of both your performance as a supervisor and valuable information for your professional development.

What are the key issues for peer evaluation?

The approach described here is that used at the University of Otago for both undergraduate teaching (usually to supplement student opinion surveys or to diagnose the cause of problems) and for postgraduate research student supervision (where the concerns are the small numbers of students and exploring issues which cannot be evaluated effectively by student questionnaire). The account below draws on the University of Otago Guidelines for the Evaluation of Teaching 2005.

Academic staff at Otago are encouraged to use peer evaluation for their supervision activities as a component of the evidence they submit for their teaching performance appraisals but there is also a strong emphasis on the developmental opportunities - the processes of evaluation are too important to be used only for summative purposes or only for formative purposes. There are three key principles in the Otago peer evaluation process. They will strengthen the experience for all parties if observed.

• That it is voluntary;
• That it is collaborative;
• That it is done for the purposes of professional learning.

Academic staff are not required by the University of Otago to undergo peer evaluation, or engage in processes that involve one peer making summative judgements about one another. Peer review involves collaborative partners working together to learn about and improve their teaching practice. Working together in this way has the potential to offer critical insights into our teaching that cannot be obtained through other sources, such as student and self-evaluations. It should not be seen, however, as a substitute for other forms of evaluation.

Suggestions for conducting a peer evaluation

The first and perhaps most important decision you must make in undertaking peer evaluation is the choice of your reviewer. If the issues you wish to review have a disciplinary focus, you may need to collaborate with a reviewer from your own department. However, if you are dealing with more general issues of teaching and learning as they apply to supervision, then a peer from any discipline can be chosen. It is essential that the relationship established with your peers is built on mutual trust and respect, as the process of exposing our teaching/supervision to the criticism of others can be threatening. If you cannot establish such a relationship with your peer, then it is likely that the review will result in a defensiveness which is inimical to learning. On the other hand, it is important that the chosen peer is prepared to be critical and challenging. If the peer evaluation exercise is to be used for performance appraisal your peers must be able to establish that they have used a credible process and have behaved in a professional manner appropriate to members of the academic community whether or not they have a reputation as successful researchers.

An example of a peer evaluation plan which might result from considering the issues above could be:

• Select a peer who can evaluate the currency and appropriateness of the disciplinary content of the supervision you provide, perhaps by examination of each student's written material and the feedback you provide; by observing your discussion with the student(s) in routine meetings or at special review times; by discussing with you your aims and practices in supervision.
• Select a person, probably not the same person as in a), who has special skills in facilitating discussion. Both you and your students must have confidence in this person but they could be from outside your discipline and might be closer to the students than yourself, for example a recently completed postgraduate student or an educational expert. The "facilitator" would be asked to meet with your students individually or as a group and hold a structured discussion on the quality of your supervision. The facilitator would then prepare a report for you which is seen first by the students so they can be assured that the report does not contain material which may cause unintentional discomfort to them or to you. Your students will want to protect their working relationship with you at all costs usually in their own interests but sometimes out of exaggerated loyalty to you as their supervisor in a special part of their lives. There can be unexpected spin-offs from this form of focus group discussion inasmuch as your students may find it helpful - and enjoyable - sharing with each other their experiences working with you as their supervisor. The process is entirely compatible with evaluating supervision of students at a distance although organising the meeting is likely to be demanding.
• Other reviewers might be needed for other special aspects of your supervision especially if you are responsible for a variety of student projects or there are special circumstances - there can be no standard formula for peer evaluation of supervision given that each situation will be unique.

There are a number of decisions which need to be negotiated before carrying out the review. Each partner will have to be clear about:

• The aims and focus of the review process;
• The way in which it will be conducted and the roles each partner will play.

In the briefing session with your peer evaluators it may be useful to begin by sharing some background details on the group of students being supervised such as the stage they are at (for example who is just beginning, who is writing up and who has completed and moved away), do your students work alone or in a laboratory team, are there any students being supervised at a distance and so on. As the person initiating the review process, you should then outline a proposed focus for the review. While identification of specific issues for review can be difficult it is preferable that a clear aim is expressed. It is equally important that the aim for a review does not become so ambitious that it cannot be met by the reviewers. This may require some reflection on what is important to your teaching practice. Other forms of evaluation or past experience may also suggest areas for exploration. All partners need to be satisfied with, and in agreement about, all aspects of the brief. Ultimately, if peer evaluation is to succeed, it will be necessary to adopt practices that suit both parties. It is especially important to meet with the focus group facilitator to decide on the core questions which will be put to the students particularly if you have some issues which really matter to you. Issues of confidentiality should always be clarified.

Reporting

Whether or not you want to use the evaluation for summative purposes, a written statement from each of your peers is always valuable and will help each of you to complete the process you have agreed on at the start. At Otago, the actual reports cannot be submitted for performance appraisal, we are minimalist in the documents which can go forward to assessing panels, but a form is signed by each reviewer to establish that they were involved and the process which was used. The peer reports can be quoted in a statement on teaching which is submitted by the teacher and by the Head of Department and the reports are then held "on-call" in case verification of the way they have been cited is required. While indirect, the process does assist in achieving the delicate balance between using peer evaluation for summative purposes and for those valuable formative opportunities. It makes little difference; each report should be quite brief, professional in the criticism given, and constructive in its suggestions for improvement. As with conducting formal examinations of undergraduate students and writing references for job applicants the credibility of the peer evaluation report reflects back on the reputation of the writer so that bland commendations unsubstantiated by evidence do neither the reviewer or the reviewee any good and can also bring the process into disrepute. The crucial issue of the confidence of the students has been discussed above.

It is usually helpful to have a debriefing session with each of the peer evaluators, probably after they have given you their reports. If the process has been collaborative and in the spirit of mutual benefit there is much to be learned from talking with those who now know a great deal about your role in research student supervision.

Prepared by Professor Chris Heath Otago University (2005)

ANU information

CEDAM offers an evaluation service for supervisors who have sufficient numbers of candidates to offer anonymity or, where sufficient numbers can be included over time.

Contact Robin Collins