Introducing Peer Review in an IS Analysis Course

Peer reviews were introduced as a teaching technique for the 2002 offering of the course SIF8035 Information Sys-tems at the Norwegian University of Science and Technology (NTNU). The students handed in conceptual models which were then double-blind reviewed by three independent peers. The review reports contained scores and defect lists, but were not used for grading. This paper conducts an analysis of the outcomes of the peer review exercises. Several approaches are used, both quantitative and qualitative, investigating the students’ performance and per-ceptions. The main conclusions are: 1) The scores given by the peer reviewers were not reliable enough to recom-mend their use for grading purposes. 2) The introduction of peer review exercises contributed positively to students’ learning in the course – but not equally so for all students. A substantial fraction of the students did so little that it is hard to claim any learning effect. The main reasons for this seem to have been poor motivation and unclear demands. In hindsight, we have discussed the distinguishing properties of three different purposes of peer-reviews that were not clear to us when the course started, and we have identified several possible ways of improving the peer reviews. In spite of the reported problems, experiences are more positive than negative, and it has been decided to continue with peer reviews in the course.