The reliability of assessment by portfolio on a course to develop and accredit teachers in higher education

Portfolios are widely used to document and assess professional development. They are used to assess university teachers on courses run by the UK Open University. These portfolios are assessed twice, by trained assessors, against a detailed set of requirements which include learning outcomes and underpinning values. A detailed analysis was undertaken of the assessment judgements involved in the assessment of 53 such portfolios. Inter-rater reliability data are reported. These results are compared with those of other studies on portfolio assessment. Consideration is given to appropriate measures of the reliability of assessment, and to some effects of the structure of assessment and of the rules for combining scores on the reliability of assessment. Some implications for practice are explored.