PEER ASSESSMENT AT SCALE.

The Popularity Problem - Managing An Application Process At Scale

The University of Edinburgh has been using PebblePad for over ten years, which is now widely embedded in practice. In this project, PebblePad was employed to support the application process for a Go Abroad Fund offered by the University's Global department. The fund provides financial support for students to undertake a short-term international experience linked to their studies, and the scheme has proved so popular it posed the challenge of how to select 200 successful applicants from around 1,100 entrants. Previously, Adaptive Comparative Judgement software was used, but the department had neither the available time nor the resources to mark this initial level in the application process.

The Peer-Assessment Ambition 

In order to support this highly popular initiative, Randall Reinhard from Edinburgh Global, and Robert Chmielewski, Online Assessment and Feedback Adviser, designed a selection process that involved applicants becoming involved as reviewers. Peer marking was considered a valuable method of initially ranking applications as it provided a means of saving tutors time from the initial marking at scale, while also enabling a meaningful peer learning activity. Having decided on the process, the team was seeking a technical solution that, alongside an intuitive interface for uploading submissions, enabled peer marking at scale and scaffolded the peer marking experience.

PebblePad offered clear advantages; it was the only technical solution available that met the fundamental requirements of the project. The sign-up process is simple, removing additional steps that the design of an equivalent process in a VLE would require. Situating all activity in PebblePad’s assessment engine, ATLAS, enables students to submit and review applications anonymously, administrators to monitor and report, and the academic tutors to pick up the rest of the process - neatly containing all activity in one workspace.

The Successful Workflow

Students sign up to be eligible for the fund, submitting their application by following a direct ‘submit to ATLAS’ link that allows them to submit their application without having to familiarise themselves with the system, a user-friendly feature of PebblePad. Designing an equivalent system in a VLE would require the student to log in, navigate to their course, seek the specific topic or quiz and then work with the relevant information. Removing a couple of interaction steps is extremely helpful in terms of reducing barriers to engagement.

After submitting their work, students also act as tutors in order to mark their peers’ applications. In a tutor role, the applicants need to mark anonymous submissions to reduce bias. Anonymity is one of the biggest advantages that ATLAS offers over other assessment systems; the students can never see whose work they are marking, but staff members can. With the user experience at the forefront, the solution design does not inhibit participation from either user perspective. PebblePad’s flexibility with permissions allows one member to act in different roles, and situating all activity in ATLAS both enables students to act in both roles and also contains all activity in one workspace.

To support the students in the review task, a bespoke Feedback Template has been created, presenting standardised Likert-style questions for the student-tutor; this reduces burdens on the student in engaging with a professional level of assessment and instead focusses on their ability as a member of the university to identify value in the submission. For administrators, monitoring and reporting is all taken care of. The reporting functionality in ATLAS automates the administrative task of compiling reports, and the CSV report on over 5,000 assessment templates can be generated within just a couple of minutes.