Description and results of the Remote Assessment Systems Pilot - SAR

Registration is now closed.

Motivated by the need to find a solution for student assessment at a time of emergency remote teaching, the FCT FCCN Unit challenges the Higher Education community to participate in a study to help find the preferred solution.

For this purpose the following platforms will be available: ProctorExam, ExamNet, TestWe or Respondus. The teacher has the possibility to perform a set of tests in order to evaluate these Remote Assessment Systems (RAS), and thus identify which best meets the necessary requirements.

FCCN SAR Pilot

Pilot Results

Through the participation of various higher education institutions, with the coordination of a team from the National Scientific Computation Unit (FCCN Unit) of the FCT, it was possible to make a learning path regarding what should be done for the future.

The first and main conclusion drawn is that the remote assessment solutions existing in the market present several implementation difficulties, taking into account the reality and the objectives of Portuguese Higher Education. Besides their high cost (the most advanced platforms in fraud prevention have values that are around 10€ per student per exam), most of the teachers who participated in the study indicate that these solutions are not very effective in fraud prevention.

On the other hand, regarding Data Protection - one of the main issues associated to this theme - the study concluded that the existing remote assessment solutions do not convey the level of confidence, regarding data management, that is required from teachers and Portuguese higher education institutions. One of the main lessons learned from this project is related to the complementary role of Colibri. As some existing platforms do not allow real-time monitoring by video, this FCCN service, which is based on Zoom technology, was used in conjunction by some of the member institutions.

According to the project's conclusions, these combined solutions seem to be sufficient for most of the uses. The integration of Colibri with other tools is well accepted by the teachers, even though it is necessary to find an integrated solution adjusted to the needs of the national HEIs that are not solved by the commercial tools. In this sense, the integration of Colibri in this study allowed us to draw some conclusions about which functionalities are critical for a future assessment platform to be successful.

The possibility of two-way communication (teacher/student and teacher/class), the cost compatible with the financial reality of the HEIs, the recording and analysis of data (audio and video) in a controlled place and with clarified legislation were pointed out as some of the strategic and necessary steps, as well as the eventual integration of the main functions into a single application that can run on various devices.

sar respondus pilot

Pioneering platform in evaluation tools for e-learning systems.

  •  Integrated into e-learning LMS platforms (Moodle, Blackboard, etc.);
  • It includes several steps of verification of the student's identity and environment;
  • Use of own browser for screen lock, applications and keyboard shortcuts;
  • AI recording and detection of suspicious behaviour.

Platform managed by the University of Aveiro

sar pilot

Fully customisable platform to suit all remote assessment requirements.

  • Identification through Citizen's Card;
  • 360º vision using a smartphone;
  • Screen sharing.

Platform managed by the University of Lisbon

pilot sar examnet fccn

Affordable platform that works on any device and browser, be it PC, MAC, iPads.

  • In mathematics exams the student has access to formulas and to elaborate graphs;
  • The student only accesses tools that the teacher authorises;
  • The student has access to the examination by means of a password.

Platform managed by UTAD

sar testwe fccn pilot

Software that privileges the teaching staff and its pedagogical content.

  • Offline assessments for students;
  • Automatic sending;
  • Online correction.

Platform managed by the Polytechnic Institute of Bragança

FAQS

Who can use this Pilot?
All higher education teachers who belong to RCTS. Check if your entity is a RCTS member in the "RCTS Entity List".

I want to do 5 exams for my 10 classes, can I?
The aim of this Pilot is to collect data from different solutions and different teachers. On platforms with a limited number of licenses we suggest you do the assessments with a reasonable number of students, but allow others to do so as well.

Some of the platforms are paid, do I pay anything?
No. FCT and the partners ensure the access to the platforms free of charge by the teachers. In exchange, they expect the completion of a set of forms, so that they can objectively perform their mission of collecting data to support the subsequent decision to scale, or not, one or more of these services to the entire academic community.

Do I need to involve my entity in the pilot?
The pilot is intended to be direct to lecturers to facilitate the process and avoid delays, but this does not justify transposing internal rules in your institution. Whenever you consider it necessary validate internally with distance learning services before requesting access to the pilot.

How is access to data controlled?
All platforms have their own terms and conditions, and all are committed to complying with the GDPR.

Which platforms are part of the pilot?
ProctorExam; ExamNet; TestWe; Respondus.

Why are the partner entities part of the pilot?
The FCCN Unit of the FCT wants the search for solutions to be actively participated by the community, so it took advantage of movements that were already occurring and that showed willingness. There is no preference of these entities over others.

I am a teacher from a partner entity, do I have to use the platform managed by my entity?
No. You can request access to any of the other platforms in the pilot.

I am a student, I don't want to participate, am I obliged?
Teachers should always propose alternatives to remote assessment. Contact your teacher.

The exam was taking place and there was a network problem, what do I do now?
In case of failure during the exam, the student must contact the teacher as soon as possible, through the means indicated by him/her. The solution varies according to the failure.

Can I still apply for the SAR Pilot?
Registration for the SAR pilot is now closed. The teachers who tested the four products (ProctorExam, ExamNet, TestWe and Respondus) moved on to a second phase of the pilot to analyse the data obtained.

More on Innovation