Five major barriers to online and on-screen assessment

An @Ofqual review has found IT provision, security and staffing issues are some of the barriers to the adoption of online and on-screen assessments in England.

An Ofqual review published today (14 Dec) has identified the key barriers to greater adoption of online and on-screen assessments in high stakes qualifications such as GSCEs and A Levels.

The review, commenced prior to the coronavirus (COVID-19) pandemic, focused on the operational barriers to delivery.

The evidence came from 3 sources:

1. A review of research literature
2. A workshop with informed stakeholders, and
3. Interviews both with experts and with leaders who have introduced on-screen or online assessments elsewhere – New Zealand, Finland and Israel.

This work is particularly timely, with and increasing interest in online assessment in the wake of the coronavirus (COVID-19) pandemic.

Online and / or on-screen assessment has been successfully implemented in a number of international jurisdictions. The review, however, found five major barriers associated with taking this approach in England. None are insurmountable, given the will, but together they do confirm that we could not move large-scale standardised tests (such as A levels) on line in the immediate future.

Five major barriers to online and on-screen assessment

1. IT provision in schools and colleges:

There is little consistency in the IT provision available in each individual school and between schools and colleges in England. This also appears to be a challenge in other jurisdictions. Variations between schools creates barriers to introducing consistent national solutions at scale.

Current provision and the ability to prepare, at pace, varies widely. Different devices and browsers/operating systems could lead to compatibility issues with the tests and differences in performance, disadvantaging some students. The cost of additional IT provision would be significant.

In jurisdictions that have successfully implemented online or on-screen assessments in this context, tested solutions to this barrier have included state funded procurement of “exam ready” devices and bring your own device (BYOD) solutions. These solutions are not without their own challenges.

2. Insufficient or unreliable internet and local network capabilities:

Barriers associated with broadband, wifi and network capabilities were the second most frequently cited in the literature, they were also discussed by participants in the Ofqual workshop and each of the international jurisdictions we contacted described how they had managed these barriers.

Substantial local differences (and issues in rural areas) were a major concern. It was also experienced in rural Finland, where they have frequent power cuts. There, encrypted assessments are downloaded ahead of time and are then distributed through a local network to maintain robust delivery without a reliance on broadband.

3. Staffing:

A lack of specialist IT staff and issues around training other staff (teachers, examinations officers) were raised by schools and colleges. In response to the coronavirus (COVID-19) pandemic, many teachers will be more familiar with delivering teaching and learning online.

However, it remains that teachers have a wide range of existing digital capabilities and awarding organisations report differing appetites to adopt on-screen or online assessment in circumstances where it has been made available. Students could receive unequal support leading to some students being better prepared than others. 

To address this concern, New Zealand recommends that only schools that use digital systems in classroom teaching, and therefore teachers are comfortable with the software, opt in to on-screen assessment systems.

4. Security:

School and college experience was often limited to managing security for paper-based examinations and variability in IT infrastructure would make security risks difficult to manage consistently. The introduction of online or on-screen assessments presents both potential improvements and new risks to security.

Rigorous security procedures are built in to every part of the current, paper based, process with established roles and practice across schools, colleges and awarding organisations. Introducing an electronic system removes some of the risks associated with storage and transport of question papers and student responses, however, new risks would be introduced to the system, for which equally or more robust security procedures would be required.

5. Planning:

The most effective approaches to introducing online/on-screen testing depended on large-scale, collaborative efforts, with clear system leadership, investment, piloting and a well-considered appetite for risk. Robust risk management plans and mitigations and robust disaster recovery were needed. This would be highly challenging to implement in the timescales available.

Following Ofqual’s consultation on the proposed changes to the assessment of GCSEs, AS and A levels in 2021, they received responses both highlighting concerns around the inequality of online solutions as well as those suggesting online assessment as a solution to current and future disruptions.

The review found evidence that students appear to have a more positive perception of a move to on-screen or online than other groups. This was also the finding of the 2020 survey of perceptions of general qualifications which suggested that young people were the most likely to agree that on-screen assessments would be fairer and more manageable than pen and paper versions.

Three measures to overcome these barriers

The review also examined measures which might be taken to overcome the main barriers. While these must address the unique context of each jurisdiction, key themes include the importance of:

1. Political support for any transition,
2. Commitment to a vision for the role that assessing on-screen or online plays in wider societal changes, and
3. A well-considered approach to addressing the inevitable risks of implementation

While the barriers to on-screen assessment at scale in 2021 are significant and likely insurmountable, this report is intended to stimulate wider discussion on the future role that the use of technology may play in improving the validity and security of high stakes assessments taken in schools and colleges.