E-ASSESSMENT IN TECHNICAL EDUCATION: TIME TO GET SERIOUS?

The pandemic has shaken the world of assessment. As with other societal habits like commuting and meetings, it has brought lurking questions and doubts back to the surface.

One of these is whether it continues to make sense to step out of the digital world to assess people’s competence to function within it.

The powers that be have somewhat hesitantly acknowledged the impact. Simon Lebus, the Chief Regulator at Ofqual, spoke of the experience of the pandemic in these terms:

“Longer term, however, there is clearly going to be scope to reflect on what we have learned during this time and what implications it might have for assessment. I am thinking especially of the large-scale of adoption of technology and online learning and its integration into pedagogy, and whether that will ultimately have a washback into assessment.”

This week sees the publication of my report for the Gatsby Charitable Foundation, a Review of the Potential for E-Assessment for Technical Education in England.

I hope this can help move things forward and perhaps accelerate the technology “washback” into assessment, at least in so far as technical education is concerned.

The challenge of assessing technical education

The first point to acknowledge is that assessing technical education is inherently complex and difficult.

Simon Field highlights the key dilemmas in his recent FE News piece based on his parallel study for Gatsby “A WORLD WITHOUT MAPS? Assessment in Technical Education”.

The central challenge is to achieve both reliability and validity. Standardised examinations can enable rigorous and replicable assessment, but are they a credible means of assessing practical occupational competences?

Assessment embedded in real world occupational tasks is likely to be more authentic, but this cannot easily be standardised.

In England, these inherent challenges are played through against the backdrop of a highly complex, centrally-regulated system of technical qualifications and apprenticeships with multiple players involved. Significant resources are expended on a range of assessment methods, seeking to achieve both validity in the eyes of employers and reliability for the purposes of comparability and academic progression.

Results are critical for how providers are funded and regulated. However, there is limited robust evidence for the effectiveness of current practices, and there are practical concerns, particularly about the amount of time and effort absorbed by assessment activities which are separate from and do not contribute to teaching and learning.

The potential of technology-based approaches

Technology can be used in a wide range of ways for different assessment methods, at different stages of the assessment process. I have found it helpful to think of different technology-based approaches being on a broad spectrum of transformative potential in relation to the challenges of assessing technical education.

Many existing approaches offer practical ways to enhance existing assessment practices, but do not necessarily change the nature of the assessment process itself. Others appear to have greater transformational potential, but may be more difficult, to introduce within the current context and require more investment.

The table below illustrates this distinction in a simplified form for specific applications of technology.

TECHNOLOGIES AND TRANSFORMATIONAL POTENTIAL
Practical improvements to existing assessment practices	Potential to transform technical assessment
automated multiple-choice tests and script marking remote proctoring and invigilation e-portfolios work-based assessment apps e-credentials and badging	adaptive assessment using the growing capability of AI data capture and performance analysis technologies used in real workplace settings simulations using VR and AR to create controlled assessment environments games technologies blurring lines between formative and summative assessment

Examples in the lefthand column apply largely to how the assessment is delivered, marked or recorded, but don’t change the assessment method itself.

Those in the righthand column have greater potential to change the nature of the assessment itself. Using these technologies in combination, an alternative model for assessing technical education can be conceived, moving away from separate ‘stop and test’ assessment to a more continuous process embedded in the teaching and learning, and in the workplace itself.  

For example, it is possible to imagine a simulation designed using data captured from performance tracking and analysis in a real workplace, using a gamified approach for assessment that adapts intelligently to how the game is played. Such a model has potential to address some of the inherent challenges and issues for assessing technical education.

Elements of what is possible can already be seen in areas such elite sports, the military and pilot training. However, take-up of the more transformative technologies is currently very limited within the centrally regulated system in England, and where technology is used it is largely to enhance more conventional assessment methods.

The impact of the COVID-19 disruption has led to a greater willingness to explore alternative approaches to assessment using technology, but it has also exposed current limitations.

Encouraging more transformative approaches

The necessary technology already exists and is developing quickly but its use in education is relatively immature. Moreover, the centrally-regulated market within which assessment of technical education takes place in England is not particularly conducive to the introduction of transformative change.

Its complexity, the number of players involved, the number of qualifications, the limited research base, and an understandably risk-averse culture all make innovation difficult. On top of that significant ethical and legal considerations need to be taken into account.

My report considers what more might be done to open up opportunities to at least test out more transformative applications of technology for assessing technical education, and includes four recommendations:

1. Leadership.

First, there needs to be more concerted system leadership. Leaving it to the market with individual Awarding Organisations responding to a hands-off regulatory framework is unlikely to work.

The main bodies regulating the assessment of technical education in England – led by the Department for Education (DfE), and including the Office of Qualifications and Examinations Regulation (Ofqual), the Institute for Apprenticeships and Technical Education (IfATE), the Office for Students (OfS) and the Awarding Organisatons (AOs) – should make a policy commitment to work with employers to encourage and explore greater and more transformative use of digital technology for assessment.

2. Promising areas for change.

Further work should be undertaken to identify and illustrate occupational areas of greatest opportunity to test out new and transformative approaches to using digital technology for assessment. The Gatsby Foundation is ready to support this work in partnership with IfATE, employers and other key interests.

3. Demonstration projects.

Building on the identification of promising occupational areas, a competition should be run to seek proposals that could be supported as demonstration projects, the organisation and evaluation of which should be considered by DfE alongside other interested parties such as JISC and the University for Industry Trust (UfI).

4. An “EdTech Catapult”.  

Discussions around the report raised the question of whether there is now a case for some form of Catapult or accelerator type organisation focused on bringing together research and innovation in EdTech more broadly. It can be argued that the limited progress with use of technology for technical assessment is illustrative of a much wider issue, which the impact of the pandemic has brought into sharper focus.  

The report therefore recommends that to support the broader development of EdTech, Innovate UK should be asked by DfE and BEIS to review the research and innovation landscape, with a view to assessing the case for an “EdTech” Catapult or similar mechanism.

Identifying occupational areas which are most promising for change

As a next step, I will be working for Gatsby on the second of these recommendations, identifying occupational areas which are most promising for change. For this, I will be seeking others’ ideas and looking for case study material, particularly from employers.

Please look out for more details and a call for ideas from FE News readers in a follow up article due to be published next Friday 29 October.

Making the case for change

The everyday lives of young people are increasingly digital. The world of work they are being prepared to enter is increasingly digital. The means by which we ask them to prove their readiness for that world seems increasingly out of step.

Yet the potential exists for more transformative use of technology that is not only better in step with the real world but could help address some of fundamental challenges associated with assessing technical education.  

With the new Ministerial team at DfE, and the new mantra of “Skills, Skills, Skills”, perhaps now is the time to get serious.  

Dr Stuart Edwards, independent consultant and adviser, former college chair, and honorary research associate at the UCL Knowledge Lab