End point assessments and the potential for more innovative use of technology

Stuart Edwards discusses the potential to use technology for assessment in apprenticeships and highlights features of the End Point Assessment model that may stifle innovation. Based on his report for the Gatsby Foundation, he suggests actions for Ofqual and IfATE to move things forward.

Over the past two years, I have been looking at the potential to apply more transformative approaches using digital technology to the assessment of apprenticeships. My focus has been on end point assessment (EPA) which is a defining feature of the occupational standards-based apprenticeship approach adopted in England following the 2012 Richard Review. While I have found some interesting examples which could have wider application, these have been few and far between. I have concluded there are barriers, including features of the endpoint assessment model itself, that could stifle digital innovation in future. Nevertheless, I believe there is still more that bodies like Ofqual and the Institute for Apprenticeships and Technical Education (IfATE) could do to encourage progress and I have some suggestions to offer.

Why using digital technology for assessment matters 

It matters for two reasons. First, digital technology has the growing potential to address some of the underlying challenges of providing an assessment that is both valid and consistent, particularly for practical skills and behaviours. Secondly, as digital technology continues to transform the workplace, if it is not fully utilised in assessing workplace competence, there is a risk that some apprenticeships become less relevant and anachronistic – particularly given the weight placed on end-point assessment. 

Sampling occupational standards and EPA plans 

In the course of undertaking my review for the Gatsby Foundation, I looked across a wide range of apprenticeship standards within four main routes: digital, engineering and manufacturing, transport and logistics, and care services. I deliberately chose a range of occupations in terms of the extent of current workplace digitalisation. I also took account of scale in terms of several apprentices needing to be trained in a particular occupation. 

Each standard has an EPA plan agreed upon by a trailblazer group of employers and approved by IfATE. The plan details the assessment methods to be used by independent End Point Assessment Organisations (EPAOs) for assessing the particular occupational standard and provides a rationale for why these methods have been selected from a menu of options provided by IfATE. At least two assessment methods need to be used, one of which must be a synoptic assessment of a combination of the individual Knowledge Skills and Behaviours (KSBs) that make up the apprenticeship standard. 

The standards that I examined typically specified two or three different assessment methods. As is the case across all standards, the most commonly used assessment method is a professional discussion or interview with an independent assessor, usually supported by a portfolio of evidence compiled by the apprentice as a pre-requirement for taking the EPA rather than being part of the actual assessment. This is then often combined with a practical observation/demonstration or, particularly at higher levels, a practical project. There may also be a written test based on multiple-choice questions or scenarios. 

Examples of transformative uses of technology 

A broad distinction can be drawn between using digital technologies that streamline or add flexibility to existing assessment methods, and those that have the potential to change the nature of the assessment process itself. The former category – such as automated multiple-choice tests, remote observation or interviewing, e-portfolios etc. – are relatively widespread and their adoption has been accelerated by the flexibilities introduced by IfATE during COVID. The more transformative uses of technology – such as data capture and performance analysis using technologies already in real workplace settings, digital simulations including the use of VR/AR to create controlled assessment environments, games technologies blurring lines between formative and summative assessment, adaptive assessment using the growing capability of AI etc. – are much rarer, but I did find a few interesting examples. 

Two types of applications stood out: 

1. Use of a virtual lab environment to simulate practical tasks and problem-solving. 
   The example I looked at most closely was for the Network Engineer Level 4 EPA. This involved an apprentice undertaking two tasks each lasting seven hours over two days, one focused on network failure and the other on network optimisation. The End Point Assessment Organisation (EPAO) that demonstrated it for me created a virtual lab using the Cisco Packet Tracer Network Simulation Tool, which is cross-platform and available for free. The software generates a lab file showing exactly how the apprentice has gone about the tasks. This is then used as evidence alongside the apprentice’s decision-making notes and diagrams as the basis for structured questioning undertaken by an independent assessor. A very similar approach is used for the Cyber Security Technologist Level 4. I also found some limited examples beyond the digital route in engineering and manufacturing, where the use of simulators in the nuclear and rail industries, particularly to assess skills and behaviours in response to emergencies. 
2. Producing and demonstrating a software artefact as a project. 
   The most interesting example I found was for the DevOps Engineer Level 4 EPA. This combines the development of a product with its practical demonstration. The apprentice first develops a piece of code through a work-based project lasting approximately 12 weeks. This is submitted for assessment with supporting material such as a diagram showing the high-level system structure, a short analysis of the project areas against KSBs (max 300 words) and examples of implementations, such as coding and files, covering all the techniques used; no project report is required. The three-hour practical assessment then involves the apprentice demonstrating a successful deployment of the piece of code and being questioned by the independent assessor on the methodology they used to develop the code. The Games Programmer Level 7 EPA is similar in that it specifies a software artefact as the project output, albeit more complex, and the apprentice gives the independent assessor a presentation rather than a practical demonstration being required. 

Significance of the EPA model 

Although limited, I believe these examples are significant in that they demonstrate how digital technology can be used within the EPA regime in more transformational ways to address some of the real challenges of assessing technical education. They enable practical assessment that is controlled and replicable, mapped precisely to a specified range of KSBs, generates direct and objective evidence, and makes authentic use of workplace technology. 

In my view, this shows how technology has the potential to take us closer to the Richard Review’s vision of a holistic “test that demonstrates that the apprentice can take the knowledge and expertise they have gained and apply it in a real world context to a new, novel problem.” There can of course always be questions about the authenticity of any digital simulation in recreating a real-world context, but as the workplace itself becomes more digitalised and the technology more sophisticated that gap is closing rapidly. And it is not as if more conventional methods used in EPAs have cracked the underlying challenges. 

The premise of our apprenticeship model is that an EPA can always be designed to capture holistically the knowledge, skills and behaviours which make up an occupational standard. This has raised the stakes but does not address the inherent difficulty of doing so. In practice, I found that current EPA plans are often a heroic attempt to do the impossible. Practical assessments in real workplaces are difficult to organise in ways that are controlled and replicable, and observations can be limited and difficult to make objective and reliable. Project reports (and portfolios) place a premium on being able to write about what you have done. Professional discussions become a way to check off lists of atomised KSBs leading to a disproportionate weight being placed on them, assuming for example that up to 30 individual KSBs can be assessed in an hour and that talking about skills and behaviours equates with demonstrating them. 

Barriers to progress 

If digital technology can significantly improve EPAs, it needs to be asked why there are not more transformational applications already and whether more could be done to encourage them across a wider range of occupations. I think it is fair to say that the examples I found, though not restricted to the digital route, tended to be in occupations where these approaches are easiest and most obvious, and there was most to be gained. It helped where the technology and software was already in existence and readily available. I also saw in other areas substantial barriers of practicalities, cost and culture that would need to be overcome, even where there were potential benefits and problems that needed solving. 

Beyond that I was struck by features of the EPA model that (whatever other merits they may have) make it more difficult to adopt transformative technology for assessment of apprenticeships. In particular: 

1. The hard-line placed between in-programme training and the end point assessment creates a barrier to introducing technology that can be used for both training and assessment, undermining any business case; 
2. The requirement for EPAOs to be independent and for it to be feasible to provide assessment that is equally accessible to apprentices at any relevant employer (regardless of size or technological sophistication), makes it difficult to expect that a particular piece of technology will be used in an EPA unless its use is virtually universal within an industry. 

This has not led me to conclude that more progress is unachievable, but it needs more encouragement and practical support.

Recent policy initiatives 

Since the COVID pandemic Ofqual and IfATE have shown greater openness to exploring the potential of digital technology for assessment. However, in practice, developments to date have been limited. Ofqual’s initial focus appears to be on the practicalities of how conventional exams can be administered and taken online, rather than the scope for technology to change the nature of the assessment itself. Similarly, the flexibilities for assessing apprenticeships that IfATE has maintained post-COVID are largely to enable EPAs to be delivered at a distance or online, and other changes that follow consultation are limited to better-integrating assessment of qualifications within EPAs.

IfATE has also published a Digital Skills Framework for the inclusion of appropriate digital content across occupational standards and a Future-facing Innovation Strategy to address emerging skills. It is noteworthy, however, that neither of these has anything to say about assessment, instead focusing almost wholly on how digital and emerging skills can better be addressed through course content and occupational standards. This seems like a missed opportunity. If the object is to keep up with digital innovation in the workplace, how assessment is undertaken is not neutral: the medium is the message.

Suggestions For What More Could Be Done 

There are no quick and easy solutions, but there are steps that Ofqual and IfATE could take to encourage more progress. Here are four suggested areas for action. 

1)Regulators could make it an explicit objective to encourage more innovative use of technology to address assessment challenges in apprenticeships and wider technical education. 

2)More practical advice and encouragement could be offered, such as: 

• challenging trailblazer groups and those devising assessment plans to consider where they could make more innovative use of technology for assessment, including through guidance in the apprenticeship builder tool and advice given by IfATE product managers. 
• giving guidance specifically on the pros and cons of digital simulation rather than assuming simulation is just a fallback when the real world is impractical. 
• gathering and publicising exemplars to illustrate what is possible, such as the use of virtual labs for simulation exercises to assess Network Engineers and software artefacts as evidence of project work completed by DevOps Engineers. 

3)More direct forms of intervention could be considered, such as: 

• encouraging or sponsoring the development of a ‘sandbox’ type environment where innovative approaches to using technology to assess technical skills can be tested out 
• commissioning the IT industry to develop digital tools to assess the digital characteristics listed in the digital skills framework, so the tools could be used for apprenticeships and T-levels across all occupational areas 
• looking at the practicality of creating a generic ‘virtual lab’ that could be used to run assessments customisable to a wide range of different occupational areas 
• exploring the relationship with vendor certifications (eg provided by Microsoft, CISCO etc) and whether more use can be made of their digital assessment capabilities

4)Given the constraints identified by EPA, an alternative focus could be newer developments with arguably more initial scope for innovation, such as:

• using the ‘Skills Bootcamps’, proposed by the Future-Facing Innovation Strategy as a rapid response to emerging skills, to develop and test innovative uses of technology for assessment. 
• looking at where technology for assessment could support the accumulation of credit-bearing modules envisaged under a future Lifetime Skills Guarantee.

A Final Word on AI 

Finally, a reflection on artificial intelligence (AI), which is currently receiving so much interest. There is little doubt that AI will be directly relevant to and will extend the capabilities of how technology can be used for assessing technical education. However, it could also raise questions about which competencies should be assessed. If it is common practice in a workplace to use AI for certain tasks, does the competency to use AI effectively need to be assessed instead of being able to complete the work without it?

Another potential impact is that AI may undermine conventional academic assessment models. Arguably, the assessment of technical education has been skewed by a presumption it needs to conform to equivalence with academic assessment, ie writing down or answering questions about how to perform practical skills competently, instead of demonstrating them in action. It will be interesting to see if the challenges created by AI to the integrity of academic assessment models create some space to give more recognition to alternative, more practical approaches, perhaps facilitated by technology.

By Dr Stuart Edwards is an independent consultant and adviser, and former college chair. 

The views and opinions expressed in this article are those of the author and do not necessarily state or reflect those of the Gatsby Charitable Foundation.

---

FE News on the go

Welcome to FE News on the Go, the podcast that delivers exclusive articles from the world of further education straight to your ears.

We are experimenting with Artificial Intelligence to make our exclusive articles even more accessible while also automating the process for our team of project managers.

In each episode, our thought leaders and sector influencers will delve into the most pressing issues facing the FE.