FE Collective: NCFE’s AI in Assessment Journey
Since 2021, NCFE has been investigating the use of artificial intelligence (AI) to support our assessment activities. AI is extremely flexible, so early work focussed on ‘the art of the possible’, understanding what capabilities NCFE may want and how our treasure trove of data may be able to get us there. We have also funded other organisations to explore AI through our Assessment Innovation Fund – a £1 million investment on behalf of the sector for advances in assessment practices.
For some subject areas, we’ve found that generative AI (genAI) was good at creating new assessment items – once the prompt engine was thoroughly trained by digitally capturing our assessment production logic. This opens the possibility of creating hundreds of new questions with a few clicks, however all outputs needed some human editing and some items had to be discarded completely or amended significantly.
For some subjects, like English, the discard rate was high enough to go back to the drawing board on the prompts for the time being. One colleague summed this up well when she said: “It looks like your AI has swallowed a thesaurus this morning.” Commenting that often genAI can use overly complicated language.
Formative assessment and natural language processing
We’ve used formative assessment and natural language processing to automatically plot a route through content for learners, personalising their journey based on their strengths (less content) and development areas (more content). Once a learning need was identified, the system searched through written and video content and served up the closest matches it could find for the learner to read or watch.
The technology worked effectively, and learners liked and benefitted from the experience. There were some access issues, and some teachers found it hard to shoehorn in screen-time to their lessons, but overall learners and educators felt that AI had enhanced learning.
Using AI in the formative assessment space
Also in the formative assessment space, we’ve found that AI can provide learners with feedback on their work that helps them to produce a better response. AI models trained using human experts were able to label natural language, allowing learners to check the comprehensiveness of their answers before completion.
Using a similar training method, learners have received feedback on the content of their responses, showing which areas are likely to be strengths and which areas could be improved. These systems support learners to craft better responses to a homework question, piece of coursework, or mock exam answer, before submitting it to a teacher for marking.
Can AI automate human work?
Automating marking and moderation should also be within the capabilities of well-trained AI models. Indeed, we’ve found that models trained using examiner marked responses could apply the logic to novel question responses well – giving the same mark as human experts up to 95% of the time on short answer questions. The alignment was slightly lower on longer responses however this requires us to ask the question of what amount of disagreement we would accept?
This question goes to the heart of whether AI can automate human work, because we’ll have to accept an imperfect computer system, just like we accept that humans aren’t perfect.
One thing is clear, the use of large language models for providing feedback will lead to some inaccurate feedback and could harm learning. The sector should be aware that automation in the form of content generation, personalisation, feedback, and marking will require investment of time and effort to train the AI models.
Relying on the large language models, which are now very accessible in the form of ChatGPT, Bard and others, will not provide the high standard of service we expect of our educators.
By Gray Mytton, Assessment Innovation Manager, NCFE
Interested in meeting Gray in person? Join us at the FE Collective in Birmingham on October 17th! The FE Collective is a significant event where we collaborate to create a sector-specific report, crafted by the sector, for the sector. Don’t miss the chance to listen to Gray discuss topics like artificial intelligence, investments in the future sector, and progression. To find out more, join us at the FE Collective – get all the details here.
FE News on the go…
Welcome to FE News on the go, the podcast that delivers exclusive articles from the world of further education straight to your ears.
We are experimenting with Artificial Intelligence to make our exclusive articles even more accessible while also automating the process for our team of project managers.
In each episode, our thought leaders and sector influencers will delve into the most pressing issues facing the FE sector, offering their insights and analysis on the latest news, trends, and developments.
Responses