On a fairly regular basis, we hear about some new, possible application for #ArtificialIntelligence which can potentially supplement or replace complex human judgement – the sort of judgements which can take humans a long time. Just last week in the news the results of a research study on AI in breast cancer screening were reported

This study reported that AI was as good as the current double-reading system of two doctors in the NHS; and it was actually superior at spotting cancer than a single doctor. The research was just that - research - and had not been used in an actual clinical setting.  But it gives the sector an idea of the potential of this technology in this context.

A role for AI in marking

At Ofqual, we want to understand whether there might be a role for AI in marking. Like reading x-rays, complex human judgement is required to make a judgement based on a complex set of information. Markers read student responses, and then interpret and evaluate them using a mark scheme to determine a mark for the student.  While we know that the quality of marking in England in GCSEs and A levels is generally good and compares well internationally, we are interested in opportunities for improvement – not by replacing human judgement, but by using AI to support markers in the role they play.

We are particularly interested in whether using AI as a second marker or as a way of monitoring marking might help improve marking.  Can AI be more effective in identifying inconsistent markers?  Furthermore, can AI be more effective in spotting an erroneous mark from an otherwise good and consistent marker?  Because marking can be a very demanding task, it is to some extent inevitable that are occasionally mistakes (and underlines the importance of the post-results reviews and appeals system).

AI competition

To help us understand the potential for AI, we are conducting some research which includes an ‘AI competition’.  Essentially, we will get several thousand student responses to an essay (a particular GCSE English language essay question), and these will be marked by human markers. All AI systems involve ‘training’ – in other words, the AI system is given a large number of examples with the ‘right answer’ or the best human judgement, which it will then attempt to replicate.  It is important that the training examples use the ‘best’ human judgement, because AI systems are only as good as the data put into them. Therefore, in our study we will use the most senior markers, and each essay will be marked multiple times to ensure the marks do not reflect error.

These responses will then be used for a competition so that many individuals and organisations with expertise in AI can train an AI system to mark similarly to the training set.  We can then test these AI systems on another set of essays (for which we know the marks, but the AI systems do not).  We very much hope this competition will help stimulate and identify the very best practice in this field.

The results from this competition will help us undertake further subsequent research work – for example, modelling the impact of AI as a second marker or as a marker monitoring system.

Enhancing marking quality

It is early days in terms of looking at AI in marking, but it is important to take some first steps on this in England, by beginning this exploratory research.  If there are genuine potential improvements, ways which might enhance marking quality, of course we want to know, so we can encourage the system to adopt such practices safely. Similarly, we want to have a deep understanding of the potential risks in operating such technology in our high stakes examinations.

Right now, we are recruiting schools to provide students’ essay responses for mock tests to a particular past question – we will give schools examiner marks and annotations and for these essays (which should be very useful to schools for teaching and learning) in return for being able to use the responses in the research and AI competition. Altogether, we are hoping for around 3,000 student responses.

This new project is exciting. Any future use of AI is likely to take some time and a lot of testing.  We are not going to suddenly see AI being used at scale in marking high profile qualifications overnight.  But we hope in the very least this work will help the sector to think about other ways and innovations to improve marking quality and the delivery of qualifications more broadly to ensure each learner’s work gets the mark it deserves.

Beth Black, Director, Research and Analysis, Ofqual

If any schools are interested in taking part, please email us at This email address is being protected from spambots. You need JavaScript enabled to view it. to express an interest and for more information.

You may also be interested in these articles:

Sponsored Video

Register, Login or Login with your Social Media account:


Advertisers

Upcoming FE Events

Advertiser Skyscrapers

Newsroom Activity

Latest Education News

Further Education News

The FE News Channel gives you the latest education news and updates on emerging education strategies and the #FutureofEducation and the #FutureofWork.

Providing trustworthy and positive Further Education news and views since 2003, we are a digital news channel with a mixture of written word articles, podcasts and videos. Our specialisation is providing you with a mixture of the latest education news, our stance is always positive, sector building and sharing different perspectives and views from thought leaders, to provide you with a think tank of new ideas and solutions to bring the education sector together and come up with new innovative solutions and ideas.

FE News publish exclusive peer to peer thought leadership articles from our feature writers, as well as user generated content across our network of over 3000 Newsrooms, offering multiple sources of the latest education news across the Education and Employability sectors.

FE News also broadcast live events, podcasts with leading experts and thought leaders, webinars, video interviews and Further Education news bulletins so you receive the latest developments in Skills News and across the Apprenticeship, Further Education and Employability sectors.

Every week FE News has over 200 articles and new pieces of content per week. We are a news channel providing the latest Further Education News, giving insight from multiple sources on the latest education policy developments, latest strategies, through to our thought leaders who provide blue sky thinking strategy, best practice and innovation to help look into the future developments for education and the future of work.

In May 2020, FE News had over 120,000 unique visitors according to Google Analytics and over 200 new pieces of news content every week, from thought leadership articles, to the latest education news via written word, podcasts, video to press releases from across the sector.

We thought it would be helpful to explain how we tier our latest education news content and how you can get involved and understand how you can read the latest daily Further Education news and how we structure our FE Week of content:

Main Features

Our main features are exclusive and are thought leadership articles and blue sky thinking with experts writing peer to peer news articles about the future of education and the future of work. The focus is solution led thought leadership, sharing best practice, innovation and emerging strategy. These are often articles about the future of education and the future of work, they often then create future education news articles. We limit our main features to a maximum of 20 per week, as they are often about new concepts and new thought processes. Our main features are also exclusive articles responding to the latest education news, maybe an insight from an expert into a policy announcement or response to an education think tank report or a white paper.

FE Voices

FE Voices was originally set up as a section on FE News to give a voice back to the sector. As we now have over 3,000 newsrooms and contributors, FE Voices are usually thought leadership articles, they don’t necessarily have to be exclusive, but usually are, they are slightly shorter than Main Features. FE Voices can include more mixed media with the Further Education News articles, such as embedded podcasts and videos. Our sector response articles asking for different comments and opinions to education policy announcements or responding to a report of white paper are usually held in the FE Voices section. If we have a live podcast in an evening or a radio show such as SkillsWorldLive radio show, the next morning we place the FE podcast recording in the FE Voices section.

Sector News

In sector news we have a blend of content from Press Releases, education resources, reports, education research, white papers from a range of contributors. We have a lot of positive education news articles from colleges, awarding organisations and Apprenticeship Training Providers, press releases from DfE to Think Tanks giving the overview of a report, through to helpful resources to help you with delivering education strategies to your learners and students.

Podcasts

We have a range of education podcasts on FE News, from hour long full production FE podcasts such as SkillsWorldLive in conjunction with the Federation of Awarding Bodies, to weekly podcasts from experts and thought leaders, providing advice and guidance to leaders. FE News also record podcasts at conferences and events, giving you one on one podcasts with education and skills experts on the latest strategies and developments.

We have over 150 education podcasts on FE News, ranging from EdTech podcasts with experts discussing Education 4.0 and how technology is complimenting and transforming education, to podcasts with experts discussing education research, the future of work, how to develop skills systems for jobs of the future to interviews with the Apprenticeship and Skills Minister.

We record our own exclusive FE News podcasts, work in conjunction with sector partners such as FAB to create weekly podcasts and daily education podcasts, through to working with sector leaders creating exclusive education news podcasts.

Education Video Interviews

FE News have over 700 FE Video interviews and have been recording education video interviews with experts for over 12 years. These are usually vox pop video interviews with experts across education and work, discussing blue sky thinking ideas and views about the future of education and work.

Events

FE News has a free events calendar to check out the latest conferences, webinars and events to keep up to date with the latest education news and strategies.

FE Newsrooms

The FE Newsroom is home to your content if you are a FE News contributor. It also help the audience develop relationship with either you as an individual or your organisation as they can click through and ‘box set’ consume all of your previous thought leadership articles, latest education news press releases, videos and education podcasts.

Do you want to contribute, share your ideas or vision or share a press release?

If you want to write a thought leadership article, share your ideas and vision for the future of education or the future of work, write a press release sharing the latest education news or contribute to a podcast, first of all you need to set up a FE Newsroom login (which is free): once the team have approved your newsroom (all content, newsrooms are all approved by a member of the FE News team- no robots are used in this process!), you can then start adding content (again all articles, videos and podcasts are all approved by the FE News editorial team before they go live on FE News). As all newsrooms and content are approved by the FE News team, there will be a slight delay on the team being able to review and approve content.

 RSS IconRSS Feed Selection Page