Reflecting on what we have learned during this time and what implications it might have for assessment

Simon Lebus @Ofqual delivered a speech, to the National Association of Head Teachers (@NAHTnews) (18 Mar), on the topic of assessment in 2021:

Good afternoon. It is a great pleasure to be here with you this afternoon at your School Leaders’ Summit. My name is Simon Lebus and I am the Interim Chief Regulator at Ofqual. I started at Ofqual on January 4th, which was the day the Prime Minister announced that the summer’s exams would not proceed normally. It has therefore proved a rather different job to what I had anticipated, and I have as a result found myself presiding over a major departure in how we manage Year 11 and Year 13 assessment in the form, for general qualifications at least, of a wholesale migration to teacher assessment, a change that in normal times would have been regarded as revolutionary.

I think your last School Leaders’ Summit was in London in February last year. The pandemic was beginning to penetrate public consciousness as the focus moved from China to gruesome scenes from hospitals in Italy and Spain shown nightly on TV. Lockdown, however, was still a month away, you were still offering attendees trips on the London Eye from which they would have been able to see a busy London as we can now hardly remember it, and none of us had any real appreciation of the speed with which the way we live was about to change or the scale of the crisis which was shortly to engulf us.

This has had enormous repercussions in virtually every dimension of our lives, education maybe especially so. Here, we have had to deal with schools being shut down, with a move to online learning using new technologies and pedagogies (something which would have taken years in normal times), with student stress and anxiety and mental health problems, with loss of learning and opportunities for social interaction and development, and with teachers suddenly faced with significant extra workload and having to carry out all sorts of extra tasks and duties and in some areas to reinvent their role. And of course, you, as school leaders, have had to manage all that while dealing with a flurry of different public health guidance, worried parents and the chaos and confusion caused by such a fast-moving set of events.

Exams, of course, are normally a fixed point on the school year’s horizon, but they too have been a casualty. I spent fifteen years of my life earlier in my career running exam boards. During that time exams were generally regarded as a necessary, entirely inevitable but utterly unloved feature of the educational landscape, so it was striking how much they were missed when they were suspended last year and replaced with Centre Assessed Grades. This, of course, is going to be the second year where a combination of public health concerns and worries about fairness have led to exams not being made available and I thought it might be helpful, therefore, in my speech to talk about a few of the considerations that have driven the design of this year’s arrangements.

One of the lessons from last year was to do with the need to have arrangements in place that we could be confident would command public support. Last year concerns about the impact of the moderation algorithm, about the appeals arrangements and a general sense of lack of student agency led to a feeling the system was not fair. This resulted in a wholesale loss of public support.

It was in part the determination to avoid this that led to the desire to carry out a wide-ranging public consultation, something we embarked on jointly with the Department for Education at the beginning of this year. We received more than 100,000 responses to this – around 50,000 from students, 25,000 from parents, 10,000 from teachers – in addition to responses from school groups, teacher representatives and other professionally interested groups, such as the NAHT. We had a large team that read through them all and the consultation responses have provided valuable input for the principles that we have embedded within this year’s approach.

One of the challenges in designing a fair system in circumstances where there has been so much disruption to learning and where it has occurred so unevenly is to make sure that students are only assessed on what they have been taught, and not on the learning that they have missed. There has been much discussion about this and whether there might be scope to try to address the issue of differential learning loss through the assessment system, above and beyond the compensation provided by focusing assessment only on what has been taught.

To do this in any systematic way would involve trying to make an estimate of potential or quantifying the amount of learning lost and integrate that estimate into the assessment of actual learning, something which would involve again resort to algorithms and I think potentially create further unfairness. It would also conceal the reality that lost learning will ultimately need to be recovered through access to additional remedial learning opportunities, something that will likely need to be dealt with in the schools, colleges, HE destinations and in some case within the employments that students graduating through this year’s exam system progress to.

Another debate has been as to the use of externally set tasks – sometimes incorrectly referred to as mini-exams – and whether these should be used as part of this year’s assessment regime, and if so, the extent to which they should be made compulsory and be taken under exam conditions. There were some strong voices in favour of that approach on the grounds that it would provide a powerful vehicle for standardisation and reduce the risk of unfairness arising out of different schools and colleges adopting different and inconsistent approaches, something that it was felt had been one of the problems in 2020.

Consultation responses on this were rather mixed (students perhaps unsurprisingly being especially unsupportive) but we in the end decided against the mini-exam approach in favour of a permissive use of externally set tasks. This was partly because the nature of the disruption that has been suffered means that the setting of mini papers could not be organised centrally as it would not accommodate students having been taught different areas of content. Partly for the practical reason of concern about the potential vulnerability of fixed assessments on set dates being vulnerable to further and unpredictable public health hazards such as new COVID variants. Partly out of a concern that it could lead to further compression of learning by encouraging neglect of non-examined elements of the curriculum. And partly reflecting the reality, as a matter of good assessment practice, that mini exams often rely on too small a sampling of what has been learned to form a reliable basis for judgement.

The approach that has been therefore adopted is that exam boards will provide a menu of tasks drawn largely from previous years’ exam papers from which teachers will be able to draw as part of their assessment strategy, to be used alongside other sources of evidence such as coursework, homework, in-class and across-cohort assignments, mock exams and so on. They will be supported by grade descriptors, exemplar materials and mark schemes. This will all therefore be part of the scaffolding that will be provided to support teachers in making their assessment judgements and is designed as an approach to help support judgement being applied in a consistent way across and between schools.

Much of this debate, of course, is predicated on assumptions about the difficulty of applying teacher judgement and some of the heavy responsibility the task places on teachers who find themselves not merely preparing their students for the next step of their life journey but also for allocating the exam grades that provide a passport to it. We know that teachers feel the weight of this responsibility and are not always comfortable with some of the moral dilemmas and conflicts with which it confronts them.

That is why we are investing very heavily with the exam boards on developing a range of materials for teachers that will provide a framework for them to use in developing and deploying their assessment strategies. The approach will be permissive in that it will allow teachers to develop approaches that best suit their context and their students.

In this respect many of the approaches likely to be adopted will draw on sources of evidence very similar to those used in 2020 but without the confounding factors of either the anticipation of the effect of a moderating algorithm or the need to rank order students.

The intention is that this, combined with some of the materials and training being provided by exam boards, and given teachers’ already existing familiarity with exam board syllabuses, will produce a broadly consistent internalisation of the overall standard that will support consistency across the system. This is clearly not going to be the same as the various very prescriptive and precise controls that operate in a normal year when exams are running, but it does represent a controlled environment in which teacher judgement can be deployed in a consistent and supportive way.

In order for that to work, teachers need to be given space to do the job properly. There is clearly extra work associated with that but we will be doing our best to try minimise this by reducing bureaucracy where possible and regulating for consistent approaches between exam boards. It is also good that the normal accountability pressures will not apply as results will again not be used this year for accountability purposes.

I have also heard concerns expressed about undue parental and student pressure being placed on teachers to try to influence their judgement. We will be providing for the reporting of all such activity, as it is essential that teachers’ already difficult task is not made more difficult by having to deal with these additional and unacceptable pressures.

More generally a lot of work is going into the quality assurance processes that will be deployed. These will come in three parts.

There will be the record keeping and data gathering that takes place in centres, accompanied by a description of supporting internal review processes and ultimately recorded as part of the head of centre attestation that goes with the school or college’s recommended grades to the exam boards.

The exam boards will then carry out further checks, both where they identify centres as being high risk as a result and also through a programme of random checking so that they can assure themselves that internal quality assurance processes have been effectively and consistently applied across centres. These are also likely to involve some contextualisation of 2021 recommended grade profiles against historic outcomes and confirmation that students have attained at a level that will allow them to progress to the next stage of their education or training.

One of the issues that arose last year was appeals. Although these attracted much publicity the actual numbers were much lower than a normal year. When I checked our records, I noted that there were around 300,000 appeals in 2019 resulting in around 70,000 grade changes. This compared to under 10,000 in 2020, though I recognise this is not a like for like comparison as the grounds for appeal were much more restricted.

I would not want to make estimates for what we should expect this year, where the system is again slightly different, but I think it is worth emphasising that many of the normal incentives that encourage high levels of appeals will not apply. In particular, because students will be receiving holistic grades based on teacher judgement, there will not be the usual pattern of students entering speculative applications for re-marks on the basis that they are one or two marks off a grade boundary.

I also believe that the provision for teachers to share with students details of which pieces of work it is that teachers are basing their assessment judgement on before the recommended grades are submitted in June will lead to fewer surprises come Results Day in August. More generally the nature of the holistic judgements that are being made this year is that they cannot be picked apart like UMS scores – the appeals system has therefore been designed to cater for situations where there have been gross miscarriages of justice or manifest failures of academic judgement rather than with the fine-grained differences of grade boundaries that we deal with in normal years.

I am very aware that continued uncertainty about precisely what the summer 2021 arrangements are going to involve is adding to concern and anxiety both within schools and among students and parents. Detailed guidance will be published by the end of this month at the latest and we are working hard to see if we can manage slightly earlier.

Once the guidance is published schools will be able to start developing their plans and commence the business of putting the necessary measures in place. I want to take this opportunity to repeat how aware I am of the additional burden this places on schools and colleges and their leadership and staff and how appreciative I am of the positive engagement there has been in my many meetings with various stakeholders as we all commit ourselves to ensuring that students do not suffer any additional disadvantage by being deprived of their access to the assessed grades that will support the next stage of their progression into learning, training or employment.

This is a collective effort involving multiple actors, many of them playing roles or operating in ways that would have been unfamiliar in pre-COVID times, and there is clearly a greater element of uncertainty as a consequence than there would be in normal times.

One manifestation of this is worry about grade inflation. I would not wish to be drawn into anticipations about this, but would emphasise that some of the factors driving inflation last year do not apply, in particular the sense that teachers might have had, when recommending grades, that their recommendations would be subject to moderation by an algorithm and some of the challenges associated with a very prescriptive approach to rank ordering. We would also be expecting schools and colleges, as part of the quality assurance process, to identify any outcomes that look atypical – in either inflationary or deflationary terms – against previous year outcomes and to provide some sort of explanation as to what might have caused that.

Overall, however, I believe that teacher judgement, supported with the scaffolding and guidance about assessment standards that are due to be provided by exam boards, is a trustworthy and sound basis on which to operate. I am also aware that teachers recognise the great importance of not awarding students grades that would mislead them about what are the most suitable progression routes for them to pursue.

The main issue therefore would be the effect of operation of benefit of the doubt. Say you have a class of 30 year 11 GCSE candidates, and 5 of them have produced work, on more than one occasion and under fairly controlled circumstances, which leads you to believe they are capable of getting a grade 9 on the day of the exam. In reality, we know that all 5s probably won’t quite manage it on the day as they may have a bad day, some problems at home or the wrong questions come up. Inevitably it is impossible to be sure which of the 5 will, and which won’t.

So, acting with complete professional integrity, using the knowledge you have of normal grading standards, the range of evidence you have of their performance, and following exam board guidance, you would likely submit a grade 9 for all 5 of them. That small act of professional judgement, made in perfectly good conscience, and with good evidence, available for scrutiny if requested, will inevitably have an impact when repeated across the system, but that will lead only to some small upward pressure on outcomes, not the ‘Weimar-style inflation ‘or ‘prizes for all’ that some commentators have unhelpfully suggested. That seems to me an entirely legitimate consequence of deploying teacher judgement for this purpose, something that I hope will be recognised and respected in the public discourse.

I have focused mainly on 2021 and we are in the thick of that at the moment. However, as summer approaches, the weather improves and we begin to be able, vaccine supplies allowing, to contemplate life post lockdown, we are also turning our thoughts to the future. The 2022 Year 13 cohort is evidently going to be unusual in that it will be coming to A levels never having before sat public exams and both Years 11 and 13 will have suffered high levels of learning disruption.

Discussion about arrangements for 2022 is already underway and these will no doubt look at what easements might be desirable. Longer term, however, there is clearly going to be scope to reflect on what we have learned during this time and what implications it might have for assessment. I am thinking especially of the large scale of adoption of technology and online learning and its integration into pedagogy and whether that will ultimately have a washback into assessment. I am also hopeful that a successful experience this year will allow us to reflect in a more substantial way on the role of teacher judgement in assessment and the contribution it can make.

I am alas now out of time, but these are some of the issues that we have been reflecting on and I hope this gives you a useful sense of how we are approaching the next few months from a regulatory point of view and of some of the issues which we think are going to need consideration longer term.

Simon Lebus, Chief Regulator, Ofqual