Why does Ofsted do research?

#rED2021 – The Chief Inspector discussed @Ofstednew’s research on education during the pandemic, remote education and curriculum at researchED National Conference 2021:

What a brilliant day at #rED2021 #red21 – so much intellectual energy and excitement. Great teacher talks including @MsJasmineMN @DanHudson84 @Mrs_McGhie @adamsingleto , and I met/re-met so many people. Thank you Tom and Hélène and all who contributed.

— Amanda Spielman (@amanda_spielman) September 4, 2021

Thank you very much for inviting me! It’s so good to be here in person – it’s my first face-to-face event since March 2020.

Funnily enough, my last non-virtual speaking event was a ResearchEd at Nishkam High School in Birmingham, a few days before the first lockdown. So it feels right to restart where I left off.

Few of us appreciated then quite what was coming. The last 18 months have been torrid – in so many other ways. But life is opening up again and being here together is a marker of how far we’ve come. Something worth celebrating, I think.

As is ResearchEd itself. I think I’ve been to every single ResearchEd National Conference, and as always there is a cornucopia of fascinating sessions. Most of them are led by classroom teachers or school leaders, and this is the real strength of ResearchEd. It prioritises and gives voice to the people who work where research meets practice.

Research is important, but it only transforms children’s lives if it is translated into the classroom. So, it needs to be discussed in forums like this.

I’ve already been to a session this morning by Jasmine Lane. I think I won’t alarm the people whose sessions I’m planning to attend this afternoon by announcing them here. 

The other day I started making a list in my notebook about the pieces of research that have been most influential in my own thinking over the years, and it is longer than I had realised.

Many will be extremely familiar to a ResearchEd audience, and some of their authors are speaking here today – Paul Kirschner for example.

One on my list is ‘The socio-economic gradient in teenagers’ literacy skills’ by John Jerrim, who I’m extremely pleased to say has just been seconded part-time to Ofsted as an expert adviser. I know he’ll bring a great deal to our work.

A second is ‘A review of the literature on marking reliability’ by Michelle Meadows and Lucy Billington – this one didn’t just inform my thinking about exams in my time at Ofqual, but also my thinking about the reliability of inspection.

And the last one I’ll mention today – and I promise that (a) I’m not making this one up and (b) it is important and relevant, is entitled ‘Computation of conditional probability statistics by 8-month old infants’.

This one illuminates how language develops in babies and young children, and informs how they can best be taught to read.

Why does Ofsted do research?

So I’m very happy to have an opportunity to talk about research. Anyone who knows me knows that research is close to my heart. My first proper job in education, many years ago now, was as Research and Policy Director at Ark Schools. And it’s an interest I’ve carried through Ofqual and into Ofsted too.

To explain how insight and research fit into our strategy, I’ll take a step back and talk for a moment about a classic model of an inspectorate. We know that inspectorates are meant to influence a system – but how do they do it? Is it really all about judgements, or is it more complicated?

I think it’s a bit more complicated, and that an inspectorate influences a system in at least 5 ways. I’ll lay out what I think these are in the schools context.

First, all inspectorates have inspection frameworks. Like others, ours sets out how we assess schools. As I hope you know, ours is as grounded in evidence as we can make it. The framework affects what schools do even before they have been inspected against it.

Then there is the inspection process itself. It has been redesigned not just to reach conclusions, but also to build in the professional dialogue that should help those at the receiving end of inspection.

Our post inspection surveys show that in the vast majority of cases school leaders agree that the inspection process has been valuable.

Next, our judgements both reassure people and also help them make choices and decisions when necessary.Then our reports aim to give parents in particular the information and reassurance that they need.

All of these elements have been significantly modified or developed in recent years, I believe for the better. The fifth way that an inspectorate has influence, and what I mainly want to talk about today, is our insights, analysis and research. If the only way we used our inspection evidence and inspectors was to publish reports on individual schools, then schools themselves, those responsible for them and also government would get much less value out of inspection than they should.

This element of our work allows us to present the big picture, highlight good practice and inform policy development. We report on many aspects of the state of education and care in England. Our work is thorough. We’re impartial. We’re honest and we’re often a critical – and independent – friend to government.

With apologies for hitting you with another list, there are several distinct strands worth unpacking here:

First and foremost, inspection evidence ought to be fully used to identify problems, successes and important themes. Our inspectors are in schools, colleges, nurseries, prisons and other settings every day. Putting this together, we have a goldmine of information that is constantly accumulating.

We use our researchers to make sense of it, and this is something we’ve been working on doing better. Researchers can spot the patterns, the shifts and the changing picture. Inspectors look at one institution at a time. But combined with research expertise we can see the big picture. We can show what teachers on the front line are experiencing and how government policy is having an impact.

Second, our inspection frameworks need to be built on strong evidence, on hard facts, not hunches. And it’s also important to be transparent about what research we use. That is why we published an overview of the research we used in designing the EIF (education inspection framework). As the evidence base evolves, we must keep up with it.

Thirdly, we use evaluation tools. Research and evaluation are, of course, two sides of the same coin. Once you’ve done your research and made your policy, you don’t just sit back, put your feet up, and carry on ad infinitum. Part of our evaluation work is looking critically at our own inspection practice. We are always working to improve inspection – making sure that it’s doing what it is meant to do. We ask ourselves some tough questions.

Almost as soon as we brought in our new education inspection framework, we asked ourselves: is this working? How do we know? So even though EIF inspections were paused after just 2 terms due to COVID-19 we set about evaluating their implementation and impact, to coin a phrase. We used post-inspection surveys, targeted questionnaires and focus groups to understand what was going well and what we could improve.

And again, because transparency is so important, we published our findings and although the results were very positive, they did help point us to some changes to how we work.

We also use evaluation to ask ourselves broader questions. Is the culmination of all our inspection and regulation having the right effect on the system? Are we continuing to be a force for improvement in education and care? One area we continue to investigate is how to keep pace with the changing accountability landscape. We see, not just in schools, where multi-academy trusts are so influential, but also in early years and social care, groups of providers have greater control over what happens in individual nurseries or children’s homes. We want to make sure inspection keeps up and holds accountability at the right level. Our evaluation work is investigating this across early years, schools and in social care.

And we don’t just evaluate our own work. The Department for Education also asks to evaluate policy initiatives, such as T-levels.

One strand of evaluation will be to look at how well the government’s catch-up plans work over the next few years. We’ll be saying more about this work soon. We are doing this because we have the independence, credibility and expertise to evaluate how policy is working on the ground. We will see what’s working well and what needs to improve. And of course, we’ll be publishing our findings and advising government on whether changes need to be made.

And finally, we also do our own research and analysis to fill some of the biggest knowledge gaps in education and social care. We do this in areas where we have a particular advantage compared to others, sometimes because we can bring together experience and perspectives from different contexts where we work. Sometimes we can do this directly from inspection evidence, but sometimes we need to design tailored visits and interviews to collect enough information.

Good recent examples of this work that you might have seen are our research reports on knife crime and “stuck schools”, and of course the various stages of our work on curriculum. Outside education, examples include reports on domestic abuse and on sexual abuse in the family environment. It’s very clear that this kind of work often informs national debate.

I’ve been heartened by the response to one of our most important pieces of research, which was carried out in record time in response to public concern. The Everyone’s Invited website carried thousands of heart-breaking testimonials of sexual abuse and harassment, many reflecting institutional or individual reluctance or sometimes failure to recognise problems.

Our work both shed light on what is really happening and where and pointed to the potential for a different approach in schools and colleges that could do more to address the issues earlier. For our part, we will be following this up more closely in our inspections.

Like our knife crime report, the sexual abuse report added weight and robust information to conversations about wider societal issues. Every day inspectors speak to children affected by domestic violence, or who have real fear there could be someone with a knife waiting around the next corner, or who have been coerced into a sexual act against their will. Ofsted helps these children’s voices be heard by decision-makers, budget holders and policymakers.

Much of what I’ve described is work that we used to do, and that most inspectorates do, but which we’d come over time to do less and less of. We’ve rebuilt the research and evaluation team to do this, and it’s become absolutely integral to our model. We now have around 40 staff, led by Verena Brähler, who is here today and hoping to meet lots of people. We have a full programme ahead of us, which I’ll say a bit more about at the end of this talk.Everything I’ve talked about so far adds up to how as an inspectorate we are a force for improvement.

What we aren’t though, is a university or a think-tank. Nor do we write opinion columns or convene groups of educationalists to talk about 23rd century education. We’re not in the business of dreaming up utopian scenarios. Like you, we are interested in what works best to educate children now. And thanks to research, we now know more than we ever have done before.

Before I talk about the future, I’d like to talk about what we’ve been doing over the past year and a half, as unsurprisingly it was a bit different from usual.

Research during the pandemic

Last summer, while hundreds of our inspectors and other staff were redeployed in government, to local authorities, to Ofqual and elsewhere, our researchers got to work on several projects.

Our first piece of COVID-specific research was on governance. We recognised almost immediately – how important good governance would be in a crisis. So, we collaborated with the National Governance Association on some research on governing during COVID-19, published last September.

We asked governors about their experiences during the first lockdown, looking at how they responded, and what they felt the major challenges were. Even at that relatively early stage, almost every governor was worried about how long it would take for pupils to ‘catch-up,’ and the long-term impact of school closure on children with disadvantages.

Interim visits

In the autumn we were again able to visit schools, colleges, nurseries, and more. We conceived the autumn visit programme to give as much information as we could on how schools were coping. We had almost all of our HMI available, so we used them to find out what was happening in schools.

We wanted to see what was being done for children in such challenging circumstances. To get underneath the issues schools and other providers were facing, and to see how they were responding. All vital insights for the public, for government, for schools, and all the other providers.

There were some prevailing themes. Critically, we had a real sense of where and how children had lost out. It was clear that lockdown hadn’t just affected the most deprived or most vulnerable children. Almost all had been affected to some degree, and of course some have been more profoundly affected than others. Children didn’t just miss out on being taught new material, they also missed out on practising and consolidating what they had already been taught.

Leaders in many schools were particularly concerned about this loss of consolidation as well as teaching for children with SEND, especially in literacy. Some leaders said that these pupils have ‘struggled’ and have ‘fallen further’ than those without SEND.

We also revealed the sheer number of children who hadn’t made it back to school and the wide variations in absence rates, as well as continued problems with remote learning.

We covered all ages, from the youngest children to apprentices. Younger children’s development was slowing, especially in language and social skills. Some were even regressing and forgetting basic skills.

Home schooling for younger children required a lot of parental input, which was hard for parents, especially for those who were juggling their own jobs and looking after pre-school children as well, as well as for the households with more obvious disadvantages like lack of laptops or other connected devices.

Many older children lost motivation. A significant minority didn’t do much at all. College education was severely constrained. Apprentices often lost their training entirely, and some lost their jobs too.

In the autumn, there was very much a sense of everyone feeling their way in the dark. Our research helped to cast some much-needed light. There wasn’t much else out there at the time, and I think we really communicated the challenges – not just in terms of the impact on children, learners and staff, but the systemic issues that had been made worse by COVID.

And despite some initial trepidation, I know that schools and colleges overwhelmingly found these visits to be helpful and constructive too. They often said it was good to talk to someone from outside about what they and their team had been through.

We published findings for all our main areas of work in instalments in October, November and December – no mean feat in terms of turning the analysis around.

And as a result of our work last autumn, we are now better at speedy and effective translation of inspection evidence into expert national reports. I’m pleased to say that we plan to do more national reporting from inspection throughout next year, with a particular eye on how children are catching up.

Remote learning

Our next COVID-19 research project was about remote education. We were among the first to pull together something actionable about what was and wasn’t working.

We did the fieldwork during the third national lockdown and published in January. As you might expect, the picture, was mixed. We did find there had been a great deal of progress with remote learning since the first lockdown. Many schools had worked exceptionally hard to get a good offer in place. Expectations of pupils were higher, and schools were doing a lot more to monitor what children were learning.

However, it was abundantly clear that remote learning is a sticking plaster at best for most children. We saw that even the most well-thought-out offer couldn’t replace classroom learning, despite everyone’s best efforts. Equality of access was a problem. Despite many initiatives to get technology to families, there were still families who struggled. And many parents simply weren’t able to help their children that much.

For older children motivation was a huge problem. Despite a lot of encouragement, a substantial minority simply didn’t engage at all.

More generally children were, frankly, worn out. Fed up with months of interrupted learning, fed up being cooped up in their rooms. Fed up with no sport, fed up with not seeing their friends.

Even some of the most motivated pupils had simply checked out by the time we got to the third lockdown. That’s something that all of you, who were probably also fatigued by home working and endless online lessons, can empathise with!

Above all, our remote education work reinforced just how important it is for children to be in school: not just for academic reasons, but for social and health reasons too.

And while we all hope that there will be no need for routine remote learning any time soon, there is still great value in these insights.

For example, it was helpful to counter some of the most persistent myths that really aren’t based in any kind of evidence. That remote education is somehow different to other kinds of teaching and learning. That it needs an entirely different curriculum. That it must be based entirely around live lessons.

Of course, remote education is just one way of teaching a curriculum – a means, not an end. Everything we know about what makes a great curriculum doesn’t just disappear once it’s being taught remotely.

First, ‘remote’ isn’t synonymous with ‘digital’. Sometimes, teaching children online will be the best way to help them learn. But there are times when a good textbook or worksheet will do the job equally well.

Then, of course, aligning your remote and classroom curriculum is vital. You wouldn’t start teaching something at random in the middle of your carefully sequenced lessons. Just as in the classroom, a remote curriculum needs to be well-sequenced and give pupils the building blocks they need to progress step by step.

These are just a few examples. But we set out to help schools strengthen what they do. I think that’s what we achieved. And, if we do ever need to return to remote by default, schools and colleges are in a better place.There’s one other lockdown project I want to mention, that we’ve recently helped bring to life. No-one can have missed how hard it has been to assess GCSEs and A levels these past two summers, with all the trials and tribulations of CAGs [centre assessment grade] and TAGs [teacher assessed grades]. This has clearly not been easy for teachers or pupils.

Ofqual analysis reassures us that, in the main, the distribution of grades in these years has been remarkably similar to previous years. Nevertheless, the scale of the increases in grades and the adaptations that have been made to qualification curriculum and assessment will affect many young people for years to come, as well as their schools and colleges.

To help researchers look at these effects, we have been working closely for the last year with Ofqual, the DfE [Department for Education], UCAS and the Office for National Statistics, to create what I believe will be a valuable research dataset.

The Grading and Admissions Data for England dataset, or “GRADE” for short, combines relevant anonymised pupil level information from DfE, Ofqual and UCAS. Over time, this dataset will allow important questions to be answered, such as how the patterns of attainment have affected applications to further and higher education, and whether interest in important future skills areas is growing in the right way.

This GRADE dataset is now available to approved researchers through the ONS Secure Research Service. I thoroughly commend this to anyone interested in this kind of research, and I’d also like to thank the Administrative Data Research Centre for making funding available for research fellowships.

Curriculum

I did mention the curriculum in the context of remote education, but it is probably years since I got this far in a ResearchEd talk without discussing curriculum in detail. So let’s correct that.

Some of you may have heard my colleague Heather Fearn speak earlier. She’s a curriculum expert, particularly in History. And, as you’d expect, our curriculum thinking is steeped in research. From early work to develop a concept of the quality of education – giving our EIF solid foundations – to the subject reviews we’ve published more recently. Again, using that all-important evidence to come to a view about what makes high-quality education in each subject.

The EIF emphasises the substance of education. But what do we mean by ‘substance’? We mean a strong, well-founded and well-balanced educational programme, competently taught, addressing all children effectively. That means a lot more than just good exam and test results. We all know that they can never convey the full picture of the quality of an education.

To apply the EIF well, to understand the substance, it is important that inspectors have a strong shared understanding of what makes a good curriculum. Without this there would be potential for inconsistency. And it is important that this is built subject by subject, to reflect the very real differences between them. Quality and progression in mathematics is very different from quality and progression in history. Genericism can be dangerously seductive, offering apparent neatness that can conceal serious intellectual defects.

Our subject reviews were therefore planned to make our concept of quality explicit, for inspectors but also as useful reference documents for schools and colleges. I should stress, though, that they aren’t inspection checklists or additional sections of the inspection handbook.

We’ve published 7 subject research reviews so far – with more to come later in the year. It’s well-known that some subjects have more supporting evidence than others, especially when we get past mathematics, science and the core humanities. Our work is laying out that body of knowledge.

We’d planned to publish our reviews before the pandemic hit, but they’re perhaps even more relevant now, as schools are having to think very carefully about curriculum as they get children back on track.

Almost all the reviews have been welcomed by teachers and subject associations alike. The mathematics review has had a more mixed reception, perhaps reflecting the level of contention within the maths subject community, where there is a long-standing divergence of views. We’ve aimed to produce a review that is manageable and accessible, to help practitioners in schools. We balanced citing supporting academic references with the need for this publication to be a manageable size and accessible to as many people as possible.

So, what’s next for our curriculum work?

The next reviews coming out are on computing, PE, and art and design and English. After these, we will be publishing a series of subject-specific reports in 2022, looking at what’s happening on the ground in a range of subjects.

And this really isn’t about prescribing what schools and colleges should do. I believe that we are reinstating here what has always been properly done by inspectorates. Many people have told me how valuable Ofsted subject reports were in the past, and I’d like them to be available again.

We’re helping teachers here by making things clearer. And in the long term, I hope our subject work will bring greater depth and consistency to how people think about a given subject, and the right kind of consistency between subjects too.

Moving forward

So, you could say that all roads point to curriculum – pre, during and post-pandemic. And this focus is very much here to stay.

Yes, it has been a difficult time for everyone in education – putting it mildly. But the pandemic doesn’t alter the fact that education substance matters. And a great curriculum is what is going to help most children catch up, though many teachers don’t like that phrase.

And we are about to restart inspections. Of course, we understand that you still have a lot to contend with. But there’s a lot to do to get children where they need to be – and inspection is an important part of that.And I do believe that the EIF is the right tool to unpick how children are doing and to help schools rebuild. By focusing on substance and on the journey schools are on, it will be much easier to allow for the effects of sustained disruption. It’s a fair and helpful way to look at how schools are doing.

No one is under any illusions about the scale of the challenge ahead.

With our research and evaluation programme, we hope to make a significant contribution in helping the education sector move forward. I’ve mentioned some of the programme already, but there’s quite a lot more in the pipeline. In schools, we are also researching:

• pupil mobility, including movements into home education
• the purpose of alternative provision
• reading in secondary school

In further education, we are looking at:

• prison education, starting with reading
• how the government’s T-level reforms are working

And in social care, we are also looking at how decisions about care leavers get made.

But all this work will only have real value if it helps you, the teachers. And the fact that you are here today shows that you want to be part of the all-important translation of research into practice, to make sure children get the very most out of their time at school. To put it simply, our children’s futures depend on you.

Thank you.

Amanda Spielman, Chief Inspector, Ofsted

References:

John Jerrim, 2012. “The socio-economic gradient in teenagers’ literacy skills: how does England compare to other countries?,” DoQSS Working Papers 12-04, Quantitative Social Science – UCL Social Research Institute, University College London.

Meadows, M. and Billington, L. (2005). A review of the literature on marking reliability. Report for the National Assessment Agency by AQA Centre for Education Research and Policy.

Aslin, R., Saffran, J., & Newport, E. (1998). Computation of Conditional Probability Statistics by 8-Month-Old Infants. Psychological Science, 9(4), 321-324.