Jacqui Molkenthin

One of the most common areas of difficulty I find when working with End-point Assessment Organisations (EpAOs) is in the development (design and testing) of their tools and materials. Some EpAOs lack the confidence or ability to design assessment materials appropriate to the level of the apprenticeship assessment plan they intend to deliver.

When I discuss ‘levels’ with EpAOs, I get a mixed response, usually one of the following:

  • the level has already been set by the Institute for Apprenticeships and Technical Education (IfATE) and trailblazer group, so if the EpAO follows the assessment plan the assessment will be at the correct level;
  • Ofqual condition E9 (qualification levels) does not apply to EpAOs;
  • the assurance of accuracy in the level rests with the External Quality Assurance (EQA) provider, whose remit during the readiness checks, as detailed in the EQA framework, table 3 and 4, is to check that materials are appropriate to the standard and level of the apprenticeship;
  • we don’t know how to.

So, let me tackle each of those responses in turn:

1) Level set by the IfATE and Trailblazer group

Apprenticeship standards and assessment plans are, rightly, designed by industry experts through the trailblazer groups, using the IfATE guidance and associated Ofqual level descriptors for qualifications and occupational competence. However, the inclusion of assessment experts within the trailblazer groups, is not mandated, which means that the groups do not have guaranteed expertise on what levels mean in practice, and how to ensure the application of levels across the design, descriptors and language of the standard and assessment plan.

The other factor to remember is that assessment plans are at a high level: for example, they do not set the questions and answers for tests or professional discussions / interviews, or the detail of the tasks for observations or projects. Under these conditions, there is no guarantee of the correct level in end-point assessment design simply by using the assessment plan. I have worked with over 50 assessment plans during the past 6 years, and have seen a huge variation in the quality and detail contained within assessment plans, which has a direct knock on to the EpAOs who are using the assessment plans to design the assessment tools and materials.

2) Ofqual Condition E9 does not apply

It is correct that Ofqual Condition E9 does not apply to EpAOs. However, the EPA Qualification Level Condition EPA4.1 refers specifically to levels and must be adhered to when designing assessment tools and materials. One could argue that Ofqual Condition E9 actually applies to the IfATE and trailblazer design groups (assigning, reviewing and managing level). This could explain the rationale behind the design of IfATE’s EQA framework and the details of the IfATE’s consideration of levels for assessment delivered by EpAOs.

3) Role of the EQA provider

EQA providers are there to check compliance with requirements, including setting assessments at an appropriate level, it is not their role to show an EpAO how to design and set assessments at the right level. IfATE’s EQA framework establishes how apprenticeship end-point assessments must be externally quality assured. EQA’s must test the readiness of the EpAO to deliver assessments, conduct appropriate monitoring, feedback and provide an annual summary report on each EpAO to IfATE’s Quality Assurance Committee per standard, help EpAOs’ to be compliant and support consistency and continuous improvement, escalating issue to IfATE if mitigation has not achieved the expected outcome.

4) Knowledge and understanding of how to design to the correct level

Just like Trailblazer Design groups, many EpAOs have been borne out of industry expertise and demand, and are not traditional awarding organisations (of the 325 EpAOs, only 51 are Ofqual recognised awarding organisations). This means that they may not have been exposed to the concept and descriptors of levels in designing the assessments, resulting in a lack of understanding of how to apply levels to assessment design.  

So why is it so important to understand levels?

End-point assessment must be robust, valid, reliable, comparable and consistent. If an EpAO designs and tests their assessment tools and materials based on the assessment plan alone, without a clear understanding of levels, there is a significant risk of inconsistency in assessment design which in turn impacts the validity of assessment.

Now you understand the risk, I’ll introduce you to some of the ways and methods of understanding and embedding the correct level in assessment tools design. To do this I have teamed up with David Jenkins-Handy of Agora Business Consulting with 20 years’ experience in qualification and assessment design. We will follow this up with further articles and training.

One of the most commonly known models for setting learning outcomes is that of Blooms Taxonomy, and replicated here are the categories for the cognitive levels, alongside some common verbs used for each category:

  • Remembering: label, recite, recognise, define, recall, list, quote
  • Understanding: describe, annotate, search, interpret, compare, summarise, explain, restate
  • Applying: demonstrate, present, apply, solve, calculate
  • Analysing: plan, organise, associate, classify, analyse, critique, prioritising
  • Evaluating: evaluate, assess, critique, justify, compare and contrast, relate, recommend, validate
  • Creating: develop, solve, modify, generate, negotiate, formulate, create, compose

Using Bloom’s Taxonomy helps assessment designers establish the number of questions that can be applied to an examination within a cognitive level; so, how many recall/remember questions versus how many application questions in an examination. If you translate this model into end-point assessments, we can develop an understanding of what this means for different types of assessment methods, for example:

 Multiple Choice Questions (MCQs)

  • At level 3 the focus of MCQ tests/examinations concerns the apprentice being given opportunities to demonstrate knowledge and understanding relevant to their occupation.
  • At level 4 the focus of MCQ tests/examinations shifts to knowledge within a context (so, for example, what health and safety legislation/regulation applies in a specific scenario) or how a particular activity should be carried out under given circumstances (so, a question might be based on what would you do if…).

Observations

  • At level 3 an Assessor should be able to find evidence that an apprentice exhibits sound knowledge and has a good level of understanding and can suggest ways to apply knowledge effectively.
  • At a level 4 an Assessor should be able to find evidence that an apprentice can articulate in detail how to apply knowledge and understanding (so, what they know and what they do are closely linked in the execution of tasks). They should also be familiar with at least one analytical method to gather and scrutinise information that facilitates the production of evidence for more senior individuals to base their decisions on (so, accuracy, validity and authenticity become important at this level).

Professional discussion

  • At level 3 the discussion would allow the apprentice to express in some detail how they go about their job, the processes carried out, the procedures applied and the specific working practices that are involved.
  • At level 4 the apprentice would be expected to know the processes, procedures and working practices in detail, but could also clearly explain methods and techniques applied in practice, what works well and what works less well in a given context, and how this shapes their ability to produce the evidence or outputs necessary for success in their role.

To aid understanding further, here is one example of why getting the level right is so important in assessment design to ensure reliability, validity and consistency:

In the passenger transport driver level 2 apprenticeship (STO338) there are two parts to the end-point assessment, an observation and a professional review. Skill S27 is assessed via the professional review and the skill requires the apprentice to “Prepare and submit documents, reports and logs containing performance, incident and technical information”.

The assessment plan gives further detail on what the pass criteria looks like, but unfortunately it does not provide any real exemplification on the skill detailed in the standard (pass = “Able to prepare and submit documentation containing performance, incident and technical information”). If you applied Bloom’s Taxonomy to Ofqual’s level descriptors, what could a professional review question look like?

  • List the reports you must complete at the start/end of your shift.
  • Identify the reports you must complete, and describe the reason you must complete them, at the start/end of your shift.
  • Identify how reports completed at the start/end of your shift contribute to the management of risks.

In broad terms, question 1 would be a level 1 question, question 2 a level 2 question, and question 3 a level 3 question. But it is important to remember that it is not just the question that must be set at the right level, the EpAO must also set out the expectations of learners’ responses, appropriate to the level, within the assessor guidance. If the EpAO designed and used a question at the wrong level, and/or set the expectation of the answer at the wrong level, then the assessment would not be valid or fair, risking adverse effects on the apprentice, and reputational damage to the EpAO and for apprenticeships.

The example provided above relates to the theory of design, but there are also practical methods of ensuring the correct level of assessment and assessment judgement, and thus the validity and reliability of assessment. Examples include: (a) interrater reliability, where the testing of scoring takes place to ensure that the assessment judgements are set at the right level as well as the questions themselves; and (b) parallel-form, developing two tests covering the same KSB and comparing the results to test reliability. Another article will be written to explain these in more detail.

We hope that this has provided a useful introduction. You may also have realised, when reading this article, that the concept of levels is equally relevant to apprenticeship training providers when designing their apprenticeship programme / curriculum based on the apprenticeship standard. We will therefore be developing and delivering training and further articles for both EpAOs and training providers on levels during the spring and summer of 2021.

Jacqui Molkenthin, JEML Consulting

You may also be interested in these articles:

Sponsored Video

#SkillsWorldLive Video: Why is it important to have diversity in apprenticeships?

Register, Login or Login with your Social Media account:


Advertisers

Upcoming FE Events

Advertiser Skyscrapers

Newsroom Activity

Edge Future Learning - Making education relevant...

Building on our work with Edge Future Learning Leaders and Partners around the world, our model includes three key principles that help bring...

Latest Education News

Further Education News

The FE News Channel gives you the latest education news and updates on emerging education strategies and the #FutureofEducation and the #FutureofWork.

Providing trustworthy and positive Further Education news and views since 2003, we are a digital news channel with a mixture of written word articles, podcasts and videos. Our specialisation is providing you with a mixture of the latest education news, our stance is always positive, sector building and sharing different perspectives and views from thought leaders, to provide you with a think tank of new ideas and solutions to bring the education sector together and come up with new innovative solutions and ideas.

FE News publish exclusive peer to peer thought leadership articles from our feature writers, as well as user generated content across our network of over 3000 Newsrooms, offering multiple sources of the latest education news across the Education and Employability sectors.

FE News also broadcast live events, podcasts with leading experts and thought leaders, webinars, video interviews and Further Education news bulletins so you receive the latest developments in Skills News and across the Apprenticeship, Further Education and Employability sectors.

Every week FE News has over 200 articles and new pieces of content per week. We are a news channel providing the latest Further Education News, giving insight from multiple sources on the latest education policy developments, latest strategies, through to our thought leaders who provide blue sky thinking strategy, best practice and innovation to help look into the future developments for education and the future of work.

In Jan 2021, FE News had over 173,000 unique visitors according to Google Analytics and over 200 new pieces of news content every week, from thought leadership articles, to the latest education news via written word, podcasts, video to press releases from across the sector, putting us in the top 2,000 websites in the UK.

We thought it would be helpful to explain how we tier our latest education news content and how you can get involved and understand how you can read the latest daily Further Education news and how we structure our FE Week of content:

Main Features

Our main features are exclusive and are thought leadership articles and blue sky thinking with experts writing peer to peer news articles about the future of education and the future of work. The focus is solution led thought leadership, sharing best practice, innovation and emerging strategy. These are often articles about the future of education and the future of work, they often then create future education news articles. We limit our main features to a maximum of 20 per week, as they are often about new concepts and new thought processes. Our main features are also exclusive articles responding to the latest education news, maybe an insight from an expert into a policy announcement or response to an education think tank report or a white paper.

FE Voices

FE Voices was originally set up as a section on FE News to give a voice back to the sector. As we now have over 3,000 newsrooms and contributors, FE Voices are usually thought leadership articles, they don’t necessarily have to be exclusive, but usually are, they are slightly shorter than Main Features. FE Voices can include more mixed media with the Further Education News articles, such as embedded podcasts and videos. Our sector response articles asking for different comments and opinions to education policy announcements or responding to a report of white paper are usually held in the FE Voices section. If we have a live podcast in an evening or a radio show such as SkillsWorldLive radio show, the next morning we place the FE podcast recording in the FE Voices section.

Sector News

In sector news we have a blend of content from Press Releases, education resources, reports, education research, white papers from a range of contributors. We have a lot of positive education news articles from colleges, awarding organisations and Apprenticeship Training Providers, press releases from DfE to Think Tanks giving the overview of a report, through to helpful resources to help you with delivering education strategies to your learners and students.

Podcasts

We have a range of education podcasts on FE News, from hour long full production FE podcasts such as SkillsWorldLive in conjunction with the Federation of Awarding Bodies, to weekly podcasts from experts and thought leaders, providing advice and guidance to leaders. FE News also record podcasts at conferences and events, giving you one on one podcasts with education and skills experts on the latest strategies and developments.

We have over 150 education podcasts on FE News, ranging from EdTech podcasts with experts discussing Education 4.0 and how technology is complimenting and transforming education, to podcasts with experts discussing education research, the future of work, how to develop skills systems for jobs of the future to interviews with the Apprenticeship and Skills Minister.

We record our own exclusive FE News podcasts, work in conjunction with sector partners such as FAB to create weekly podcasts and daily education podcasts, through to working with sector leaders creating exclusive education news podcasts.

Education Video Interviews

FE News have over 700 FE Video interviews and have been recording education video interviews with experts for over 12 years. These are usually vox pop video interviews with experts across education and work, discussing blue sky thinking ideas and views about the future of education and work.

Events

FE News has a free events calendar to check out the latest conferences, webinars and events to keep up to date with the latest education news and strategies.

FE Newsrooms

The FE Newsroom is home to your content if you are a FE News contributor. It also help the audience develop relationship with either you as an individual or your organisation as they can click through and ‘box set’ consume all of your previous thought leadership articles, latest education news press releases, videos and education podcasts.

Do you want to contribute, share your ideas or vision or share a press release?

If you want to write a thought leadership article, share your ideas and vision for the future of education or the future of work, write a press release sharing the latest education news or contribute to a podcast, first of all you need to set up a FE Newsroom login (which is free): once the team have approved your newsroom (all content, newsrooms are all approved by a member of the FE News team- no robots are used in this process!), you can then start adding content (again all articles, videos and podcasts are all approved by the FE News editorial team before they go live on FE News). As all newsrooms and content are approved by the FE News team, there will be a slight delay on the team being able to review and approve content.

 RSS IconRSS Feed Selection Page