How to Seek and Analyse Feedback When Designing Apprenticeship Assessment
In the current apprenticeship end-point assessment model, end-point assessment organisations (known as awarding organisations to Ofqual – AOs) collect feedback to support the ongoing review, continued appropriateness, and improvement of their end-point assessments (Condition D3). As apprenticeship assessment moves into a new model, AOs will need to use feedback not only to inform review and improvement, but also to shape the initial design of assessments.
This shift is driven by two key changes. Firstly, the new assessment plans will be higher level, requiring AOs to take greater responsibility for detailed assessment design work. Secondly, Ofqual proposes to stop disapplying Condition E1, which will mean that apprenticeship assessments must have an objective and support.
Within the proposed apprenticeship assessment qualification level guidance, Ofqual have provided an example of how an AO may meet Condition E1 “when designing the Apprenticeship Assessment, seeking and analysing feedback from employers, employer groups or professional bodies who have an interest in the Apprenticeship Assessment – this may include groups of employers convened by Skills England”. However, across a number of Ofqual and Skills England webinars, AOs have raised concerns and questions about the proposal:
- (a) There is concern about potential duplication of effort – Skills England already work with employer groups when designing the assessment plans. Linked to this is the risk of employer fatigue, particularly if employers are asked to provide feedback both to Skills England and to multiple AOs.
- (b) Some AOs are concerned about increased bureaucracy and the additional administrative burden this could create.
- (c) Some AOs have highlighted the risk of assessments becoming tailored to individual employers, rather than remaining focused on the needs of the occupation as a whole.
- (d) Questions have been raised about the type and scale of feedback required, and about Ofqual’s expectations regarding the evidence needed to demonstrate compliance.
Whilst I’m not in a position to address the concerns raised, I can share some ideas around the types of feedback to consider and how:
1. Consider All Types Of Users
The Ofqual Conditions refer to “users of qualifications”. This is defined as “Persons who have a legitimate interest in the qualification or type of qualification made available by the awarding organisation, which may include – (a) Learners and Learners’ representatives, (b) Centres, (c) Teachers, (d) employers and employers’ representatives, (e) further and higher education establishments, (f) schools, (g) government departments and agencies, and (h) professional bodies”.
This definition is important to be aware of because across the Ofqual / Skills England webinars, and in the Apprenticeship Assessment Qualification Level Guidance, they have only referred to employers, employer groups or professional bodies. I believe this misses incredibly valuable sources of feedback, the training provider and the Learner.
Whilst employer feedback is critical as the main purpose of apprenticeship assessment is occupational competence, the learner can provide feedback on their experiences of types of assessment and the impact of assessment on their confidence, and the training provider can provide feedback on the logistics and perspectives of assessments types / models. Don’t forget the training provider is also important because they have employer relationships, help prepare apprentices for assessment, and some may become Centres.
2. Focus on Quality Rather than Quantity
An AO does not need to reach out to all users, in fact, in the webinar on 3rd February Ofqual did make the point that it would be hard to reach out to all employers, and/or across the whole country! They also reiterated verbally that Ofqual are not being prescriptive about the amount and type of feedback that they expect an AO to get, and they are not expecting AOs to produce engagement strategies. However, AOs must explain within their qualification level assessment strategies how they have engaged and how that’s helped them to ensure that their apprenticeship assessments are fit for purpose.
3. Focus On Assessment
AOs are being regulated on the apprenticeship assessment, so they do not need to seek feedback on the training aspect of the apprenticeship. However, it may still be relevant, for example, if a training provider delivers the knowledge of an apprenticeship early on they may feedback that any knowledge based assessments are best delivered early as opposed to waiting until [nearer] the end of the apprenticeship. Equally an employer may feedback that the knowledge needs to be assessed early for the assurance of safety when entering the workplace.
4. Carry Out Initial Assessment Design Work Before Requesting Feedback
At the Ofqual webinar on 3rd February, Ofqual suggested focussing on feedback that helps an AO strengthen the apprenticeship assessment, as opposed to lots and lots of feedback that doesn’t really tell them anything about the assessment. I have found that developing a range of assessment options which take into account the Ofqual regulation, funding, and DfE rules has facilitated higher quality and more focussed feedback. Users know what would work best for them / their business, but not always how that would sit in the context of rules and regulations. All options will have pros and cons, so it is important to explore where there may need to be trade-offs to help users prioritise and focus their feedback.
5. Ask Questions Instead of Simply Asking for Opinions
Avoid simply asking for opinions, such as ‘what assessments do you want’, this will leave an AO with multiple perspectives with no real mechanisms to work out what is best. However, if an AO does choose to seek views and opinions, don’t forget to ask why the user said or suggested that, as it can inform an AOs decision making and it makes the user see that their opinion matters.
For example, an employer may wish to use a project to determine learner competence in the workplace, an AO may prefer scenario based activities to ensure consistency and authenticity, and a learner may like a project but worry about whether the employer can provide the right facilities / time / opportunities to run a project. Another example is assessment timings; an employer may prefer early assessment so the learner can take on more responsibilities quicker. Equally a learner may like the idea of an early assessment to build their confidence, but they may also fear an early assessment in terms of readiness. However, if an AO is considering timings, don’t forget there are some DfE funding rules that must be followed in addition to the Ofqual Conditions and assessment plan: “paragraph 350. An apprentice can only undertake their final apprenticeship assessment after: 350.1 They have met the minimum duration of the apprenticeship practical period; 350.2 They have satisfied the requirements set out in the assessment plan and reached gateway to completion (including any specific duration criteria); and 350.3. Their employer (in consultation with the main provider) is content they have attained sufficient skills, knowledge and behaviours to successfully complete the apprenticeship.”
6. Capture Feedback Across a Range
Feedback will vary depending on (a) employer size, employer location and employer age (well-established organisation vs new or emerging organisation); (b) provider type (ITP vs College), provider offering, and whether they already operate as Centres for AOs; (c) the respondent, a response from an individual vs a response from a body representing groups of employers / learners / providers.
It is important to be able to capture and analyse feedback across a range of users and make a balanced decision from the feedback received. Feedback collection methods may include the use of workshops involving a range of types of user to check and challenge views as they emerge, or surveys (to be sent out or used in meetings) with a set of standardised questions which can then be analysed collectively to identify any trends. This leads onto my next recommendation.
7. Create a User Feedback Template
This is not designed to be bureaucratic, it is designed to help an AO capture feedback in a consistent manner and to keep conversations focussed. An AO may even consider detailing the overarching approach to capturing user feedback within their organisation level assessment strategy, placing the detail of the findings shaping the assessment into the qualification level assessment strategy.
8. Remember That Not Everything is Equal
It is important not to let one person or organisation dominate the feedback, but at times it may be necessary to recognise that not all feedback carries equal weight. For example, the risk level of certain types of assessment, non-Ofqual regulatory requirements (such as health and safety), and the need to ensure transferable skills across / within on occupation, may mean that some feedback must be prioritised.
Where this is the case, it is important for the AO to justify the decisions they make around the weighting of feedback. I have found that the Ofqual General Purposes A-C may help with the weighting / prioritisation.
9. Explore Current Feedback
An AO will already have intelligence about assessments from their current assessment work. This is extremely valuable in determining approaches and options when it comes to designing apprenticeship assessment. Have a look at findings from assessors and IQA standardisation activities, and dig down into complaints, appeals and results to see if there are any themes such as the burden of different types of assessment, assessments timings or durations, and so on.
10. Remember It Is Not Always Possible To Make Everyone Happy
Assessment design decisions must be based on ensuring validity, fairness, and reliability. This means that assessment design decisions must be balanced and considered taking into account user feedback, benchmarks, the approaches of other AOs, the requirements of professional bodies, the requirements of the regulator Ofqual, the DfE apprenticeship funding rules, and the DfE Requirements and guidance for apprenticeship assessment.
Responses