This research was carried out pre-pandemic, in 2019, as part of our normal technical evaluation.
Functional skills maths teachers judged how difficult their students would find individual questions from both live legacy assessments and the sample assessments for reformed qualifications using a comparative judgement technique. From this we obtained measures of expected difficulty for all questions.
We found only small differences in expected difficulty between sample assessments offered by each awarding organisation, which could be accommodated when setting pass boundaries. We also found the expected difficulty between legacy and reformed qualifications was very similar.
More recently, some education and training representative bodies have commented they believe reforms to functional qualifications in 2019 have made them more difficult. Ofqual’s Chief Regulator, Dr Jo Saxton, has heard the concerns that people within the sector have raised. Ofqual will continue to monitor and evaluate these qualifications and to listen to the sector.
The reformed versions of functional skills qualifications were available for first teaching from September 2019. The reform of these qualifications followed decisions made by the Department for Education to ensure the qualifications better met employer needs in terms of the knowledge and skills achieved by learners, but not to change the demand of the qualification. Some of the main changes to the functional skills maths qualification were the introduction of a non-calculator paper and a focus on underpinning skills, referring to fundamental mathematical knowledge. These changes aimed to put more emphasis on skills required by both learners and employers, and build the qualifications’ recognition and credibility among employers.
Ofqual’s rules require awarding organisations (AOs) to ensure comparability over time, and with other AOs’ qualifications. The work reported here was carried out in 2019 as part of our technical evaluation of assessment materials for these new qualifications. The study aimed to assess the difficulty of functional skills sample assessments between AOs and compared to legacy functional skills qualifications offered prior to the reforms. This was done using a comparative judgement methodology. We used expert judgement of items to derive estimates of the expected difficulty of functional skills maths items from sample assessments for the reformed qualifications together with items from legacy assessments.
We found that the expected difficulty of items in legacy assessments and reformed sample assessments were very similar. The close similarity of both the average (median) and the overall distribution of item difficulties suggests that AOs have remained consistent when setting the difficulty of their assessments following reform. This is encouraging as the reforms were aimed at introducing improved, new-style content but not to increase demand, which this study suggests has been achieved.
Considering only the reformed qualifications, each AO’s sets of assessments had a clear separation of difficulty between the level 1 and level 2 qualifications. Overall, the assessments were of comparable difficulty across AOs. The findings of this study suggest that using the sample assessments as a guide, AOs will be able to set tests pitched at the appropriate difficulty and with adequate separation between levels. Moderate adjustments of pass boundaries should be sufficient for ensuring comparability between functional skills maths tests between and within AOs.