Reformed functional skills mathematics: evaluation of difficulty
This research was carried out pre-pandemic, in 2019, as part of our normal technical evaluation.
Functional skills maths teachers judged how difficult their students would find individual questions from both live legacy assessments and the sample assessments for reformed qualifications using a comparative judgement technique. From this we obtained measures of expected difficulty for all questions.
We found only small differences in expected difficulty between sample assessments offered by each awarding organisation, which could be accommodated when setting pass boundaries. We also found the expected difficulty between legacy and reformed qualifications was very similar.
More recently, some education and training representative bodies have commented they believe reforms to functional qualifications in 2019 have made them more difficult. Ofqual’s Chief Regulator, Dr Jo Saxton, has heard the concerns that people within the sector have raised. Ofqual will continue to monitor and evaluate these qualifications and to listen to the sector.
Executive summary
The reformed versions of functional skills qualifications were available for first teaching from September 2019. The reform of these qualifications followed decisions made by the Department for Education to ensure the qualifications better met employer needs in terms of the knowledge and skills achieved by learners, but not to change the demand of the qualification. Some of the main changes to the functional skills maths qualification were the introduction of a non-calculator paper and a focus on underpinning skills, referring to fundamental mathematical knowledge. These changes aimed to put more emphasis on skills required by both learners and employers, and build the qualifications’ recognition and credibility among employers.
Ofqual’s rules require awarding organisations (AOs) to ensure comparability over time, and with other AOs’ qualifications. The work reported here was carried out in 2019 as part of our technical evaluation of assessment materials for these new qualifications. The study aimed to assess the difficulty of functional skills sample assessments between AOs and compared to legacy functional skills qualifications offered prior to the reforms. This was done using a comparative judgement methodology. We used expert judgement of items to derive estimates of the expected difficulty of functional skills maths items from sample assessments for the reformed qualifications together with items from legacy assessments.
We found that the expected difficulty of items in legacy assessments and reformed sample assessments were very similar. The close similarity of both the average (median) and the overall distribution of item difficulties suggests that AOs have remained consistent when setting the difficulty of their assessments following reform. This is encouraging as the reforms were aimed at introducing improved, new-style content but not to increase demand, which this study suggests has been achieved.
Considering only the reformed qualifications, each AO’s sets of assessments had a clear separation of difficulty between the level 1 and level 2 qualifications. Overall, the assessments were of comparable difficulty across AOs. The findings of this study suggest that using the sample assessments as a guide, AOs will be able to set tests pitched at the appropriate difficulty and with adequate separation between levels. Moderate adjustments of pass boundaries should be sufficient for ensuring comparability between functional skills maths tests between and within AOs.
Analysis of ‘median data’ is nonsense when the mocks papers provided are not a reflection of the difficulty of the exams. When the pass mark is 51%!!!! something is very wrong. Learners who are experts in their sector are unable to complete courses due to tests, that are not fit for purpose, being far too difficult. We should be looking at national first time pass rates, and talking to actual learners and FS tutors delivering the quals. It’s heartbreaking seeing a learner who is trying, progressing, completing mocks with pass marks, then unable to achieve the real thing after many attempts. I know of no FS tutors who think reforms have been positive.
‘using the sample assessments’ – yeah, this doesn’t work. Due to the timescale of the introduction of the reforms, the sample assessments were a rush job and not based accurately on what the curriculum actually entailed – this came from the head of functional skills at C&G, so I should think she knows what she’s talking about. This is just another example of the governmental department holding the telescope up to its blind eye and saying ‘I see no ships’. no one has access to the live papers but the examining bodies and the students and the word from the students is ‘this is very hard’. I trust my students more than the examining bodies.