Missing the mark at GCSE English: the costly consequences of just failing to get a grade C

School students who narrowly fail to achieve a grade C in their GCSE English exam pay a high price, according to new research by Stephen Machin, Sandra McNally and Jenifer Ruiz-Valenzuela.

Getting above or failing to reach thresholds in high-stakes public examinations is an important feature of success or failure in many people's lives. One well-known example is the need to obtain a grade C in English and maths in the age 16 school-leaving exams in England (or Grade 4 in the new system).

This is in part because achievement of good literacy and numeracy skills is recognised as an important output of the education system. It is also because achieving a ‘good pass’ (grade C or better) in these exams has long been recognised as a key requirement for employment. In fact, this level of achievement is deemed so important that since 2015, it has become mandatory for students to repeat the exams if they fail to get a C grade in English or maths and wish to continue in some form of publicly funded education thereafter.

New research by the Centre for Vocational Education Research (CVER) [Discussion Paper 014] analyses the benefits (or costs) for students who just pass (or fail) to meet a key threshold in these exams. More specifically, evidence is presented on the importance of just obtaining a grade C in GCSE English Language (which is the form of English exam undertaken by 72% of students in the cohort under study).

The administrative data that we use follow the cohort that took the GCSE exam in 2013 over the next three years of their lives. Comparing students on the threshold of success and failure makes it possible to explore whether just passing or just failing has consequences for them in relation to their probability of early drop-out from education (and employment) and their probability of accessing higher-level courses, which are known to have a positive wage return in the labour market. The analysis also looks at the effect on the probability of entering higher education.

The question is not so much whether it is important to perform well in English, as whether it is important to get past the specific threshold of a grade C. In other words, the focus is on isolating the effects of good or bad luck, which lead a student to end up on either side of the C threshold. Up to now this has not been evaluated empirically, even though getting a grade C in English is given great weight within English institutions and in public conversation.

Our study makes use of the distribution of exact marks around the important threshold of grade C, using data provided by one of the four national awarding bodies (the AQA). One key feature of English exams is the right to appeal, and while the administrative data contain final (post-appeal) grades (i.e. from A* to G), we have also obtained access to student-level data on the pre-appeal and post-appeal marks. Marks range from 0 to 300, where the C threshold lies at 180 marks.

This is important since we can use these data to ascertain whether or not what looks like manipulation in the data is actually due to the re-grading process through appeals. Our research is unique in having the ‘pre-manipulation’ and ‘post-manipulation’ distribution of marks for the same students.

The findings we report show that just failing to achieve a grade C in English has a large associated cost. Put another way, the marginal student would have performed significantly better in the longer term had he or she not been so unlucky at this point. The results show that narrowly missing the C grade in English language decreases the probability of enrolling in a higher-level qualification by at least 9 percentage points (illustrated in Figure 1). There is a similarly large effect on the probability of achieving a higher (‘full level 3’) academic or vocational qualification by age 19 – which is a pre-requisite for university or getting a job with good wage prospects. There is also an effect on the probability of entering tertiary or higher education.

Perhaps most surprisingly, narrowly missing a grade C increases the probability of dropping out of education at age 18 by about 4 percentage points (in a context where the national average is 12%) – illustrated in Figure 2. It increases the probability of becoming ‘not in education, training or employment’ by about 2 percentage points. Those entering employment at this age (and without a grade C in English) are unlikely to be in jobs with good progression possibilities. If they are ‘not in education, employment or training’, this puts them at a high risk of wage scarring effects and crime participation resulting from youth unemployment in the longer term.

We show some evidence on the mechanisms through which failing to obtain a grade C in English leads to poor outcomes. These involve a narrowing of opportunities that arise within the educational system on the choice of post-16 institution and course the year after failing to get a C grade in GCSE English: students end up in institutions with less well performing peers.

In a well-functioning education system, there would be ladders for the marginal student – or at least alternative educational options with good prospects. The CVER study suggests that the marginal student who is unlucky pays a high price.

Our analysis does not suggest that having pass/fail thresholds are undesirable. Achievement of a minimum level of literacy and numeracy in the population is an important social and economic objective. But the fact that there are such big consequences from narrowly missing out on a C grade suggests that there is something going wrong within the system. It suggests that young people are not getting the support they need if they fail to make the grade (even narrowly).

It also suggests that other educational options available to people who cannot immediately enter higher academic or vocational education are failing to help a significant proportion of young people make progress up the educational ladder. Thus, it is symptomatic of an important source of inequality in education, with associated negative long-term economic consequences for young people who just fail to pass such an important high-stakes national exam taken at the end of compulsory schooling.

Choosing the best counterfactual for assessing the returns to qualifications

Sophie Hedges of London Economics summarises a new paper which finds that, for both males and females, non-achievers are generally closer in their observable characteristics to the achievers, than are individuals who only complete the qualification at the level below.”

What’s new?

The labour market returns to qualifications have typically been estimated by comparing the wages of individuals who achieve a particular qualification with the wages of two contrasting counterfactual groups: either level-below (i.e. similar individuals in possession of the qualification at the level below); or non-achievers (i.e. similar individuals who start studying the qualification but then fail to achieve).

The choice of counterfactual used has typically been driven by the data available. Achievement at level-below has generally been used when working with survey data such as the LFS since these surveys only gather information on qualifications achieved. In contrast, non-achievers have been utilised as the main counterfactual group when using administrative data as (until recently) this only covered learners with some interaction with the Further Education system (and reported no, or very limited, information on school and higher education attainment). However, using information from the new Longitudinal Education Outcomes (LEO) administrative data, it is now possible to directly compare the estimates of returns to qualifications from both types of counterfactual using a common dataset.


What do we do?

The objective of CVER Discussion Paper 013 is to understand which counterfactual group (level-below or non-achievers) is relatively more suited for estimating the returns to qualifications in that it comprises individuals who are most similar to those who successfully complete the qualification of interest. In order to do that, we match each individual who achieved the qualification (the treatment group) with their most similar counterpart within the pooled counterfactual group encompassing both non-achievers and achievers at the level below. We then compare the composition of the combined counterfactual group pre-match and post-match (i.e. the sub-sample of the pre-match pooled counterfactual group who were paired with treated individuals). If one counterfactual group (non-achievers or level-below) is over-represented in the post-match sample, then there is a relative preference for that particular control group.

Whilst this process identifies the most similar individuals for the counterfactual in terms of their observable characteristics, it is important to note that it still cannot address the problem of unobservable differences between the treatment and control groups (e.g. innate ability, motivation etc.).

What do we find?

For both males and females, non-achievers are generally closer in their observable characteristics to the achievers, than are individuals who only complete the qualification at the level below. This is particularly true for apprenticeships (both Advanced and Intermediate), NVQs at Levels 2 and 3, and BTECs at Level 3.

How does the counterfactual affect the earnings differentials for vocational qualifications?

We then explored whether estimates of earnings differentials for vocational qualifications vary significantly across the two different counterfactuals. The findings indicate that estimates of earnings differentials using the non-achievers counterfactual group are positive for men achieving vocational qualifications at Levels 3 and 2, although the magnitude is (often considerably) smaller than the earnings differentials estimated using achievement at the level below as the counterfactual (the exception are individuals holding BTECs). For females, the estimated differentials are also positive for all vocational qualifications at Levels 3 and 2, but there is not such a strong pattern in terms of magnitude relative to the estimates using the level-below group as the counterfactual.