Choosing the best counterfactual for assessing the returns to qualifications

Sophie Hedges of London Economics summarises a new paper which finds that, for both males and females, non-achievers are generally closer in their observable characteristics to the achievers, than are individuals who only complete the qualification at the level below.”

What’s new?

The labour market returns to qualifications have typically been estimated by comparing the wages of individuals who achieve a particular qualification with the wages of two contrasting counterfactual groups: either level-below (i.e. similar individuals in possession of the qualification at the level below); or non-achievers (i.e. similar individuals who start studying the qualification but then fail to achieve).

The choice of counterfactual used has typically been driven by the data available. Achievement at level-below has generally been used when working with survey data such as the LFS since these surveys only gather information on qualifications achieved. In contrast, non-achievers have been utilised as the main counterfactual group when using administrative data as (until recently) this only covered learners with some interaction with the Further Education system (and reported no, or very limited, information on school and higher education attainment). However, using information from the new Longitudinal Education Outcomes (LEO) administrative data, it is now possible to directly compare the estimates of returns to qualifications from both types of counterfactual using a common dataset.


What do we do?

The objective of CVER Discussion Paper 013 is to understand which counterfactual group (level-below or non-achievers) is relatively more suited for estimating the returns to qualifications in that it comprises individuals who are most similar to those who successfully complete the qualification of interest. In order to do that, we match each individual who achieved the qualification (the treatment group) with their most similar counterpart within the pooled counterfactual group encompassing both non-achievers and achievers at the level below. We then compare the composition of the combined counterfactual group pre-match and post-match (i.e. the sub-sample of the pre-match pooled counterfactual group who were paired with treated individuals). If one counterfactual group (non-achievers or level-below) is over-represented in the post-match sample, then there is a relative preference for that particular control group.

Whilst this process identifies the most similar individuals for the counterfactual in terms of their observable characteristics, it is important to note that it still cannot address the problem of unobservable differences between the treatment and control groups (e.g. innate ability, motivation etc.).

What do we find?

For both males and females, non-achievers are generally closer in their observable characteristics to the achievers, than are individuals who only complete the qualification at the level below. This is particularly true for apprenticeships (both Advanced and Intermediate), NVQs at Levels 2 and 3, and BTECs at Level 3.

How does the counterfactual affect the earnings differentials for vocational qualifications?

We then explored whether estimates of earnings differentials for vocational qualifications vary significantly across the two different counterfactuals. The findings indicate that estimates of earnings differentials using the non-achievers counterfactual group are positive for men achieving vocational qualifications at Levels 3 and 2, although the magnitude is (often considerably) smaller than the earnings differentials estimated using achievement at the level below as the counterfactual (the exception are individuals holding BTECs). For females, the estimated differentials are also positive for all vocational qualifications at Levels 3 and 2, but there is not such a strong pattern in terms of magnitude relative to the estimates using the level-below group as the counterfactual.