Location data

Analysis of Employment location questions – Postcode and town/city

Introduction and context

Graduates in certain types of employment who state that their place of work is in England, Scotland, Wales, or Northern Ireland during the survey are asked to provide the postcode for their place of work. Whilst this is a mandatory question, there is a response option of ‘don’t know’ that is available for graduates to select. Graduates who select ‘don’t know’ or provide a short postcode arerequired to provide the nearest city or town to their place of work in the next question, whereas the town/city question is optional for graduates who provide a full postcode. Respondents to surveys can be reluctant to provide personal information, particularly if they feel that this information may not be kept confidential or if the questions are administered by an interviewer (Tourangeau and Yan, 2007). This may be exacerbated by more people working from home, potentially making this data feel more personal to them. Equally, respondents may not know certain information about their place of work, and this may be a particular problem for postcode in the CATI completion mode if respondents do not have access to this at the time of the call. 

In year two, validation was added to the postcode question to check the first two digits of the postcode and to ensure the formatting and length were correct. An assessment of the year two data in the 2nd edition of the Graduate Outcomes Survey Quality Report highlighted improvements in the quality of the data collected, likely as a result of the validation but also potentially other factors such as a change in working patterns. However, there was a slight increase in item non-response and whilst levels of ‘don’t know’ selection clearly reduced, many graduates were still selecting this option. Next steps in the report highlighted that efforts should be made to reduce this further. Offering a ‘don’t know’ response option can increase missing data; however, probes have been found to reduce missing data across different survey completion modes without negatively influencing respondent’s attitudes about a survey (DeLeeuw, 2018). In Cohort D of Year three an additional validation pop-up was added when ‘don’t know’ was selected to try and encourage respondents to provide at least a partial postcode, which appeared to have a positive impact on response to the question in the online mode in the third year. Assessment of the impact last year also indicated a positive improvement as a result of the pop-up, with a decrease in ‘don’t know’ selection. Further information can be found in the 4th edition of the quality report.  

Following the findings from previous investigations, we also highlighted that the CATI completion mode may benefit from some further improvement to the postcode question. We suggested that discussion with the contact centre may aid in reducing the prevalence of ‘don’t know’ further, alongside further review of the validation text, as using these probes and messaging correctly can be very effective in reducing the selection of ‘don’t know’ in different ways across modes (DeLeeuw, 2018). As a result, an action plan was put in place for the CATI completion mode during cohort C of Year 4 and the information text was also reassessed to ensure it was clear for interviewers and respondents. Assessments last year indicated that there were reductions in ‘don’t know’ selection, with generally stable item non-response. These had been converted into long and short postcode selections, highlighting positive results due to the changes and action plan. However, we highlighted in the report that tracking would continue for the Year 5 survey to ensure the action plan continued.  

Once graduates have answered the postcode question, they will then move on to the town/city question. Before year three of the survey the question was worded as follows, with the additional text in italics provided for context:

• “What was the town, city or area in which you worked? Please type in the town, city or area where your employment was/will be based and not the county. For example, if your employment was/will be based in London, please give the local area e.g. Holborn.”

From year three onwards the question wording was changed to:

• “What is the (nearest) city or town for your place of work?”

The change to the town/city question aimed to make it easier to answer and to provide continuity both in terms of question tensing and for future comparisons as a list of towns and cities was implemented for year four of the survey. It was also aimed at improving the usability of the data and to ensure that areas provided by graduates can be correctly identified for outputs.

As mentioned, a location list was added to the town/city question in the fourth year of the survey (C20072). HESA worked with devolved administrations and existing location information to develop the list, with the aim of improving the usability of the data and the graduate experience of answering the survey. The list has a search function to ensure the ease of use. We are committed to regularly review and improve the drop-down list, to ensure it is fit for purpose. In depth assessment of the results from the list were ongoing through the year, to determine additions, removals and clarifications that may be required and this work continued, with changes made during Year 5. 

Methods and results

 
Impact of validation on postcode responses

Assessments of the postcode provided to both employment types are in Figure 2 and Figure 3, split by mode. Graduates included in these tables have answered the question before postcode (related to the country of their place of work) as England, Scotland, Wales or Northern Ireland. 

Results are extremely positive this year, with all completion modes and work types seeing a further reduction in the number of  ‘don’t know’ selections.. Indeed, selections of long postcodes have increased across all groups. Levels of ‘not answered’ indicating drop-out have remained very stable with only minor decreases or increases in rates.  CATI has seen the largest improvements, with decreases in levels of ‘don’t know’ selection of over 10 percentage points. This indicates that the CATI action plan has led to improvements in the data being collected for the survey. In terms of the online completion mode, it does seem that the additional validation has continued to benefit the question.

Improvements to the town/city list so far

As mentioned, we are committed to improving the town/city list which was introduced in Year 4. The list continues to undergo quality checking and alterations as required. We are also working with the contact centre on reducing the reliance on ‘other’ free-text use and are using this data to ensure that the list is fit for purpose. 

Initially during Year 5 there were 17 new entries for Northern Ireland. Investigations indicated that coverage could be improved, with some areas included that were not used and other largely populated areas not on the list. Where additions created duplicates in the areas covered, entries were removed from the list. There were also changes to a number of the labels presented to graduates to make it clear when these options should be selected. For example, ‘Greater London’ was replaced with ‘London – borough unknown’, as investigations indicated confusion around which option to select due to the presence of boroughs in the list, and the option to select Greater London. Clarification was also added to ‘City of London’ following a review of the options, to ensure that graduates were aware this was not referring to London as a whole. Country was added to an entry and further clarification was added to a village named ‘Othery’ which seemed to have increased selection, likely as a result of confusion with the option ‘other’. ‘Greater Manchester’ and ‘Greater Glasgow’ were also removed, as other areas at this level were not available to respondents, and this allows more granular mapping. Research into current selection of these options indicated that graduates do have other options to select from the list that are more relevant to their area, and that removal would not have a detrimental impact on data collection.  

Whilst not related to the newly published data, for further information, another assessment occurred ahead of the Year 6 (22/23) collection which is currently surveying graduates. Changes have been applied, with plans to continue improving the list. Changes can be viewed in the survey results coding manual revision history. 

Impact of the changes to the employment questions on location mapping

Following the introduction to the list in Year 4, over 97% of graduates were able to be mapped from a postcode or a drop-down list entry. This was a vast improvement on previous years, where only free-text responses were provided to town/city. The change has greatly reduced the amount of free-text mapping required. For further details on the levels of mapping between years, see the in the section regarding location of work data from the next section on Processing error.  

Conclusions

Across both completion modes, the quality of postcode data has continued to improve, likely as a result of the additional validation and CATI action plan. This has led to a further reduction in the selection of ‘don’t know’  for both employment types. Both modes and work types saw increased selection of long postcodes predominantly, with item non-response remaining stable.  

When considering the ability to map graduates to an area of the country, improvements are evident as a result of the introduction of the town/city list. We will continue to work on improvements to the list, to improve usability and attempt to reduce the use of ‘other’ online. We will also continue to work with the contact centre on reducing the use of ‘other' on the CATI completion mode. 

Next: Processing error

References

DeLeeuw, E.D., 2018, August. Mixed-mode: Past, present, and future. In Survey Research Methods (Vol. 12, No. 2, pp. 75-89).

Tourangeau, R. and Yan, T., 2007. Sensitive questions in surveys. Psychological bulletin, 133(5), p.859.