The need for representative samples in quantitative research
Just having large numbers of people responding to your surveys is not enough. The way in which people are selected to take part is the basis of the reliability of the results. While 90,000 people may go to the cup final no-one would suggest that they represent the public as a whole. You need to make sure that the sample is not biased towards those more interested in the topic or people who in some way have the same characteristics.
The most famous story is that of George Gallup of Gallup Surveys. In 1936 The Literary Digest of America poll said Roosevelt would lose the forthcoming election, 56% to 44%. George Gallup predicted Roosevelt would win – and he did, by a landslide.
The Literary Digest posted questionnaires to millions of households across the USA asking them to select the candidate they would be voting for and to post the questionnaire back. Gallup conducted polls of a sample of about 2,000 people, each one selected according to the rules of survey research. The Digest selected potential respondents using registers of phone and car numbers. 1936 was the middle of the Depression and millions of potential voters did not have phones or cars. The sample was biased towards the better off, and this was reflected in the result of the poll.
PSP works within a rigorous survey research framework when undertaking quantitative research to ensure that the sample is robust and the results are therefore reliable and able to stand-up to scrutiny.
Sampling methods vary depending on how you plan to collect the data – it may be easy to select a simple ‘1 in N’ sample for a postal, telephone or Internet survey, but if the interview is to be conducted in people’s homes, you need to cluster the sample geographically to make it cost effective. In general there is a trade-off between the quality of the sampling method in terms of representativeness and being able to measure bias on the one hand, and cost on the other.
Social survey researchers have spent decades perfecting techniques that ensure that survey results are representative of the whole group whose views and/or behaviour they want to understand and that any bias introduced by the sampling method can be measured. They also look at how to take account of differential response rates, that is, where some groups respond more then others, for example, a greater proportion of women than of men respond.
When bias can’t be measured you have to ask whether the estimate is ‘good enough’, is it ‘fit for your purpose’? One way to consider this question is to profile the sample responses by certain characteristics and see how that compares with national data for the group. You could then see if you can weight your data to be more representative.