The need for representative samples in quantitative research
Just having large numbers of people responding to your surveys is not enough. The way in which people are selected to take part is the basis of the reliability of the results. While 75,000 people may go to the cup final no-one would suggest that they represent the public as a whole. You need to make sure that the sample is not biased towards those more interested in the topic or people who in some way have the same characteristics. Everyone who is eligible to be included in your sample should have an equal chance of taking part in your research, but the costs of this can be high and there is a cost-quality trade-off in sampling approaches. Sampling methods often focus on specific characteristics (including location) but once the overall characteristics have been identified, there should be no reason why one person who meets the criteria is selected over another, apart form chance.
The questions you need to ask yourself are: when I find out that 80% of my sample hold a particular view, what does this mean? Can I say 80% of the whole group in which I am interested (for example, 80% of all UK academics believes x) or can I only say 80% of those academics who answered my questionnaire believe x? Social survey researchers have spent decades perfecting techniques that ensure that survey results are representative of the whole group whose views and/or behaviour they want to understand.
The most famous story is that of George Gallup of Gallup Surveys. In 1936 The Literary Digest of America poll said Roosevelt would lose the forthcoming election, 56% to 44%. George Gallup predicted Roosevelt would win – and he did, by a landslide.
The Literary Digest posted questionnaires to millions of households across the USA asking them to select the candidate they would be voting for and to post the questionnaire back. Gallup conducted polls of a sample of about 2,000 people, each one selected according to the rules of survey research. The Digest selected potential respondents using registers of phone and car numbers. 1936 was the middle of the Depression and millions of potential voters did not have phones or cars. The sample was biased towards the better off, and this was reflected in the result of the poll.
PSP works within a rigorous survey research framework when undertaking quantitative research to ensure that the sample is robust and results therefore reliable and able to stand-up to scrutiny.