Over recent times, our federal member, Andrew Laming, has used surveys as a means to gather constituents’ opinion.
Fine in theory. But while he proclaims that the surveys reveal what the people think and want, these surveys are entirely amateurish and produce meaningless data – not useful information.
Unfortunately for us all, they are a significant waste of time, paper and taxpayer dollars. Garbage in, garbage out.
Professional survey design and surveying techniques are based on research methodology and psychometrics, and are generally undertaken by independent research personnel who specialise in this science. This is not what happens in Redlands, at the behest of our MP.
Survey design principles
Legitimate surveys are based on many important design principles, including:
- having a clear and genuine purpose
- identification of survey ‘population’ to be included in sample [e.g. every single constituent, or only those who have a post office box?]
- very careful choice of wording/phrasing and structuring of questions to avoid bias
- inclusion of suitable response options [e.g. what they are, how many are included, is there space for non-standard responses?, etc.]
- selection of best overall survey strategy [e.g. interview or paper questionnaire?], as well as ways of responding [tick-a-box, written answers, etc.]
- determination of appropriateness [or not] of respondent anonymity
- how the questionnaires are distributed [‘broadcast’ e.g. letterbox drop versus individually targeted via personal interview or letter]
- whether the survey is seen to be handled professionally and independently/ without any overt or hidden agenda, or whether [as in Laming’s surveys] they are clearly politically biased
- how the results are analysed and then reported
- what the respondent sample size is and their demographic spread, compared with the number actually surveyed
How the results are fed back to us is just as significant as the survey design. For instance, we have no way to assess what the reported results mean when we are not told how many people actually responded to the questionnaire. What we are told is that a certain percentage of Redlands respondents voted a certain way. If the number of people who replied to the survey is very low (say, 20), then it’s absurd to trumpet that “47% want a bridge to Russell Island” – since it really means that only about 10 people think that way! And are those 10 people all from Russell Is.? We are never told how many people responded to Laming’s surveys, or what the demographic array is.
It is notoriously difficult to design questionnaires that really deliver the goods – namely, objectively valid data. But without valid data, there is no meaningful basis for the use of the survey results for subsequent decision-making and planning. It’s a doddle to concoct a questionnaire that delivers desired results – any fool can do that. And it produces results as superficial as did the quizzes that used to be the mainstay of cheap women’s magazines.
In mid-February of this year, we constituents were proudly informed by Andrew Laming of the results of ‘Redland’s Biggest Survey.’ However, when such a patently simplistic method is used with questions that are limited and obviously ‘loaded,’ only the naïve or ignorant would regard the results as having any objective validity.
I was prepared to believe that Laming’s intentions were good, although his actions and survey methodology were misguided and based on a poor understanding of proper survey design methodology. So I wrote to outline my concerns. These concerns were met with lame excuses and attempts at justification [eg. “It is …all I can do with the budget allowed.”] He told me the questions were taken from the community “and in most cases they don’t want us editing!”
My conclusion from Laming’s comments is that he is not qualified in the arena of survey methodology himself, nor has he delegated the task to a suitably qualified professional survey designer. His own comments reveal that he believes getting a cheap, superficial [= invalid] survey concocted by amateurs is better than none at all. Not true, Andrew! It’s misleading and unethical. Such prejudiced methods directly affect who will reply and what replies will come. The fact is that the surveys are biased to get the kinds of responses that suit him.
If he genuinely wants to get VALID DATA on which to base future decisions, policies and actions, I suggest he turn to independent, professional persons or organisations to undertake the work. This could result in credible, valid data of real value and use. Not just the cynical pretence of it, which is current practice.
If our MP is unwilling or unable to organise proper, meaningful research, then I recommend desisting altogether from these prejudiced charades, which harm his credibility and do no service to our community.