Legitimate surveys follow important design principles
Over recent times, our federal member, Andrew Laming, has used surveys as a means to gather constituents’ opinion.
Fine in theory. But while he proclaims that the surveys reveal what the people think and want, these surveys are entirely amateurish and produce meaningless data – not useful information.
Unfortunately for us all, they are a significant waste of time, paper and taxpayer dollars. Garbage in, garbage out.
Professional survey design and surveying techniques are based on research methodology and psychometrics, and are generally undertaken by independent research personnel who specialise in this science. This is not what happens in Redlands, at the behest of our MP.
Survey design principles
Legitimate surveys are based on many important design principles, including:
- having a clear and genuine purpose
- identification of survey ‘population’ to be included in sample [e.g. every single constituent, or only those who have a post office box?]
- very careful choice of wording/phrasing and structuring of questions to avoid bias
- inclusion of suitable response options [e.g. what they are, how many are included, is there space for non-standard responses?, etc.]
- selection of best overall survey strategy [e.g. interview or paper questionnaire?], as well as ways of responding [tick-a-box, written answers, etc.]
- determination of appropriateness [or not] of respondent anonymity
- how the questionnaires are distributed [‘broadcast’ e.g. letterbox drop versus individually targeted via personal interview or letter]
- whether the survey is seen to be handled professionally and independently/ without any overt or hidden agenda, or whether [as in Laming’s surveys] they are clearly politically biased
- how the results are analysed and then reported
- what the respondent sample size is and their demographic spread, compared with the number actually surveyed
Survey reporting
How the results are fed back to us is just as significant as the survey design. For instance, we have no way to assess what the reported results mean when we are not told how many people actually responded to the questionnaire. What we are told is that a certain percentage of Redlands respondents voted a certain way. If the number of people who replied to the survey is very low (say, 20), then it’s absurd to trumpet that “47% want a bridge to Russell Island” – since it really means that only about 10 people think that way! And are those 10 people all from Russell Is.? We are never told how many people responded to Laming’s surveys, or what the demographic array is.
It is notoriously difficult to design questionnaires that really deliver the goods – namely, objectively valid data. But without valid data, there is no meaningful basis for the use of the survey results for subsequent decision-making and planning. It’s a doddle to concoct a questionnaire that delivers desired results – any fool can do that. And it produces results as superficial as did the quizzes that used to be the mainstay of cheap women’s magazines.
In mid-February of this year, we constituents were proudly informed by Andrew Laming of the results of ‘Redland’s Biggest Survey.’ However, when such a patently simplistic method is used with questions that are limited and obviously ‘loaded,’ only the naïve or ignorant would regard the results as having any objective validity.
Misguided…or misleading?
I was prepared to believe that Laming’s intentions were good, although his actions and survey methodology were misguided and based on a poor understanding of proper survey design methodology. So I wrote to outline my concerns. These concerns were met with lame excuses and attempts at justification [eg. “It is …all I can do with the budget allowed.”] He told me the questions were taken from the community “and in most cases they don’t want us editing!”
My conclusion from Laming’s comments is that he is not qualified in the arena of survey methodology himself, nor has he delegated the task to a suitably qualified professional survey designer. His own comments reveal that he believes getting a cheap, superficial [= invalid] survey concocted by amateurs is better than none at all. Not true, Andrew! It’s misleading and unethical. Such prejudiced methods directly affect who will reply and what replies will come. The fact is that the surveys are biased to get the kinds of responses that suit him.
If he genuinely wants to get VALID DATA on which to base future decisions, policies and actions, I suggest he turn to independent, professional persons or organisations to undertake the work. This could result in credible, valid data of real value and use. Not just the cynical pretence of it, which is current practice.
If our MP is unwilling or unable to organise proper, meaningful research, then I recommend desisting altogether from these prejudiced charades, which harm his credibility and do no service to our community.
Elinor Drake – 13 July 2015
Please note: Offensive or off-topic comments will be deleted. If offended by any published comment please email thereporter@redlands2030.net
Not only a misleading survey but a misleading MP! The other day Laming was claiming a great victory for Joe Hockey v Fairfax. newspapers. What a joke when Hockey actually has to pay 85% of the court costs. Laming sees a headline and believes it. Shallow comes to mind yet again.
I read the questions with a view to responding but could see no logic, just bias and questions worded to get the required answers. I binned the survey! I believe the good doctor should concentrate his efforts to the Federal arena and getting Australia back to a real working nation or we will follow Greece
Andrew Laming’s ‘survey’ techniques are exactly the same as those that were employed by Campbell Newman. One only needs a modicum of education in design and analysis of surveys to confirm this to be the case. If interested read up on ‘statistical validity’ and ‘sample size’.
No matter which way a person responds to these types of ‘surveys’; the ‘results’ are able to be manipulated to give or gain a positive spin or a supposed mandate to perform some action desired by the instigator.
The only way we can have any confidence in the results is if no one fills in the ‘survey’. But even then it may be the ‘results’ will be manipulated to reflect an affirmation of what the instigator is seeking to do or not do.
In a time when we are being told we need to tighten our belts, why are tax payer funds be used on these ‘surveys’ when nothing of any value to the residents can be gained because of their lack of scientific design?
Push polling is never acceptable, and I’m in full agreement with Elinor Drake that Mr Laming’s surveys are poorly designed, poorly executed and designed not to gauge the electorate, but to influence us.
I support Elinor Drake’s comments about this survey, and others that have been set up like it. They are often exercises designed to influence public opinion rather than to gauge it.
It is very difficult to construct a survey of this type that will produce valid, objective results. Even professionals can get it wrong and the number of replies needed to reach a decent conclusion must be quite large.
I’m afraid many of them should be seen as a PR exercises in which an individual or organisation is seeking to appear to have “widespread support” on some issue, rather than being an effort to gather genuine evidence.
It seems that anything Andrew Laming tries to do is panned. As far as professional surveys go, I have no faith in them either, as the only people to bother answering a professional phone survey are those with nothing else to do, or are offered some incentive to do so. I do not believe a fair sample of the community is in fact obtained. I am glad that a Polly is trying to garner some input from his constituents. Has Elinor Drake queried the statistics for Toondah Harbour and Shoreline? How about the Gay Lobby who claim the majority of Australians are in favour of Gay Marriage?
I am expecting thousands of Redland responses; a higher number than any other electoral survey in the nation. I also targeted sms poll with the largest mobile ph surveys Australia-wide. I can adjust by age, gender, geography and political leaning; anything. But sorry, that is all I tell you, because I am busy responding to those who participate.
How Do You Rate Redlands? Doogie the Dugong wants to know.
Andrew Laming should cease and desist from mailing his trivial questionnaire to residents in the Bowman electorate. As well as underestimating the average person’s intelligence, this patronising exercise is a waste of time and sorry misuse of taxpayers’ funds.
As if residents are not importuned enough by snake oil salesmen, we are being harassed also by the Federal MP.
A Redlands Community Survey this is not. Not even close.
PS: Animals deserve dignity and respect. Representing a dugong as a clownish mascot for the ‘survey’ is not amusing. It’s a shame. It insults dugongs and humans.
It strikes me that Andrew Laming’s responses prove the point of Elinor Drake’s article.
Rather than responding to his constituents legitimate concerns in a substantive way he resorts to “you must be a Labor voter” or “trust me, this survey is huge”.
Both types of response indicate that he is unaware or uncaring that constituents expect their representative, when in government, to put aside their political biases and spin and govern wisely. Good governance relies upon acting on the best information available. I know of no decent decision in history that was made based on bullshit.
Yet Dr Laming seems convinced that, rather than the true facts of the matter that could be gathered from a more professional survey, political-party allegiance or mere numbers of respondents to a biased question are the basis on which to govern.
Unless Andrew Laming can answer my question about the independent auditing of his ‘surveys’, I’ll continue to ignore them. Unless there is an audit, he can claim any result he chooses.
LInda, as Australia’s largest and most effective Electoral survey, I can always anticipate that some Labor voters will boycott it. It actually has zero impact on the results.
No one here as any substantiated complaint about the survey. Linda, I don’t even know if you have a specific question ! Why would I independently audit the survey. I read every survey, note every response, hit a button and have the results adjusted for any skew you can come up with. You sound a bit like a big-government kind of ALP supporter.
Although we did complete it, Laming’s ‘survey’ was frankly laughable. Most 14 year olds could have put together better science projects.
To start with the survey envelope looked like junk mail – no Government crest, no mention of Laming. Most will have gone on the bin.
Marketing appeal: 0
Then we get into the questions, aaaand, amateur-hour pigeon-holing. Are you: A person of faith, Boatie, Volunteer, Commuter, ‘Greenie’, Small Business Owner or a Gamer? No stereotyping people into the labels you want to tar them with going on there… I can imagine the results ‘analysis’: Wants NBN – Gamer obviously – who else needs broadband, right? Gay-haters – sign here…
No pre-conceived conclusions involved in questions: 0
And then there’s the covering letter spruiking in advance what a great job Laming is doing – ‘After 6 years abandoned, Southern Redlands will get fast broadband’. Really, well no really – what they are getting is a heavily speed restricted fibre to the node rollout, not FTTP as previously in-line under Labor. ‘Plans for the North will be release shortly. Faster internet finally’. Well, faster press releases finally!
And who has been the Federal Member for the last 6 ‘abandoned’ years? Surprise!!!
Who has been vociferously opposing the NBN and making I’ll-informed & disingenuous comments about it since 2007? Shock!!!!
Truthfulness in claims: 0
And then the coup de grace: rate your Councillor, Mayor, state and Federal Members. I scored my Councillor, the Mayor and my State Member well. Laming? I’m sure you can guess…
And then of course we get to the point – the Marriage Equality survey. Ticked ‘Person Of Faith’ earlier? sign off on Laming’s opposition here. In over 20 years in sales & marketing I’ve rarely seen u have an amateur, narrow-minded and leading ‘survey’.
Overall score: 1 (my 5yo Daughter liked the dugong on the envelope)
Totally agree, Elinor. The intentions may have been good but, given the above factors, it can be a half-baked product at best and may merely give the (costly) impression “look, I have showed you that I care”, even if the collected statistics fail to be comprehensive at best and utterly useless at worst. The bridge survey is a perfect example here. It only delivers one survey per household; misses out delivering even to mainland suburbs of the Redlands; and completely ignores absentee owners and ratepayers who, due to some strange parochial legislation, can’t even vote at local elections in Queensland! An all-island association has tried to ask for a bridge survey to be put in the rates notices but turned down by the Council (while a similar request by another one was granted – talk about discrimination or equal opportunity!). Now, Mr Laming may not have powers to have access to ratepayers’ data under Federal law, I understand – however, what he could certainly do is to raise this issue with Council and State Government and put pressure on Council to cease such discriminatory practice with its ratepayers. It is somewhat odd (even to someone tryly residing on one of the Bay Islands, such as yours truly) that absentee ratepayers would only count when charged levies for things they would never or hardly ever use, such as the Translink levy – yet denied a vote or ignored for local surveys in matters such as bridging. Instead of trumpeting issues totally irrelevant to the local community of the Redlands such as the Queen’s Birthday Awards, cruise ship oil, Halal tax, gay marriage etc, Mr Laming should kindly reconsider his policy priorities benefiting his local constituency. After all, elections are coming up soon and candidates will be judged on how much they deliver to their own electorates, rather than championing campaign or publicity crusades that may serve their own benefit more than those of the community who elected them.
Sadly for you Laming has no real weight in any of those areas he is the Federal member and his job is federal issues. Your recourse is to the State Member for the Redlands. and your councillor.
I’d also suggest that only the people on the islands would be interested in a bridge. Given that the bulk of the repayments would ultimately come out of ALL ratepayers pockets.
But if we think that way, should not only the people on the islands have been interested in commissioning new water taxis ($1M each), too? Or why is the Translink levy charged to SMBI residents only when their Redland counterparts get it for granted and at a far more friendly zoning rate (6 or 7, instead of 12)? These people pay equally high rates yet receive the fraction of infrastructure the ‘luckier’ half of Redlands pay. Isn’t it time this tide should turn?
I survey all issues and share the results with everyone. Next question.
I stopped answering Mr Laming’s questionnaires simply because as someone who studied the area at university and an ex marketing executive All I saw was push polling and digging for contacts ( to bombard with his peculiar view on what his job is. From previous surveys he appears to be ‘dabbling’ in all jurisdictions ( one can wonder ‘doesn’t he think the state members, the right wing dominated council and and the Mayor are up to their jobs?’) working ( gathering names, contacts data?) for the State LNP party. Many of his questions have more state interest that federal.
Keep in mind he’s been investigated for misusing his federal expenses to that end and refused to supply a copy of his signature… check it out.
On that basis is he prepared to accept the opprobrium for the State LNP’s failures/mistakes too see the mooted mental heath facility at the Redland hospital ?( TV last night linked that dropping of the facility with 3 young people’s deaths?). My guess is not.
NB I have no interest in his personal life to me he is employed to represent all the voters of the electorate Not to Proselyte his metaphysical biased opinion ( read not objective or scientifically based).
My area of Cleveland did not even receive a questionnaire , so we’re feeling very rejected ha ha!!
It hasnt been delivered yet Sue.
Mr Laming…I am confused!
How can you say the survey is “Australia’s largest and most effective Electoral survey..” [which is a past tense statement meaning the survey is complete and respondents answered] when you have said above that the survey hasn’t even been delivered to Cleveland yet?
I have to agree! There is far, far more to deciding a credible survey than simply tossing in a few biased and meaningless questions! Surveys by nature produce very rubbery statistics and can be made to reflect totally incorrect data unless done by professional people who know how to design them! Well intentioned probably but the outcome is unlikely to depict a genuine, representative opinion on this oh-so-boring subject that I wish would just go away and die!
I have multiple qualifications in this area’ there is no problem with this survey and I am sorry if you don’t like it. Thousands upon thousands of locals disagree. The response rate is the highest in the nation. In fact, these threads of annoyance prove their effectiveness. If none of you cared, then I would know it was ineffective.
You state you have “…. multiple qualifications in this area…” Please advise which qualifications you hold and the year and the university/ies at which you gained these qualifications.
You say “…these threads of annoyance prove their effectiveness. If none of you cared, then I would know it was ineffective.”
You do not seem to understand Mr Laming. Whilst I am not able to talk on behalf of others please be aware, I for one do not fit your scenario and before you begin to use disparaging remarks to shut down the discussion, I wish it known that I am have not been a Labor voter nor a Greens voter nor an independent & I used to be a life long Liberal voter.
I care enough to comment because the survey is a waste of public money due to its poor design including the total lack of any construct validity. I do not believe you are ‘sorry if [I] don’t like it.
Please stand by your words and provide the evidence that “The response rate is the highest in the nation…” Actually I didn’t know you were surveying the nation and if so then this ‘survey’ is definitely a waste of our tax payer dollars.
Mr Laming is simply using a cynical political tool called “push polling”. He should stop ! And stop pretending that the “survey” represents the communities’ views
Thank you Elinor, I have filled in a couple of his surveys around 10 years ago. They were basically asking for voters preferences on LNP policies of the time. Could only conclude he was gathering stats for his own poll across his electorate. Was quite a simple format easy for some to understand