Research Support
Despite new innovative ways of gathering consumer data, ranging from neuro-scanning to geo-tracking, surveys (online or paper based) are still a popular way of “measuring the temperature” of the market to identify how consumers feel about products, services, brands and more.
They are accessible to any business, respondents are used to them and there many free platforms available online. These include, for example, KwikSurveys, a platform I often use in class with my students.
However, despite its vast adoption, surveys are still often done with recurrent mistakes. So would like to show you how to develop a great survey questionnaire by pointing them out and hopefully it will help you while developing your own study.
Ready? Here we go!
AVOID THESE MISTAKES TO DEVELOP THE PERFECT SURVEY!
Before we get into the details, I would like to point something out: these are my personal suggestions. Please remember to discuss them with your supervisor(s) as academics often have different views, ok?
Alright, so here are some of the most common mistakes you should avoid:
1. Asking Demographic Questions at the Start
- Problem description: I often see surveys where the first question (s) is about one’s demographic characteristics. These may include questions related to age, gender, income, level of education, profession and more.
- Why is it an issue?
- First, because it is better to have one’s fresh mind answering important questions. If a respondent has already answered 4-8 demographic questions before even addressing your topic, he or she will be “tired” once you start asking the actual important questions for your study. Have participant’s fresh minds answering important questions of your study!
- Second, because some potential demographic questions such as “monthly income” may be sensitive to some respondents. If one feels this way, they will immediately leave your survey and not answer the other relevant questions.
- What should you do?
- I suggest you leave all your demographic questions to the end of the survey. The respondent will have already answered all relevant topics and, as they will be tired, they will quickly address the demographic questions. In case they abandon, they will already have answered all other important topics.
2. Using “Multiple Choice” Questions When Not Suitable
- Problem description: I often see questionnaires in which companies ask questions such as: “Which alcoholic drink do you most drink”? or “In which season of the year do you most drink coffee”? and in each of these similar-type questions, respondents can only choose one or more options to answer.
- Why is it an issue?
- To explain, I need to refer to the examples I just listed above.
- First, because just think about it: we don’t just drink ONE type of alcoholic drink. We drink MANY types of alcoholic drinks, but in different frequencies. So if you want to know which one I prefer, you simply need to know which one I drink most frequently.
- Referring to the second example above, we drink coffee in ALL seasons, but just in different frequencies.
- So by asking the respondent to chose ONE or TWO options through “multiple-choice” questions, it will provide an extremely superficial overview of the respondent’s behavior.
- To explain, I need to refer to the examples I just listed above.
- What should you do?
- I suggest you as perceptual or behavioral questions through “matrix” format, in which you can list, for example, the types of drinks or seasons of the year, and let respondents rank EACH option on a Likert-Type scale (e.g. ranging from 1-Dislike it very much, to 5-Like it very much; or 1-Never, to 5-Great deal).
- By doing so, you will not only address the original question (which drink someone prefers or in which season they most drink coffee), but you will also gather data about each option, which is much more enriching.
- Here is a compilation of Likert-Type achors for you to DOWNLOAD.
IMPORTANT: Also, when asking questions on “multiple-choice” format, the data generated will be of “categorical” nature, mostly “nominal”. This implies that there is not much you will be able to do in terms of statistical tests (e.g. Mainly frequencies or Chi-Square). When measuring through “Likert-Type” scales, the data generated will be “Interval”, which will allow you to conducted many other tests such as correlation, t-tests, ANOVA and much more. But unfortunately explaining this in detail is beyond the scope of this article.
Want to Know More?
3. Forcing Respondents to Position Themselves on a Topic
- Problem description: I have seen this recently in two different scenarios (during a flight and at a hotel). In both situations the companies were developing surveys to measure satisfaction of travelers and guests. However the scales they used for the question did not allow for a mid-point. This was their range: 1-Very Dissatisfied, 2- Dissatisfied, 4-Satisfied and 5-Very Satisfied.
- Why is it an issue?
- Well, what if the respondent is simply “neutral” to what you are asking? They do not have to necessarily agree or disagree on something. In my case, the flight and hotel were simply OK. There was nothing different that would make me feel satisfied or dissatisfied. I just got exactly what was expected. So I was “indifferent”. But in both cases, the way the question was asked induced me to position myself.
- What should you do?
- I suggest that whenever you measure one’s perception regarding something, to always allow a “mid-point”, often labeled as “neutral”, “indifferent”, “neither agree nor disagree”. This way, respondents will be able to stand neutral in case they do not have necessarily an opinion.
- I suggest that whenever you measure one’s perception regarding something, to always allow a “mid-point”, often labeled as “neutral”, “indifferent”, “neither agree nor disagree”. This way, respondents will be able to stand neutral in case they do not have necessarily an opinion.
Have You Seen Our “Thesis / Dissertation Tips” Series?
4. Not Having Evenly Distributed Answer Options
- Problem description: Believe it or not, but often I see questions in which I cannot answer simply because my profile does not fit to the answer possibilities. For example, once a demographic question, options ranged from “Under 18 years old”, “19-25”, “26-31” and so on.
- Why is it an issue?
- There are two important mistakes on the example I mentioned above.
- First, if someone is 18 years old, what should they answer? There no option available.
- Second, the age gaps range from “19-25” (thus, a 6 year gap) and then from “26-31” (a 5 year gap). Because the gaps are uneven, it compromises all statistical comparison between groups because you simply have an uneven distribution.
- First, if someone is 18 years old, what should they answer? There no option available.
- There are two important mistakes on the example I mentioned above.
- What should you do?
- First, double-check to make sure that all possible age groups are included or whichever relevant groups that you are asking.
- Second, when asking ranges of age, price, time, income… and so on, that there is an even distribution of gaps! Otherwise, you will not be able to compare them.
5. Formulating Invalid Questions
- Problem description:
- It is absolutely OK to develop your own questions when you wish to measure simple perceptions, such as “what type of product do you currently own”? However, often I see complex constructs, such as “visual appeal”, “purchase intention” and “service quality” being measured through single items.
- Why is it an issue?
- First and most important, these more complex constructs are often multi-dimensional and there are multiple facets to them that need to be measured separately. For example, “Service quality” is often measured through the famous “servqual” scale, which contains five dimensions. Thus, measuring through a single item a construct which is so complex is extremely limiting.
- Moreover, when measuring through a single item is not possible to calculate the reliability of the measurement, using tests such as Cronbach alpha. This is only possible when using multi-items scales.
- What should you do?
- Extremely easy. Simply use the handbook of marketing scales. There you will find a compilation of hundreds of previously validated scales, with all the original items. This way you will be able to simply add them to your questionnaire and, if needed, make minor adaptations to the items to suit the purpose of your study.
6. Repeating the Same Question Format or Answers
- Problem description:
- It is quite common to find studies in which almost all questions are framed exactly the same way. For example, all with multiple questions or matrixes, or with similar answer anchors (Likert-Type scales ranging, for example, from Fully disagree to Fully agree).
- Why is it an issue?
- The moment the respondent has answered a few questions with the same format or possible answers, their perception will simply “desensitise”. Which means, it will be less likely that they will notice changes and therefore it will affect negatively their understanding and attention level. Or simply put, it will seem to them that all questions re the same and they will lose their motivation while answering. And in these scenarios, it is probable that will simply answer somewhere along the middle of the scale to finish the task quickly.
- What should you do?
- While developing the questionnaire, try to combine the questions you wish to ask with different formats. Of course, you cannot compromise measurements to use different question formats (e.g. multiple choice or matrix) simply for the sake of changing. But whenever it is possible, use different question formats and also different Like-type scale anchors. This way you will be constantly triggering the respondents’ attention and perception, which will increase the likelihood that they will answer more attentively to your survey.
7. Addressing Respondents That Are Not Part of The Population of Your Study
- Problem description:
- My students often mention that they will share their online survey with “everyone” or “all students on campus”. And even worse: they have nothing on their questionnaire to allow them to know which respondents are part of the sample of their study and who is not.
- Why is it an issue?
- This is a huge mistake because you will mostly likely have answers from respondents that are not part of the population of your study. This will completely invalidate your results. So be very careful with this!
- Oh, and in case you are not sure what the population of your study is, talk to your supervisor(s)! (Although my friend, let’s be honest… you should know it, right?).
- What should you do?
- Very simply. The best thing for you to do is to have one or more “filter question(s)”. These are questions that will allow you to know if the respondent is part of the population of your study or not. In case they are not, they should exit the study. and usually you would place the question at the start of the survey.
- For example, imagine you are doing a study about “alcohol consumption patterns among retirement people”. This means ALL your respondents must be alcohol consumers and retired. So you should have some initial questions asking those two factors and in case a respondent say the do not drink or that they are still actively working, they should exit. This way, in the end you will only have respondents that are part of the population of your study. Got it?
Final Thoughts
My friend, if you are reading this article until here you are a true warrior. Well done, I am very proud of you!
I just wanted to make sure you would not make basic mistakes that we see so often in questionnaires shared by researchers and companies. Do not forget to address all those 7 issues and you will definitely be on the right track for success!
All the absolute best and rock on with your research!
Cheers!
Comment(01)