A form does not collect the truth. It collects what someone is willing to type into a box while wondering who will read it. Most of the gap between those two things is put there by the person writing the questions, usually without noticing.
Three habits cause most of it.
Wording that answers the question for them
“How helpful did you find our onboarding?” has already decided that onboarding was helpful, and is asking you to grade the helpfulness. The neutral version is duller and more useful: “What did you think of onboarding?”, or a scale running from very poor to very good with the midpoint genuinely in the middle.
The tell is an adjective in the question that also appears in the answer you are hoping for. Strip it out and read it again. If the sentence now feels flat, it is probably fair.
A scale that only goes one way
A five-point scale labelled good, very good, excellent, outstanding and exceptional will produce a wonderful average and tell you nothing. A usable scale is symmetrical: as many steps below the midpoint as above it, with the ends labelled and meant.
The same applies to choices. If a multiple choice question offers four flattering options and one grudging “other”, people who disagree will pick a flattering one rather than mark themselves out. Give disagreement a plain, unembarrassing option, and give people a way out with “prefer not to say” when the question is personal.
Worth knowing: in Questionful, conditional logic runs on multiple choice, checkboxes and dropdowns, not on scales or sliders. If you want the follow-up question to depend on how someone rated you, ask for the rating on a choice question, or put a short choice band next to the scale and branch on that.
A question they know you can trace
People answer differently when they think you can tell who they are, and they are usually right to think so. A form that asks for a name and email at the top, then asks how the team is doing, is a signed complaint with a smile on it.
Be clear about which it is:
- Named, and say so. Fine for requests, applications, bookings, anything where the answer is only useful attached to a person.
- Anonymous, and mean it. Do not ask for a name, a staff number, a team of four people, a job title that only one person holds, or a free-text box that invites someone to identify themselves. Four demographic questions on a thirty-person survey are a name in all but spelling.
Be careful about promising more anonymity than a form can give. A submission still arrives with a timestamp, and if you sent the link to one person you can work out who answered. The honest line is “we will not ask who you are and we will report this in aggregate”, not “this is completely untraceable”.
Ask about behaviour, not intention
“Would you use a feature that did X?” gets yes, because saying yes is free. “When did you last do X, and what did you use?” gets something you can act on. Past behaviour is a matter of record; intention is a mood.
The same swap works for satisfaction. Instead of asking how likely someone is to recommend you in the abstract, ask what they were trying to do, whether they finished, and what got in the way. You lose a tidy number and gain a sentence you can take to whoever owns the problem.
One idea per question
“How clear and useful was the documentation?” is two questions wearing one label, and an answer of 3 tells you nothing about either. Split it. Two scales take ten seconds longer to answer and save you a week of guessing.
Watch for the word “and” in a question, and for the word “not”. A negative question with a scale attached (“the instructions were not confusing”) makes the respondent do algebra before answering, and some will get it wrong.
Read it once as the respondent
Before you send it, open the form and answer it honestly as if you were on the receiving end, with the same stake in the answer. Where you hesitate, that is where your data will be soft. Either change the question or accept that the answer is a rough one and stop treating it as a number.
Most bad survey data is not collected by accident. It is collected by questions that were written to be agreed with.