In the Regulator’s 2024/25 statistics, surveys collected by post gave an overall satisfaction (TP01) of 80.2% and surveys collected online 60.1%. These are sector-level figures for every survey collected by each method across all submissions, rented and low cost home ownership alike, weighted by the number of responses. Most landlords used more than one method, so they are not the scores of postal-only or online-only landlords. The Regulator reads this evidence as suggesting “a significant impact of survey collection method on the satisfaction score”, so the method you choose is part of what your published number measures. This guide is for the insights or research manager who has read the cornerstone TSM guide and is now choosing: what drives the cost of each mode, how the “mode effect” works, and how to make a defensible decision.
We’ll cover what each mode spends money on, why response rate is the wrong target, the mechanisms that can open a gap between modes, mixed-mode design, and a decision framework you can actually use. RSH publishes neither unit costs nor response-rate benchmarks by mode, so this guide does not quote sector ranges for either.
1. What drives the cost of each mode
Most TSM cost discussions stop at “online is cheap, phone is expensive.” That hides what actually drives the bill. RSH does not publish unit costs, and quotes depend on your stock, your sample and the provider, so the useful comparison is what each mode spends money on.
Postal
Print, outbound postage, freepost returns, reminder mailings and data entry. The hidden multiplier is the response rate: every questionnaire that does not come back still had to be printed and posted, so a low return rate means many mailings for each completed response.
Telephone (human interviewer)
Interviewer time dominates: dialling, call-backs and the interview itself, plus supervision and quality checks. Each call-back to a quota cell that is short on responses adds labour without adding data, and interviews in other languages need interviewers who speak them.
Online / email
The lowest variable cost: a survey platform and invitations sent through your existing tenant records. The catch is coverage. Tenants with no email address on file, or who rarely open email from their landlord, have to be reached another way or not at all, and surveys collected online gave the lowest overall satisfaction of any collection method in the 2024/25 statistics.
AI-phone (automated voice / conversational AI)
Call time, scripting and quality assurance, with no interviewer recruitment or shift patterns. It keeps the telephone’s reach into older and non-digital tenants, asks every question the same way and can keep a recording or transcript of every call for audit. The honest caveat is the absence of a published TSM-specific response-rate benchmark for AI-phone fieldwork.
Mixed-mode
Cost depends on the mix. A “push-to-web first, postal sweep for non-responders, phone follow-up for under-represented quotas” design spends most on the modes that reach the tenants the cheaper ones miss.
Whichever mode you choose, ask for one fixed price against the achieved sample your accuracy band requires, rather than a price per call placed or per questionnaire mailed.
2. The mode effect — how collection method can shift scores
The 20-point gap between postal and online surveys in the 2024/25 sector statistics is not a comparison of postal and online landlords: each figure covers every survey collected that way, weighted by the number of responses. The Regulator calls collection method “fundamental context” when assessing a landlord’s scores, and notes that satisfaction also varies with landlord context such as region, size and the amount of supported housing. Survey methodology points to mechanisms that can push scores apart by mode, and understanding them is the difference between choosing a method and being chosen by one.
Acquiescence bias (postal advantage)
A paper questionnaire arriving from the landlord you’ve had a tenancy with for ten years carries a cooperative framing. Tenants who fill it in are disproportionately tenants who feel obligated to be helpful, which correlates strongly with marking satisfaction in the upper half of the scale. This is well-documented across survey methodology literature; the effect is biggest on agreement-scale items (TP08: “my landlord treats me fairly”) and meaningful on satisfaction-scale items.
Selection bias (postal + face-to-face)
The tenant who returns a paper questionnaire is typically older, longer-tenured, more settled, and more likely to live in general-needs stock than the demographic average. These are also the most-satisfied cohorts in the sector, consistently across published TSM data. Postal mode under-samples the dissatisfied not because they boycott the survey but because the medium itself is friction-heavy for younger, more transient, and ESL households.
Self-selection bias (online disadvantage)
An online survey is a frictionless channel for the actively annoyed. The tenant who has a specific complaint clicks through. The tenant who is broadly content does not feel motivated to open the email. The result is the inverse skew to postal — online over-samples dissatisfaction.
Interviewer effect (human phone)
A polite voice on the phone gets politer answers. Tenants soften criticism when speaking to a real person in real time, especially when the interviewer is identifiable as fieldwork on behalf of the landlord. That is one plausible reason human telephone sits between postal and online in the published figures. AI-phone may reduce the effect, because there is less social pressure to be polite to a software process, but there is no published TSM evidence on it yet.
3. Response rates — why they are the wrong target
The Regulator publishes no minimum response rate, only margin-of-error bands by stock size, and there is no published sector benchmark for what each mode achieves. Response rates move with your tenant mix, the quality of your contact data, call-back rules and the number of reminders, so treat any single quoted figure with care.
Response rate is not the metric the Regulator anchors on — margin of error against the population is. For a 10,000 LCRA stock you need ~965 completed responses to hit ±3%, whether they come from a postal wave, a phone wave or a mix of the two. Our free TSM sample size calculator works both figures out from your stock: the achieved sample your band requires, and the contact volume an expected response rate implies for each mode.
4. Mixed-mode design — the practical playbook
The Regulator’s 2024/25 commentary describes mixed-mode as the methodological gold standard for representativeness. The reason is structural: each mode catches cohorts the others miss. A workable design:
- Push-to-web first. Email or SMS invitation with a unique survey link to every tenant with an email address on file. Costs almost nothing and catches the digitally engaged.
- Postal sweep for non-responders after 14 days. Paper questionnaire and freepost return to every tenant who didn’t complete online plus every tenant without an email address. Catches the paper-preferring part of the remaining frame.
- Phone (human or AI) follow-up to under-represented quotas after 35 days. Once the online + postal returns are in, identify the quota cells (age, stock type, geography) where achieved sample is below population share. Phone the under-represented cells to fill the gap.
- Cut-off + weighting at day 50. Apply weights against the six representativeness characteristics named in the Technical Requirements. Publish achieved-vs-population comparisons in the summary of approach.
The cost of this design is operational complexity. The cost of NOT doing it, if your achieved sample is skewed, is a representativeness challenge from RSH plus a result that nobody on the board trusts.
5. The year-on-year comparison problem
You cannot validly compare year-on-year TP01 if your mode mix has materially changed. Adding postal to a previously online-only design can lift the score; switching from human phone to AI-phone may reduce it. Three things to do if you are changing methods:
- Run a parallel. In transition years, run a portion of your sample on the old method and the rest on the new. This is expensive but lets you publish a mode-corrected like-for-like for one year.
- Document the rationale. The summary of approach is where the regulator and your board look for the methodology audit trail. State the change, the reason, and the expected score effect.
- Report per-mode scores within the same year. Even if you don’t change methods, publishing the per-mode TP01 alongside your weighted headline gives you defensibility and gives the board context for year-on-year shifts.
“You’re not going to fool the regulator by changing collection method.” — Jonathan Cox, Chief Data Officer, Housemark
6. A decision framework
The right method depends on stock size, demographic mix, budget, and how much representativeness pressure you expect from regulator inspection. The patterns we see:
| Profile | Recommended primary mode | Why |
|---|---|---|
| Under 1,000 units, low-budget | Postal census | Proportionate requirements apply; census is acceptable and avoids representativeness debate |
| 1,000–5,000 units, mixed demographic | Mixed-mode (push-to-web → postal sweep) | Best representativeness at moderate cost; achievable in-house with light vendor support |
| 5,000–15,000 units, ageing tenant base | Phone-led (AI-phone or human, depending on budget) | Phone reaches older tenants reliably; AI-phone takes interviewer labour out of the cost |
| 15,000+ units, multilingual stock | AI-phone with multilingual handoff | Only mode that scales to large stock without recruitment ceiling; multilingual auto-detect handles non-English households |
| Heavily supported / sheltered housing | Face-to-face supplement | Trust + literacy concerns mean push-to-web alone under-represents; face-to-face in shared communal spaces fills the gap |
7. What to ask a vendor — methodology fitness
Whether you run TSM in-house, outsource, or use SignalLine, the methodology-specific questions worth asking:
- Verbatim wording confirmation. Can the vendor produce a screenshot of the call script or paper questionnaire showing TP01 first, exact wording throughout?
- Mode-mix transparency in deliverables. Will the report-back include per-mode response counts and per-mode TP01 breakdown alongside the weighted headline?
- Representativeness weighting. Against which characteristics is the vendor weighting? Are they using your stock-data weights or sector-standard weights?
- Year-on-year defensibility. If you used the vendor last year, can they produce a like-for-like comparison that controls for mode change?
- Recording + audit trail. For phone modes, is the call recording (or transcript) retained long enough for inspection?
Sources and further reading
- TSM cornerstone guide — the regime, the 22 measures, scope thresholds, the full compliance pitfalls list.
- How to run a TSM perception survey programme — once you’ve chosen your method, the week-by-week operational playbook from board approval to NROSH+ submission.
- TSM reporting deadlines and NROSH+ submission — the 2026/27 calendar: submission windows, who submits vs who publishes, and the website publication checklist.
- TSM sample size calculator — free tool: your stock figures → required completed responses under the RSH accuracy bands → contact volume at your expected response rate.
- RSH Technical Requirements — accuracy bands, weighting characteristics, mode rationale documentation expectation (updated 3 March 2026 for Awaab’s Law).
- TSM 2024/25 Headline Report — Table 2, overall satisfaction by collection method cited in this guide (postal 80.2%, telephone 69.4%, internet 60.1%), and the Regulator’s reading of it.
- Inside Housing — “Landlords will not fool the regulator by changing TSM survey methods” — Housemark and RSH comments on mode-driven score changes.