TSM survey methodology: postal vs phone vs online vs AI

The Regulator’s 2024/25 statistics show landlords using postal-only fieldwork publish an average TP01 satisfaction of 80.2%; landlords using online-only publish 60.1%. Twenty points of difference between two landlords whose tenants are not actually that different. Methodology, not satisfaction, is what mostly drives the published number. This guide is for the insights or research manager who has read the cornerstone TSM guide and is now choosing — what each mode actually costs, what the “mode effect” really is, and how to make a defensible decision.

We’ll cover cost-per-response with named components, response-rate benchmarks where they exist, the mechanics behind the published satisfaction gap, mixed-mode design, and a decision framework you can actually use. Where numbers are inferred from practitioner consensus rather than RSH-published statistics, we say so explicitly.

1. Cost-per-response: what each mode actually costs

Most TSM cost discussions stop at “online is cheap, phone is expensive.” That hides the components that drive the real number. Here is the cost stack per mode for a typical UK landlord running 1,000 completed responses, synthesised from practitioner commentary and consultancy operating models (RSH does not publish unit costs).

Postal — typically £8–£15 per completed response

• Print: ~£0.30–£0.50 per piece at volume (cover letter + questionnaire + reply envelope).
• Outbound postage: ~£0.85–£1.10 per piece (large letter, second class, 2026 Royal Mail rates).
• Freepost return: ~£1.00–£1.30 per returned piece (you pay only on responses, but assume ~20% return rate).
• Reminders: two reminders standard; doubles print + outbound postage on the unconverted tail.
• Data entry: ~£0.50–£1.50 per response, manual or OCR-assisted.
• Project management + analysis: ~£2–£4 per response amortised across the campaign.

The hidden multiplier is the response rate. Postal averages 15–25% (practitioner consensus from Housemark/Acuity case studies — exact 2024/25 sector benchmark not publicly published). That means to land 1,000 responses you mail 5,000–6,500 invitations. Print + outbound + freepost + reminders + data entry on that volume is where the cost really lives.

Telephone (human interviewer) — typically £15–£30 per completed response

• Interviewer labour: ~£15–£25 per hour (UK CATI research industry rate). A 10–15 minute completed interview, including dial time and call-backs, works out to ~£12–£22 of labour per response.
• Telephony cost: ~£0.05–£0.15 per completed response (negligible compared to labour).
• Project management + quality assurance: ~£3–£6 per response.
• Multilingual interviewer surcharge: +£3–£10 per response for non-English needs.

The cost driver is human time. Each call-back to a quota cell that is short on responses adds labour without adding data. This is the most expensive mode on a per-completed basis in almost every realistic operating scenario.

Online / email — typically £1–£3 per completed response

• Platform cost: ~£0.10–£0.30 per response on enterprise survey platforms; near-zero on self-hosted tools.
• Email delivery: negligible if invitations go via your existing tenant CRM.
• Project management + analysis: ~£1–£2 per response.

The cheapest mode on a unit basis — and the one with the largest published satisfaction discount versus postal. The cost saving on a 1,000-response programme is real (~£10k versus postal) but the downstream cost is the ~15–20 point downward bias on your published score.

AI-phone (automated voice / conversational AI) — typically £3–£6 per completed response

• Per-minute telephony + voice-AI cost: roughly £0.30–£0.50 per minute fully loaded (telephony + ASR + LLM + TTS). A 6–8 minute completed interview lands at £2–£4 of variable cost.
• Sampling, scripting, QA: ~£1–£2 per response amortised across the campaign.
• No interviewer labour, no recruitment, no shift-pattern constraints.

Sits between online and human phone on cost. Retains phone’s reach into older / non-digital tenants, eliminates interviewer drift, and provides full call recordings for audit. The honest caveat is the absence of a published TSM-specific response-rate benchmark for AI-phone — first-mover data, low published evidence base.

Mixed-mode — typically £6–£12 per completed response

Cost depends on the mix. A typical “push-to-web first, postal sweep for non-responders, phone follow-up for under-represented quotas” design costs in the £6–£12 band per completed response, weighted by the share of each method.

2. The mode effect — same tenants, different scores

The 20-point gap between postal and online in the 2024/25 sector statistics is not random. Three mechanisms drive it, and understanding them is the difference between choosing a method and being chosen by one.

Acquiescence bias (postal advantage)

A paper questionnaire arriving from the landlord you’ve had a tenancy with for ten years carries a cooperative framing. Tenants who fill it in are disproportionately tenants who feel obligated to be helpful, which correlates strongly with marking satisfaction in the upper half of the scale. This is well-documented across survey methodology literature; the effect is biggest on agreement-scale items (TP08: “my landlord treats me fairly”) and meaningful on satisfaction-scale items.

Selection bias (postal + face-to-face)

The tenant who returns a paper questionnaire is typically older, longer-tenured, more settled, and more likely to live in general-needs stock than the demographic average. These are also the most-satisfied cohorts in the sector, consistently across published TSM data. Postal mode under-samples the dissatisfied not because they boycott the survey but because the medium itself is friction-heavy for younger, more transient, and ESL households.

Self-selection bias (online disadvantage)

An online survey is a frictionless channel for the actively annoyed. The tenant who has a specific complaint clicks through. The tenant who is broadly content does not feel motivated to open the email. The result is the inverse skew to postal — online over-samples dissatisfaction.

Interviewer effect (human phone)

A polite voice on the phone gets politer answers. Tenants soften criticism when speaking to a real person in real time, especially when the interviewer is identifiable as fieldwork on behalf of the landlord. Human telephone consistently scores between postal and online for this reason. AI-phone reduces this effect because there is no social pressure to be polite to a software process.

3. Response rates — what is realistic in 2024/25

The Regulator publishes no minimum response rate; only margin-of-error bands by stock size. Practitioner benchmarks for what each mode actually achieves:

Response rate is not the metric the Regulator anchors on — margin of error against the population is. For a 10,000 LCRA stock you need ~965 completed responses to hit ±3%, regardless of whether that comes from a 20% response on a 5,000-mailing postal campaign or a 50% completion on a 2,000-contact AI-phone campaign.

4. Mixed-mode design — the practical playbook

The Regulator’s 2024/25 commentary describes mixed-mode as the methodological gold standard for representativeness. The reason is structural: each mode catches cohorts the others miss. A workable design:

1. Push-to-web first. Email or SMS invitation with a unique survey link to every tenant with an email address on file. Costs almost nothing, catches the digitally-engaged 5–15%.
2. Postal sweep for non-responders after 14 days. Paper questionnaire and freepost return to every tenant who didn’t complete online plus every tenant without an email address. Catches the paper-preferring 15–25% of the remaining frame.
3. Phone (human or AI) follow-up to under-represented quotas after 35 days. Once the online + postal returns are in, identify the quota cells (age, stock type, geography) where achieved sample is below population share. Phone the under-represented cells to fill the gap.
4. Cut-off + weighting at day 50. Apply weights against the six representativeness characteristics named in the Technical Requirements. Publish achieved-vs-population comparisons in the summary of approach.

The cost of this design is operational complexity. The cost of NOT doing it, if your achieved sample is skewed, is a representativeness challenge from RSH plus a result that nobody on the board trusts.

5. The year-on-year comparison problem

You cannot validly compare year-on-year TP01 if your mode mix has materially changed. Adding postal to a previously online-only design will lift the score; switching from human phone to AI-phone may reduce it. Three things to do if you are changing methods:

• Run a parallel. In transition years, run a portion of your sample on the old method and the rest on the new. This is expensive but lets you publish a mode-corrected like-for-like for one year.
• Document the rationale. The summary of approach is where the regulator and your board look for the methodology audit trail. State the change, the reason, and the expected score effect.
• Report per-mode scores within the same year. Even if you don’t change methods, publishing the per-mode TP01 alongside your weighted headline gives you defensibility and gives the board context for year-on-year shifts.

“You’re not going to fool the regulator by changing collection method.” — Jonathan Cox, Chief Data Officer, Housemark

6. A decision framework

The right method depends on stock size, demographic mix, budget, and how much representativeness pressure you expect from regulator inspection. The patterns we see:

7. What to ask a vendor — methodology fitness

Whether you run TSM in-house, outsource, or use SignalLine, the methodology-specific questions worth asking:

• Verbatim wording confirmation. Can the vendor produce a screenshot of the call script or paper questionnaire showing TP01 first, exact wording throughout?
• Mode-mix transparency in deliverables. Will the report-back include per-mode response counts and per-mode TP01 breakdown alongside the weighted headline?
• Representativeness weighting. Against which characteristics is the vendor weighting? Are they using your stock-data weights or sector-standard weights?
• Year-on-year defensibility. If you used the vendor last year, can they produce a like-for-like comparison that controls for mode change?
• Recording + audit trail. For phone modes, is the call recording (or transcript) retained long enough for inspection?

Sources and further reading

• TSM cornerstone guide — the regime, the 22 measures, scope thresholds, the full compliance pitfalls list.
• How to run a TSM perception survey programme — once you’ve chosen your method, the week-by-week operational playbook from board approval to NROSH+ submission.
• RSH Technical Requirements — accuracy bands, weighting characteristics, mode rationale documentation expectation (updated 3 March 2026 for Awaab’s Law).
• TSM 2024/25 Headline Report — the per-mode satisfaction breakdown cited in this guide (postal 80.2%, telephone 69.4%, online 60.1%).
• Inside Housing — “Landlords will not fool the regulator by changing TSM survey methods” — Housemark and RSH comments on mode-driven score changes.