Satisfied? Student Surveys, Social Position, and the Limits of Experience Data

Posted on by
Class, cultural capital, and social reproduction Method

Every spring, nearly half a million final-year undergraduate students across the United Kingdom complete the National Student Survey. They rate their teaching, their feedback, their academic support, and their sense of being heard. The results are scrutinised, folded into league tables, and cited in institutional strategies as evidence of quality. A high score signals that an institution is doing something right. A low score prompts anxious internal reviews.

The question the NSS rarely invites is the one it most needs: what, precisely, is it measuring? Satisfaction is not the same as learning, and the social position of the student completing the survey shapes what they report in ways the instrument cannot detect. The problem is not simply that the NSS asks imperfect questions. It is that satisfaction is a socially patterned measure being used as if it were neutral – and when that measure is fed into governance mechanisms that distribute reputational capital and resource, its limitations become politically consequential in ways that fall unevenly across the sector. They follow the contours of social class. A flawed instrument used cautiously can still generate useful information. The same instrument converted into rankings and marketing claims about teaching excellence becomes something rather different.

What Student Surveys Actually Measure

The NSS was established in 2005 and has since become the dominant mechanism through which student experience is made visible to institutional managers, prospective students, and external regulators. It covers teaching quality, assessment and feedback, academic support, organisation and management, learning resources, and student voice. The results inform domestic league tables and have historically fed into the Teaching Excellence Framework, though the Royal Statistical Society noted as early as 2017 that there was “no reliable association” between NSS scores and TEF ratings – a finding that did little to dent the instrument’s authority (Royal Statistical Society, 2017).

Before examining those deeper validity questions, it is worth noting that the instrument’s own administrators have recently demonstrated their uncertainty about what Q27 – the overall satisfaction question – was actually measuring. From the 2023 survey onwards, Q27 was removed from the NSS in England, following an OfS-led review concluded in late 2022 (Office for Students, 2022). The stated rationale was that the question was too consumerist in nature – a curious objection, given that the entire instrument is a product of the same marketisation of higher education it was apparently trying to moderate. Scotland, Wales, and Northern Ireland retained the overall satisfaction question, creating an inconsistency across the UK that makes cross-national comparisons methodologically problematic. A Scottish institution’s overall satisfaction figure is a direct survey response; an English institution’s equivalent is a post-hoc aggregate constructed from the remaining 25 core questions, a practice the OfS itself anticipated and implicitly endorsed (Advance HE, 2022). These are not equivalent data structures. Treating them as such in comparative rankings reproduces, in a less transparent form, precisely the validity problem the removal of Q27 was supposed to address.

The NSS is not unique in this kind of difficulty. The United States has the National Survey of Student Engagement, Australia the Course Experience Questionnaire, and UK postgraduate equivalents include the PTES and PRES. The critique developed here applies in varying degrees to all of them. The NSS serves as the primary case study because it is the most consequential UK example and its data is publicly available.

Whether the NSS measures what it claims to measure has been contested since its inception. Richardson (2005) provided the earliest systematic analysis of the instrument’s psychometric properties and, whilst finding some support for its factor structure, acknowledged that the relationship between student satisfaction and educational quality was far from straightforward. There remains no settled definition of student satisfaction and no accepted measure of it (Lenton, 2015). The NSS records something real – students’ reported perceptions – but the inference from those perceptions to conclusions about institutional quality is considerably more treacherous than the instrument’s public use implies.

The most fundamental conceptual problem is the conflation of satisfaction with learning. Paul Ramsden’s foundational work on approaches to studying distinguishes between deep and surface approaches to learning, and the conditions that promote each (Ramsden, 2003). A student who feels satisfied has not necessarily engaged in deep learning; they may have been made comfortable, well-supported, and adequately informed about assessment requirements – all educationally worthwhile, but none equivalent to intellectual development. Conversely, genuinely challenging teaching may generate dissatisfaction in the short term whilst producing lasting learning gains. The NSS captures the former and is treated as if it measured the latter. As one sector analysis puts it bluntly, a course could be taught entirely by staff on zero-hours contracts with no relevant professional experience and, as long as they answered their emails and explained things clearly, the NSS would record satisfaction (Dickinson, 2024).

Key concept: Surface and deep measures

A surface measure records a directly observable phenomenon – here, reported satisfaction. A deep measure captures the underlying reality that the surface measure is assumed to represent – here, educational quality or learning gain. The NSS is a surface measure being used as if it were a deep one. That gap is not a technical imprecision. It is a structural feature of the instrument that no amount of question refinement can eliminate.

Burgess et al. (2018), analysing ten years of NSS data, found consistent upward trends in satisfaction scores. Over the same period, student support infrastructure has expanded substantially: pastoral services, mental health provision, financial bursaries, disability support, academic skills development. The NSS cannot distinguish how much of any improvement in scores reflects improved teaching quality and how much reflects improved conditions surrounding teaching. It aggregates both and presents the result as a verdict on the former.

The Social Skew: Why Position Shapes Perception

The conceptual problem above is a property of the instrument. The social skew is a property of who completes it. Even if satisfaction and learning quality were sufficiently correlated to justify the NSS’s use, a second problem would remain: reported satisfaction is not a direct response to educational experience. It is produced by the interaction between experience and the expectations students bring to it – and those expectations are shaped by social position, prior educational experience, and what Bourdieu (1984) calls habitus: the durable, largely unconscious dispositions acquired through social experience that shape what feels normal, what counts as good enough, and whether expressing dissatisfaction feels legitimate at all.

Students from working-class backgrounds and non-traditional educational routes – those disproportionately represented at widening participation institutions – tend to arrive at university with expectations calibrated to a different range of experience. Research consistently documents this. Reay et al. (2009), in their study of working-class students at an elite English university, found that students from families without higher education experience frequently felt they did not quite belong, expending considerable effort to perform in ways that more advantaged peers managed with apparent ease. Their sense of what the institution owed them was structurally different.

The consequence for satisfaction surveys is direct. Students who arrive with lower baseline expectations are more likely to report satisfaction from a given level of provision than students who arrive expecting more. A student for whom university is a genuinely unfamiliar institution may report high satisfaction from provision that a student with different expectations would rate as adequate at best. This is not dishonesty; it is the normal operation of expectations as a filter on experience.

Burke (2012), in her analysis of widening participation and the right to higher education, documents how students from non-traditional backgrounds frequently engage in what she describes as misrecognition: a process by which the norms and values of the academic field are internalised as natural and legitimate even when they systematically disadvantage the student. An institution that makes a first-generation student feel adequately supported has done something genuinely valuable – but the satisfaction that student reports on the NSS measures their distance from a lower baseline, not the absolute quality of the provision.

It is worth acknowledging that the relationship between social position and satisfaction reporting is more complex than a simple model of working-class students as more grateful and advantaged students as more demanding. Students from more privileged backgrounds are often more skilled at extracting available institutional resource – navigating office hours, accessing networks, interpreting bureaucratic processes – in ways that shape their experience independently of their expectations. The socially patterned character of satisfaction reporting runs through multiple mechanisms, not one.

That complexity notwithstanding, Lenton (2015), using panel data from 121 UK universities, found large and persistent differences in NSS scores across different groups of institutions and concluded explicitly that raw scores should not be used as a method of ranking. The differences reflected structural features of the institutions and their student populations, not differences in teaching quality.

The Composite Problem: What Satisfaction Is Actually Tracking

Where the previous section addressed a subjective mechanism – how habitus shapes what students expect and therefore report – this section addresses an objective one: what students are actually responding to. Satisfaction is not a simple response to teaching quality. It responds to the totality of the student experience: teaching, pastoral support, financial security, disability provision, social environment, and the general sense of being adequately resourced and cared for. Institutional score comparisons collapse these into a single figure.

Institutions that invest heavily in student support infrastructure – bursary schemes, disability services, mental health provision, academic skills support – are likely to generate higher satisfaction scores, not because their teaching is better but because the conditions surrounding their teaching are more supportive. This investment is concentrated in post-92 and widening participation institutions, precisely because their student populations have greater support needs. The result is that the NSS rewards institutional investment that cannot be reduced to teaching quality and presents the resulting scores as evidence of teaching excellence.

The disability dimension is worth noting, though the specific mechanism in NSS responses remains more inferential than directly evidenced at scale. Disabled students are over-represented at widening participation institutions. They are also, for plausible reasons, more responsive to the quality of institutional support in their satisfaction reporting – because adequate support is more salient to their experience. An institution that invests substantially in disability services and reasonable adjustments may score well on the NSS partly because it has reduced the friction of being a disabled student. This is an educationally and ethically important thing to do. It is not, however, what NSS scores are typically taken to show.

The same logic applies to financial support. Students in financial difficulty experience university differently from those who are financially secure, and financial security shapes overall satisfaction in ways that have little to do with the curriculum. Institutions that invest in bursaries, hardship funds, and subsidised services are improving their students’ lives – and those improvements appear in satisfaction data. Treating those scores as primarily reflecting teaching quality is a misreading of what produced them.

On graduate outcomes as a validity check

If NSS scores were a reliable proxy for institutional quality, they should correlate with other quality indicators. The evidence suggests they do not. Sector analysis of the OfS Proceed metric finds little to no relationship between NSS scores and graduate progression outcomes, with higher-tariff institutions tending to score well on progression for reasons bound up with student profile and employer recognition rather than teaching quality (Kernohan, 2023). Post-92 institutions that score well on the NSS frequently serve student populations whose post-graduation trajectories are constrained by factors that precede the degree: the labour markets they enter, the social networks available to them, the regions in which they live. High NSS scores and modest graduate outcomes in combination do not indicate institutional failure. They indicate that the two instruments are measuring different things – and that neither is a straightforward proxy for educational quality.

Towards a More Reflexive Use of Student Data

It is worth being direct about a specific practice the preceding analysis renders indefensible: the use of high NSS category rankings as claims to teaching excellence. Across the sector, institutions that top particular NSS categories incorporate those rankings into marketing materials, vice-chancellorial communications, and strategic narratives, presenting them as evidence that their students receive an education of exceptional quality. Some of those institutions serve student populations characterised by high proportions of widening participation entrants, significant disability disclosure rates, and substantial investment in bursary and pastoral support. Their high NSS scores are, on the analysis developed here, at least partly a product of those characteristics and that investment – not primarily a verdict on the quality of their teaching.

An institution that ranks first in a satisfaction category and simultaneously produces graduate outcome data inconsistent with that ranking is not necessarily failing its students; it may be succeeding with a population whose trajectories are shaped by structural constraints the degree cannot fully offset. But it is not, on that basis, entitled to present its NSS position as evidence of teaching excellence. The gap between what the data shows and what is claimed from it is not a minor rhetorical flourish. It is a misrepresentation of what the instrument measures – and one that prospective students, many of them navigating higher education without the social capital to interrogate marketing claims, are poorly positioned to detect.

A more reflexive approach would begin by being honest about what satisfaction surveys measure and what they do not. It means resisting the temptation to treat high scores as evidence of teaching excellence and low scores as evidence of failure. It means disaggregating satisfaction data by student characteristics before making institutional comparisons. It means being explicit about the confounding factors – the differential distribution of support infrastructure, financial investment, and disability provision – that shape satisfaction independently of pedagogy. And it means supplementing satisfaction data with other measures rather than treating any single metric as a sufficient account of quality.

The sociology of knowledge asks not only “what does this data say?” but “what social conditions produced this data, and what is rendered invisible in the process?” Applied to the NSS, the answer is clear: the social conditions that produced the data include the habitus, expectations, and support needs of a diverse student population whose differences the instrument flattens into a single score. What is rendered invisible is precisely the structural inequality that widening participation policy claims to address. If the NSS is read as a neutral verdict on teaching quality, it tells a misleading story – not because the data is fabricated, but because what it records is not simply educational quality, but the unequal social conditions through which that quality is experienced, measured, and claimed.

References

Cite this article

Select a style and copy the citation into your document.