Research claims within the education industry: managing reflective practice

Claims-making within the education industry

Claims are widely used across the education industry to illustrate the quality and benefits associated with the materials and services that an organisation or individuals are providing. The wide array of claims, claims-makers and intended audiences are reflective of the diverse companies and groups within the education industry. Incorporating what are called ‘edu-businesses’ (Hogan et al., 2016; Ball, 2012), the education industry is a growing transnational sector with increasing amounts of global mergers and acquisitions (Ball, 2012). As this sector expands, providers rely on claims to market their goods to potential consumers.

Before analysing claims-making within the education industry, it is valuable to present what is meant by ‘claims-making’ and why this is an area worthy of ethical reflection. According to Lindekilde (2013), claims-making refers to the process of performing or articulating assertions that bear on someone else’s interest in order to depict something or someone in a certain way. Lindekilde (2013) also states that the purpose of making a claim must be considered in the cultural context in which it is framed. According to the Cambridge English Dictionary (n.d.), a claim is when something is stated as true or a fact, although it may not be proven or others may not believe it. For the purpose of this article, a claim refers to any statement used in the promotion of an educational product or service that is not known as fact by the audience interpreting the promotional message. In this sense, it warrants a degree of trust on the part of the audience that they accept what is being claimed. Regardless of whether it is a fact, the audience is trusting the message without having access to, or a deep understanding of, the substantiating research. Drawing on the academic discipline of informal argumentation, a claim can be defined as ‘a conclusion whose merits we are seeking to establish’ (Toulmin, 2003: 90). Claims represent statements of belief which are open to debate and subject to ongoing scrutiny and challenge (Toulmin, 2003; Blair and Johnson, 1987; Walton, 1989; Pinto, 2001). Based on related literature, it can be surmised that:

• Claims are at best tentative (Toulmin, 2003, 2001). Accordingly, a claim ‘deserves some degree of trust only when it has survived serious attempts to falsify it’ (Cronbach, 1980: 103).

• Claims do not need to be watertight, but they do need to be sufficiently strong. This raises a key question: what constitutes a defensible claim? If claims are to be plausible, then the sufficiency, relevance and acceptability of evidence (Blair and Johnson, 1987; Johnson, 2009), for example, needed to justify them must be convincing to a variety of different groups of stakeholders (House, 2014; Cronbach, 1980).

• Claims are defeasible in that they can be successfully countered even when the argument upon which the claim is predicated is regarded as generally sound (Toulmin et al., 1979; Newton, 2017).

Claims made by companies, organisations and individuals within the education industry are as diverse as the industry itself. For example, claims can relate to the effectiveness of a product, the recognition of a particular qualification or the success of previous consumers or users. Like other competitive markets, claims are essential in order to give potential consumers details as to why a particular product or service is worth their investment. Many of these claims are based on thorough research, and are legitimate claims of the quality of an educational tool or service. However, this is not true for all claims made within the far-reaching and lucrative education industry. Enser (2019) argues that with the rise of privatised educational goods and services there is an increase in ‘research-washing’. Enser (2019: 23) defines this as ‘the application of a thin veneer of research credibility to a product or idea that’s being sold to schools or teachers’. He explains that this increase in research-washing claims has occurred due to an ever-increasing number of companies competing for an often-shrinking budget of schools, school boards and individual customers. Furthermore, ‘appealing to the sector’s growing interest in research-informed practice is a way to make your company, and your product, stand out in a crowded field’ (Enser, 2019: 24).

Enser’s (2019) arguments of research-washing refer to claims that are inaccurately associated with research. However, it is important to remember that research-based claims that are accurately underpinned and validated play an important role in progressing and sharing high-quality teaching and learning strategies across contexts. In addition, there are many companies and individuals within the education industry that genuinely want to develop and offer high-quality goods and services. Not all companies are exclusively focused on filling their ‘coffers’, even though their research base is ‘lacking (to put it mildly)’ (Enser, 2019: 23). In truth, research-based claims are both inevitable and necessary in the field of the education industry in order to support those in the education sector to invest their resources effectively. However, the existence of research-washing potentially undermines the consumer’s trust of the education industry, which makes an exploration into educational claims-making a worthwhile endeavour.

Unwarranted research claims in education research are manifold, and notoriously fraught with controversy (see Gorard, 2002, 2017; Tormey, 2014). Gorard (2017) contends that an education research study must be comprehensible to a range of stakeholders in order for its account to be persuasive. Gorard (2017) further argues that evidence is often weak or not readily available in research reports, which only serves to enhance scepticism among readers regarding the plausibility of the claims made.

Since there is a lack of literature analysing education industry claims and their influence specifically, it is valuable to draw from wider literature pertaining to claims. It has been found that, although widely used and invariably believed, consumers are often wary of claims for various reasons. First, terms may not have a clear, verifiable meaning (Carlson et al., 1993). An example of this is the idea of being ‘environmentally friendly’. The term implies that the product is not an enemy of the environment, but what that means in practice is vague and unverifiable. Second, the scientific knowledge required to understand a claim may require too much effort for the consumers (Carlson et al., 1993).

It is important to note that many international education industry bodies have internal research groups, or have research capacity which engenders trust, but which also brings a level of ethical responsibility. We argue that these claims should follow an internal claims verification process, especially because of the significant implications that these claims may have in guiding policy, service provision and child/social development.

Benefits and risks of claims-making within the education industry

Along with risks to potential consumers, education industry claims also bring about potential risks to the claims-makers. For example, misleading claims can result in financial and reputation loss for product makers, and potentially significant health and life implications for product users (ASA, 2017). Lindqvist and Nordänger (2007) advance the belief that education, and the teaching profession in particular, has been saturated by a new form of risk consciousness and risk consideration. This extends beyond market risk to also include risks related to wider societal repercussions such as, but not limited to, a decrease in education levels, mismanaged education funding and a potential increase in social divisions.

As a result of these acknowledged risks, several regulating bodies have been formed at the national and international level to provide guidance as to what ethical advertising involves, and to police those who do not advertise responsibly. Although some restrictions are targeted towards limiting offensive material and socially sensitive discourse (Wang et al., 2018), many restrictions are focused on preventing misleading and false claims (Durant, 2010).

In the UK, regulations are largely in line with European Union directives and regulations on unfair commercial practices and misleading advertising (Durant, 2010; Barendt and Hitchens, 2000). The Advertising Standards Authority (ASA) oversees the implementation of the British Code of Advertising Practice and investigates allegations pertaining to breaches in the code. ASA guidance for claims in the education sector include tips to ensure that advertising is legitimate, advocating the essential requirement to maintain evidence. For example, the ASA (2017: n.p., our emphasis) guidelines stipulate that: ‘It’s essential that you hold documentary evidence to substantiate a comparative claim before you make that claim.’ This stresses that the evidence must already be collected, analysed and available for review before any claims are shared in the public domain. In the past, the ASA has been critiqued as being too relaxed in their policing of advertising claims in the field of education (Bradley, 2013, 2018). This critique has led to the ASA increasing their attention (ASA, 2017).

In addition to being influenced by regulating policy, education industry claims can also influence policy and governance. For example, Carlson et al. (1993) argue that claims have the power to influence the activities of pressure groups, influential individuals, academics, policymakers and practitioners. They give the example of claims that highlight a social concern which must be addressed. If publicised effectively, this social concern can raise legitimate unease among the public, leading to pressure for government action and policy change. Carlson et al. (1993) also use the example of concerns relating to children’s health, which often invoke a public focus on a fear for the future of children’s health and well-being. This public fear can result in increased government action and policy change. There is great responsibility in how this is steered.

Although there are several risks related to claims-making, it is most likely unreasonable to expect that all claims should go through the same level of review before being shared in the public domain. We argue that some claims, which carry higher levels of risk, should go through a thorough review process in order to ensure their accuracy, and that supporting evidence is at hand.

Since different types of claims require different levels of attention, it is worthwhile to construct a claims typology that could easily be referred to so as to assess risk and review processes. With the help of a typology, claims that tend to be of higher risk to an organisation could be given more attention, while lower risk claims can be published and shared more quickly. This would help to ensure that risks are mitigated without needlessly stifling the organisation’s public presence.

Claim typologies

Several studies in the field of claims-making have developed different typologies that cater to specific aims and markets. Based on the findings of a literature review focused on claims typologies, three relevant typologies are discussed below. Each possess elements that are useful for creating an education claims typology.

The first typology emerged after Carlson et al. (1993) conducted a study to systematically examine different types of environmental claims used in advertisements. To do this, they created a matrix that identifies, first, the different types of environmental claims and, second, the likelihood that such claims will be judged as misleading and/or deceptive. A broad sample of environmental advertisements were examined and a set of categories was formed. At the end of this process, four categories of claims were confirmed: product claim; process claim; image orientation claim; and environmental claim. The second phase of the typology involved designing another level of categorisation for potentially misleading and/or deceptive aspects of claims (Carlson et al., 1993). The two-level categorisation of this model would be helpful for education-based typology, so that claims could be categorised based on claim type and risk level. The method of forming categories with one sample group of claims, and then trialling the categories with additional groups of claims until the categories are saturated also appears to be an effective method which could be applied to an education-based typology.

Another typology that is worth consideration is the typology created by Koopmans and Statham (1999) to analyse political claims made within political campaigns and election media coverage. To do this, political claims were coded within newspapers using a multi-step approach, then regrouped into summary codes to consolidate the categories. Although time consuming, this method may serve as a valuable source of data for education industry organisations because of the detailed and longitudinal claims data it procures.

The final typology that will be discussed here, and arguably the most relevant for the purpose of education-based claims-making, is the typology created by Bradley (2013, 2018), which focuses on claims emerging from higher education institutions. Bradley (2013) completed the first UK-wide study and assessment of the claims that were being made by higher education institutions. His method involved a thorough assessment of the prospectuses of eight universities in the UK. Based on the information gathered from a fact-checking exercise, Bradley classified the claims by combining the typologies of Gardner (1975) and Hastak and Mazis (2011). Bradley’s work highlights the importance of reflecting on education-based claims from a legal, ethical and financial perspective, leading to a subsequent ‘crackdown’ on misleading advertising in UK universities (Harding, 2017).

Although these typologies are useful, there is a paucity of recent research in this area. This article aims to contribute to, and reawaken, the debate using examples from the current landscape of the education industry.

Case study: development of a method to support accurate claims-making

Discussions around the expansion of the education industry, as well as the risks associated with the increased marketisation of education (Rikowski, 2019), prompted this case study. Two questions guided the research:

• For education industry claims that are verifiable, is there evidence at hand to prove that this claim is accurate and can be trusted? If not, what is the risk associated with making this claim?

• For education industry claims that are unverifiable (vague or commonly known), do they provide enough strength to adequately promote products and services?

To assess more accurately the risks associated with claims-making within the education industry, it is valuable to devise a method for categorising and assessing risks. This case study serves to present a method for reflection, and not a strict model that is intended to be universally applied to all organisations. Any risk framework should be bespoke to reflect the types of educational claims and audiences specific to an organisation. Examples of typical risk impact categories are shown in Table 1.

Drawing on the work of Carlson et al. (1993) and Koopmans and Statham (1999) discussed above, the first step in the method of development was to collect a broad sample of claims from an international awarding organisation (IAO), which is within the education industry. A total of 358 web pages from the public website of an IAO were initially reviewed by three researchers, and 284 claims were collected. All web pages that were considered were primarily written in English. Once these claims were collected, they were entered into a spreadsheet. Each claim was coded based on the focus of the claim, such as learning, assessment quality or future aptitudes. Like Koopmans and Statham (1999), the codes were not closed in the first instance. Once the initial coding was complete, the three researchers conducting the coding met and merged the various codes until a concise list was created. High agreement among the coders indicated a level of consistency in understanding and coding. Using this revised list of categories, the research team retrieved another 30 claims from IAO websites, and two researchers (the authors of this article) tested the claims against the code list individually. (For confidentiality purposes, the websites will not be shared. However, the aim of this article is to identify the need for a method of claims reflection, leading to sustainable change, as opposed to policing current claims, which only leads to finite impact.) The two researchers then met and discussed further revisions of the code list required to ensure that it was inclusive as well as parsimonious. The revised list was then tested against a further 30 new claims taken from education industry websites. This cycle of testing and reviewing categories continued for an additional four rounds until it was felt that the list was saturated and no further additions or merging were required. The iterative nature of the test/review cycle afforded increasingly higher levels of inter-rater (and intra-rater) reliability. In total, the final list contained six categories (see Table 2).

Once these categories were established, a definition for each category was written and examples of subcategories were listed. For each claim category, a risk weighting was given. This weighting reflected the likelihood of the risk for claims in that category. For example, Table 2 shows that a ‘claim of expertise’ with the subcategory of ‘assessment’ is given the claim category weighting of ‘5’ (on a scale of 1–5, with 5 being a high-risk weighting). However, the level of weighting will vary depending on what an organisation deems to be their areas of expertise or concern. It should be emphasised that the numbers are merely illustrative of the method applied.

Upon completion of the category list, the research team brought together a group of 15 other researchers from the education industry for a feedback workshop. The aim of this workshop was to present the typology and the development method to elicit feedback on the accuracy and usefulness of the categories for identifying high-risk claims. Overall, the workshop participants were satisfied with the categories presented. One important insight garnered from the workshop was that categories alone did not signify the risk level of a claim (Table 1). The availability of evidence and the context in which the claim is shared would also have a significant impact on the level of risk. As a consequence, it was deemed necessary to devise a second and third level of claim categorisation which would encourage reflection on the context of a claim (see Table 3) and the evidence availability of a claim (see Table 4).

Therefore, according to our method, a claim’s risk is based on combining: (1) the weighting of the typology category (Table 2); (2) the context in which the claim is shared (Table 3); and (3) the available evidence to support that claim (Table 4):

Risk score = Typology weighting + Context risk scale score + Evidence availability score

Again, it should be noted that the numbers (points awarded in the various categories) are simply demonstrative of the method used.

The next task was to connect the risk scores of the three-level categorisation with levels of potential risk. Two ‘risk advisers’ within the education industry were consulted in order to develop a risk matrix which could be used to assess education industry claims (Table 5).

Table 5

Risk scoring matrix (Source: Authors, 2022)

We argue that risk impact categories can be characterised as residing on a continuum, with extremes ‘insignificant’ and ‘extreme’ flanking more ‘moderate’ risks. The likelihood continuum, which intersects with the impact continuum, describes whether a risk is, say, ‘almost certain’, ‘possible’ or ‘very unlikely’. In Table 5, all numbers within the black/grey zone are considered high-risk claims. Associating these levels with a point system (with 1 point referring to ‘insignificant’ and ‘very unlikely’ risks, and 25 points referring to ‘extreme’ and ‘almost certain’ risks) allowed the creation of a quantifiable scale that could be connected to the claims typology.

What follows is a worked example of how two separate, distinct claims can be reviewed using our method to determine a final risk category. It should be noted that these claims are not actual claims made by the IAO. They are, however, typical of the kind of research-based education claims encountered more generally in publicly available information released by IAOs.

Table 6 demonstrates that two very different claims also pose very different risk levels. Once the risk level has been identified, those responsible for making a particular claim will need to reflect on whether there is sufficient evidence available to support the claim, or whether further review of the claim is required. In the case of Claim 2, a low-risk claim, the claims-maker may decide that no further reflection is necessary and that the claim can confidently be shared in the public domain. This may not be the case for Claim 1, which is a high-risk claim and demands further review before public dissemination. For these cases, institutions will need to decide on an appropriate internal review process for claims-makers to follow before the claim is publicly shared.

Conclusion

This case study presented a possible method or approach for critically reviewing the claims being made by companies within the education industry. The numbers suggested within the case study are for illustrative purposes only, and were devised based on a specific organisational context. Although the case study does not present a model that can be directly applied across all organisations, it does present the reasoning, theory and method for developing an effective comparable model.

Decisions regarding typology weightings and scale levels should be made at the organisational level. In order to make these decisions appropriately, organisations will require a detailed knowledge of the types of claims that are being created within their institution, where these claims are being shared, and the potential risks associated with these claims. All of these elements differ from organisation to organisation. To gain an understanding of each context, it is pivotal that key stakeholders within the organisation are consulted, and that there is a consistent, institution-wide approach that is developed and introduced. Institutional engagement and buy-in are essential, if risks are to be mitigated effectively.

Based on the literature and the consultations with those involved in claims-making and risk assessment in the education industry, it has emerged that the most important element of this process is reflection, which has implications for ethical practice. Claims-makers and claims-sharers should reflect on claims in a critical way that is mindful of the potential risks of sharing each claim. Any review method that is developed should have this reflective practice at its core.

In addition, accuracy and efficiency should be considered. The review process should not lead to a policing environment or a needless stifling of creativity. A review process should be created that, first, supports accurate claims being shared and, second, maintains the efficiency of the work that is required to be completed. To ensure efficiency, the reflection process must be built into the operationalisation of what all people think and do. It also must be ongoing, as the accuracy and validity of a claim may change or evolve. Over time, claims-makers and claims-users will become more aware of the considerations and risks that must be taken into account before creating and sharing a claim, and they will naturally begin to integrate more risk-averse claims-making because they are more aware of the different elements to be considered.

We have identified a number of dimensions that can be used to characterise claims within the education industry, and to support organisations in developing ethical and accurate claims-making. We argue that this reflective claims-making process is pivotal for ensuring that the education industry offers high-quality products, and that trust is maintained between education industry organisations and consumers, as well as users. We believe that our proposed method of review will help to maintain and develop trust and to decrease the likelihood of risk for all parties involved.

There are several additional areas of research that are worthy of further investigation. Similar to the work of Bradley (2013), it would be valuable to complete a broad analysis of claims being shared across the education industry and compare them for type of claim, available supporting evidence and the potential impact that these claims are having. As the education industry is a global industry that impacts the lives and futures of millions of learners through education industry products each year, additional research must be done into the ethical responsibilities of education industry companies. In addition, it would be worthwhile to investigate how research is interpreted by the consumer, and what can and should be done to support the consumer’s ability to critically interpret the research claims associated with educational products and services.

There is currently a lack of official monitoring on education industry claims-making, which means ethical standards are also vague and potentially inconsistently applied. A critical investigation into the ethical parameters surrounding claims-making in this field would support the development of guidelines and principles that could be incorporated into education industry business practices. As global market pressures rise, critical reflection of claims may become increasingly important in ensuring that the education industry remains an equal playing field for all companies, all consumers and, most importantly, all learners.

Research ethics statement

Not applicable to this article.

Consent for publication statement

Not applicable to this article.

Conflicts of interest statement

The authors declare no conflicts of interest with this work. All efforts to sufficiently anonymise the authors during peer review of this article have been made. The authors declare no further conflicts with this article.