Rapid reviews as an emerging approach to evidence synthesis in education

Systematic reviews

Systematic reviews use rigorous and explicit methods to identify, appraise and synthesise studies addressing a structured question. Each stage requires specific decisions, dependent on the scope and epistemology underpinning the question, as well as the time and resources available (Gough et al., 2017). Although presented as a linear journey, these processes are often iterative rather than linear.

The first stage comprises review team formation, stakeholder engagement activities and defining the scope of the review (including specifying the questions it will address and selection criteria). Initial conceptual and definitional work forms the basis of the review’s conceptual framework.

The second stage involves searching for and selecting studies. The search strategy includes a description of relevant sources, core keywords and an outline of benefits and costs for different combinations of sources. Selection of studies is made against prespecified eligibility criteria.

The third stage involves describing the studies that have met the review’s inclusion criteria to gain a high-level overview of the research literature relevant to the review. Codes relate to variables such as country, types of outcomes and study design. The emphasis is on summarising what evidence there is, rather than what the evidence says (Saran and White, 2018). The mapping stage forms a key step in analysing a group of studies, irrespective of whether this is a precursor to a synthesis. Given the large number of studies that may be identified, the map can be used to help reviewers and other stakeholders decide on which areas of the evidence to focus in greater depth at a later stage (James et al., 2016).

In the fourth stage, reviewers undertake more detailed coding, going beyond simply describing a research field. Data extraction involves gathering individual study findings for synthesis suited to answering the review question. Quality assessment also prepares studies for analysis by ensuring that ‘only the most appropriate, trustworthy and relevant studies are used to develop the conclusions’ (Gough et al., 2012a: 154). Dependent on their question, systematic reviews vary in their approach to quality assessment, but it is widely accepted that all should ‘aim to avoid drawing misleading conclusions because of problems in the studies they contain’ (Gough et al., 2012b: 4). Using standardised coding frameworks helps ensure consistency, while duplicating the data extraction and quality assessment processes (where at least two people work independently) reduces both the risk of making mistakes and the possibility that coding is influenced by a single reviewer’s biases (Li et al., 2021).

As the final stage, the synthesis of findings can lead to new knowledge based on information collected from individual studies. Given the diversity in synthesis methods (Barnett-Page and Thomas, 2009), it is useful to distinguish between aggregative and configurative approaches. Aggregative reviews are closely related to theory testing, and studies included are often relatively homogeneous. Configurative reviews are likely to include more heterogeneous studies, and they are closely related to hypothesis generating and theory exploration. Most reviews, however, include some elements of configuration and aggregation, reflecting study design and methods of analyses (Gough et al., 2017).

Rapid reviews

The term ‘rapid review’ denotes an acceleration in the process of reviewing a body of literature. A wide range of terminology is used to describe this approach to evidence synthesis, such as ‘brief review’, ‘scoping review’, ‘rapid evidence synthesis’ (Tricco et al., 2015) and ‘restricted systematic reviews’ (Plüddemann et al., 2018). These terms indicate a hastening of the synthesising process, denoted by the notion ‘rapid’, and they suggest a restriction in the scope, and potentially the quality, of the work, through use of adjectives such as ‘restricted’.

Rapid reviews should be understood as a spectrum of different products. For health, Featherstone et al. (2015) found a difference in their purpose and aims, as well as methods and definition. While they are often regarded as a variation of a systematic review balancing time resource constraints with considerations for bias, some do not follow a specific methodology. Those rapid reviews that closely resemble systematic reviews adapt established methodological approaches by accelerating one of the common stages of a systematic review in some way (for example, database search) or by leaving out a stage altogether (for example, quality assessment). Before review initiation, researchers should consider the steps that might be reduced compared to a standard systematic review, and the potential consequences of taking a different, more pragmatic, approach. Final decisions about synthesis methods take place following assessment of the evidence base.

By rapid reviews here, we mean those systematic reviews that apply fewer resources and are less ambitious than a full systematic review, while still aiming to limit bias and ensure transparent reporting (Schünemann and Moja, 2015). Given that rapid reviews have no established definition, choosing an appropriate terminology might imply a mindful description of the review question. Most important, however, is to concentrate on a full description of the review stages (Gough et al., 2012b).

Finally, one should keep in mind that there is always a trade-off between time and resources to conduct a review, and the comprehensiveness of the final product.

Table 1 presents the common stages in a systematic review, and provides examples of acceleration strategies that might be used in rapid reviews. As noted by James Thomas et al. (2013), their applicability will depend both on the individual context and on the expertise of reviewers.

Review 1: employment (Burge et al., 2012)

This policy-oriented review was funded and carried out by the National Foundation for Education Research (NFER), a leading independent provider of education research that exists to improve outcomes for young people. It is one of four reviews related to the theme ‘From Education to Employment’ that collectively identify strategies to support young people at risk of becoming NEET (not in education, employment or training) to make effective transitions into learning or employment. The study scope included five research questions: (1) In what ways do employers engage with primary and secondary schools in the UK and abroad?; (2) What are the key features and principles of successful employer involvement?; (3) What is the impact of employers’ involvement with schools on young people progressing to education, employment or training after compulsory education?; (4) What gaps are there in the evidence base?; and (5) What are the implications of the findings of this review for policy and practice?

Review 2: school closure (EEF, 2020)

The intended audiences for this review are teachers and key decision makers in schools, as well as policymakers working with innovations in education. It was funded and carried out by the Education Endowment Foundation (EEF), an independent charity with the vision to break the connection between family, income and educational achievement. The review is authored by a large experienced team from the EEF and other institutions. Drawing on guidelines by the Cochrane Collaboration Rapid Review group (Garrity et al., 2021), the Civil Service Rapid Evidence Assessment (Government Social Research Service, 2009) and the Cochrane Collaboration on overviews of reviews (Pollock et al., 2020), this rapid review assessment was motivated by the COVID-19 pandemic and resulting school closure in many countries. Given large heterogeneity, it aimed at including all relevant evidence, focusing specifically on the differential impact of school closure on disadvantaged students. There are three closely related research questions: (1) What evidence currently exists about the impact of different kinds of school closure (for example, due to summer holidays) on differential academic attainment for disadvantaged/others, and on other outcomes related to education (for example, impact on IQ)?; (2) What factors moderate the impact?; and (3) What evidence and theory helps us to understand the mechanisms by which school closure leads to learning loss and widening of attainment gaps?

Review 3: digital technologies (Major et al., 2018)

This review claims to be the first that links school-based classroom dialogue and digital technology, an emerging and relatively new area of research. Supported by the Research Council of Norway, and published in the international journal Education Information Technology, the review is aimed at researchers, policymakers and practitioners. The guiding questions were formulated broadly to incorporate an extensive base of existing literature: (1) In what ways does research suggest that use of digital technologies enhances productive classroom dialogue?; and (2) What challenges are reported that may impact on the successful use of digital technology to support dialogic teaching and learning?

Review initiation and review questions

Since rapid reviews are designed to deliver information in a relatively short period of time, an experienced team ideally needs to be in place at the outset. Restricted timescales can make recruitment and appointment of a new researcher very difficult. Reporting a time frame of eight months, Review 1 (employment) was carried out by researchers with expertise in all the areas expected of a review team. This study was part of an NFER programme of work to inform policy and practice. Review 2 (school closure) aimed to publish its findings within six weeks of starting the review. It used a relatively large review team, consisting of four experienced senior researchers and an additional ten people for screening and data extraction. No information is reported on the time frame for conducting Review 3 (classroom dialogue and digital technologies), or on team experience. None of the three reports describe involvement of external stakeholders (a decision which can speed the early development stages).

All three examples review relatively broad research areas, but they are scoped differently in terms of overall review objectives and specific questions. Review 1 had an intervention focus and addressed diverse types of question relating to strategies supporting young people at risk of becoming NEET. While a key aim is to investigate the impact of employer involvement on young people’s achievement and progression, additional questions go beyond this to capture the ways in which employers engage with schools, and the features and principles of successful school–business partnerships. Review 2 was also conducted to inform policy decisions. Initiated in the context of the COVID-19 pandemic, it answers a series of deliberately narrow questions relating to the impact of different kinds of school closure on educational outcomes, the effects of moderating factors (for example, age), and the underlying mechanisms of change. By contrast, the main purpose of Review 3 was to examine a body of research to provide an overall picture of the available evidence and to highlight knowledge gaps in the field. Such results generally allow for the identification of future research initiatives, including whether a full systematic review is needed, rather than directly informing policy or practice. Arksey and O’Malley (2005) applied a scoping review approach, while they extended the conventional parameters of this type of review through a thematic synthesis of the study findings.

It is recommended that authors develop the scope of their review with care, even when selecting concepts that have been well defined elsewhere (Thomas et al., 2020). This helps prevent conceptual incoherence and irrelevance, and it will save time in later review stages. All three examples discussed here used existing definitions for core concepts, but to different degrees. Review 1 was to some extent able to proceed along predefined boundaries in terms of the concept of NEET, although this phenomenon (disengagement among youth) is understood and named differently in countries other than the UK (Holte et al., 2019). Review 2 examined school closure and a wide range of academic and non-academic outcomes. Studies in the meta-analysis defined and operationalised concepts in different ways. Review 3 focused on a more complex field of inquiry, combining the concepts of dialogue, schooling and digital technology. This review team revealed spending time thinking through concepts to operationalise the search strategy effectively.

One of the main similarities between the reviews was the decision to limit inclusion to certain types of study. All three specified language, publication date and/or document format restrictions in the eligibility criteria for studies. For instance, Review 3 included only peer-reviewed studies, whereas the other reviews also considered grey literature. While at least two of the reviews developed a protocol as the basis for the ongoing research, none appears to have published or registered a formally peer-reviewed protocol before proceeding further with the review.

Identification of relevant studies

All three reviews addressed multiple questions and incorporated different types of evidence, so searching for and selecting studies was more complex than an effectiveness review answering a single question (Cooper et al., 2018). Although the level of detail reported about the search strategy varied across the reviews, they used broadly similar strategies, with completeness of searching determined by both time and scope constraints. None of the reviews aimed for exhaustive searching; instead, each used a targeted and specific search approach limited to key resources (for example, academic databases). Reviews 2 and 3 both reported extensive information on the search strategy. Review 1 provided few details on this or on the study selection process. There are indications that, to save time, both Review 2 and Review 3 chose not to duplicate the study selection process and used only one reviewer to screen abstracts. However, because of the substantial potential for errors, both reviews included quality assurance and control measures to help alleviate concerns relating to the misapplication of the eligibility criteria. The researcher in Review 2 had the option of marking items as ‘unclear’ for review by a second person; in Review 3, another team member double-checked a sample of included studies. Review 2 also used text-mining technologies to speed up the selection of studies. A ‘priority screening’ function was used to order results by probability of inclusion (based on a training set), and screening stopped once relevant studies were no longer being identified (O’Mara-Eves et al., 2015).

Description of study characteristics (mapping the research field)

In response to time and resource constraints, Reviews 1 and 2 both opted to omit the mapping stage. Neither review describes the extent and nature of the included literature before examining it in depth. Review 3 chose not to employ either of the two main ways of accelerating this stage: collecting information based on titles and abstracts only, and mapping only a representative sample of included studies. Instead, it mapped all relevant evidence to highlight areas of saturation and research gaps in an emerging field, with the suggestion that full-text reports were used to collect information.

Detailed coding (data extraction and quality assessment)

None of the three examples reported use of a standardised data extraction form which, as well as providing consistency in a review, has the added advantage of saving time. Review 3 refers to two reviewers independently coding each study, suggesting no acceleration strategy was used. In contrast, Review 2 reduced time spent on data extraction by using a single reviewer, with a sample (10 per cent) of studies double data extracted so that coding reliability could be assessed. This review also saved resources by limiting data extraction to a minimal set of required coding items. Information on the data extraction process is missing for Review 1.

For all three reviews, the level of detail reported about the quality assessment of included studies was poor. The protocol for Review 2 indicates that individual studies were judged as ‘low’, ‘medium’ or ‘high’ quality (that is, ratings that reflect the extent of confidence in the estimate of effect). Review 1 reports the use of three categories for quality assessment: ‘strong’, ‘moderate’ and ‘impressionistic’ evidence. Neither of these reviews provides details of the tool used or offers a statement of its validity (Majid and Vanstone, 2018). Quality assessment is generally not a priority in scoping reviews, and Review 3 did not assess each of the included studies. Although not explicitly stated, the authors may have used peer-reviewed publication as a proxy for good quality.

Synthesis

None of the three examples reports using abbreviated synthesis methods. Yet they all accelerated this stage of the review process to some extent. Review 2 applied an aggregative approach using meta-analysis, which was limited to the most important outcomes to ensure that the review could be completed within the timeline for the project. The authors also modified the original research questions to reduce the scale of the review from 58 to 11 included studies. Often, synthesising evidence narratively can save time in a rapid review context, and Reviews 1 and 3 both took this approach. To cope with the large number of eligible studies, Review 3 chose also to adapt an existing thematic synthesis method (Thomas and Harden, 2008) involving inductive line-by-line coding of text to generate descriptive themes. This process was undertaken by two reviewers working independently, then collaborating with other team members to revise, refine and finalise themes. No attempt was made to further theorise the themes presented.

Table 3 compares the challenges, strengths and limitations of the chosen rapid review approaches across the three review examples.

Consider temporal, financial and personal resources

Timescale and other conditions set by the commissioning body can affect decisions regarding the breadth and depth of the review, shaping processes and outputs accordingly:

• As a rule, it is difficult to review broad areas of research quickly. Rapid reviews tend to address highly refined research questions, as these result in a smaller set of studies being required in the synthesis. However, reviewers should be wary of defining the scope and focus of a rapid review so narrowly that it fails to meet user needs.

• For rapid reviews in complex fields of policy inquiry or less-established research areas that lack accepted or common operational definitions of concepts, time and resources may need to be weighted towards the earlier stages of the process.

• Reviews of controlled trials may take less time to complete than other types of review because there are more supporting resources (for example, critical appraisal tools) available to review this type of research.

• If the review team lacks previous experience of carrying out systematic reviews, it might be best to undertake a systematic map of the research field that is limited to describing a set of studies. However, a rapid review lacking appraisal of study quality and synthesis of findings may not align with policy needs.

Think broadly about different kinds of team experience and expertise

In selecting a review team, it is important to consider variety and breadth of expertise among its membership:

• It is good to have a mix of methodological (knowing how) and substantive (knowing what) experience and expertise within review teams.

• Stakeholder involvement is not always a cause of delay, especially if the reviewers draw on existing networks and have experience of collaborative work. Support from advisory groups may help focus the resources and time available.

• Having review team members with prior knowledge of the field, especially conceptual issues relating to the topic areas under review, can be important in making progress under tight timescales.

• Take the expertise of team members into account when allocating roles and responsibilities. For example, use experienced team members to check the accuracy of what novice reviewers have done and to resolve coding disagreements.

Estimate the size and nature of the literature to be reviewed at the outset

It is important to have some understanding of the amount, type and variation of evidence available before deciding which of the reviewing options are most appropriate. Quick scoping searches based on a simple search strategy can be undertaken while developing the protocol to provide an overview of the studies likely to be retrieved. A common challenge is balancing the amount of time spent on scoping the review and identifying studies, with the amount of time spent appraising and synthesising:

• Lack of an established, recognised field might mean deciding on a deliberately inclusive approach to study selection while streamlining later processes (for example, data extraction and synthesis).

• Identification of a large volume of literature in an initial scoping search may present a challenge, with too many hits for screening impacting on the time available to undertake detailed appraisal and synthesis of included studies. Solutions include narrowing the scope, so that the total number of records is manageable in the time frame (for example, by abandoning some research questions or restricting eligible study designs to randomised controlled trial evidence).

• The need for flexibility is also crucial when the final search identifies far more research than originally expected. One response is to reduce the level and detail of study appraisal and/or synthesis.

• Expectations of perceived weaknesses in the evidence base, such as lack of randomised controlled trials for reviews that have an intervention focus, can shape the review team’s view of what kind of review is needed, for example, one on the effect of learning motivation interventions.

• Conducting a review of existing systematic reviews is one solution to broad topics, as well as to the challenge of completing a review in a faster time frame. Systematic reviews (or overviews) of reviews can save time and resources, because reviews are often easier and quicker to locate than primary research (for example, using registers of published systematic reviews) and their findings may be summarised narratively, with little real synthesis being required. It is important to appreciate, however, that there are also inherent limitations to using this approach, partly because available systematic reviews are likely to be of diverse quality and scope.

Consider the purpose of the review and nature of the topic

To ensure that reviews are ‘fit for purpose’, identify reviewing approaches that are relevant and feasible for the review purpose and topic:

• The more likely it is that a rapid review will be used to inform policy or practice decisions, the more careful reviewers need to be about the shortcuts taken. Undertaking a thorough quality appraisal of the evidence is seen as critical for policy-oriented reviews to increase confidence in the validity of the findings. The costs and benefits of different ways of accelerating (or even abandoning) this stage of the review process require careful consideration.

• Alternatively, if the emphasis is on mapping research activity and highlighting knowledge gaps, reviewers generally have more time to spend on definitions and study identification, since evidence maps generally do not judge research quality or combine the findings of studies.

• Typically, the main purpose of the review and the topic will influence whether it is more appropriate to present an in-depth analysis of a small number of studies or less information from a wider range of studies.

Choose a synthesis method appropriate for the review purpose and question, the evidence base and the reviewers’ expertise

It is important to clarify priorities and strategies for analysis at an early stage, so that the other review stages can be shaped accordingly:

• Thematic summaries of the results of the included studies, or meta-analysis based on primary outcome(s) (if data support the use of this statistical procedure), are often the most suitable modes of synthesis in rapid reviews.

• The type and depth of synthesis that it is possible to conduct will, to some degree, depend on the reviewers’ familiarity with different approaches to research synthesis.

Provide a full description of the rapid review approach and discuss the potential limitations and biases of the chosen methods

Although generally less ambitious than a full systematic review, rapid reviews are expected to conform to many of the same standards. To ensure that users have confidence in the final product, they need to be made aware of the limits placed on the review, and the impact of decisions taken:

• Review methods should be described in full to ensure that the review process is transparent, reproducible and accountable.

• Since shortcuts are likely to introduce bias, and can affect confidence in the conclusions of the review, transparency will be increased by highlighting where standard systematic review methods were omitted or modified.

• Limitations and biases associated with the chosen methods (for example, risks of introducing publication bias due to reduced searching) should be reported, and the implications for the review findings discussed.

Be mindful of the anticipated audiences and tailor outputs accordingly

To increase uptake and use, it is important to consider how best to present the evidence required by users of the review (for example, its commissioners) and wider audiences:

• The complexity of the evidence base, and the type of audience anticipated, will influence the likely structure, format and length of the final report, including how much data should be presented, in what ways and in what level of detail.

• There is a range of user-friendly communication tools (for example, summary of finding tables) that can be used (Petkovic et al., 2018).

• When producing evidence summaries (Khangura et al., 2012), it may be less important to adhere to PRISMA reporting standards (Moher et al., 2015). However, detailed information on review methods should be made available elsewhere (for example, as an online supplement).