The challenges of assessing the impact of a comedy programme aimed at improving the mental well-being of young people

Introduction

Since 2002, The Comedy Trust (TCT) has been delivering community-centred workshop programmes to support the personal and social growth of the public through the art form of comedy. The Feeling Funny Youth programme was established in 2018 and aims to create an alternative support system for young people to help them feel happier, and to gain personal skills in areas such as their confidence and self-esteem. It also aims to improve and create relationships among like-minded peers, and provide participants with a creative platform to express themselves while tackling difficult situations and topics, such as talking about their mental well-being in a light-hearted, accessible way. In 2018, TCT successfully applied for research funding from the Cultural Education Research Initiative (CERI) to work in partnership with Liverpool Hope University to evaluate their Feeling Funny Youth programme. One of the key aims of the CERI initiative is to support the development of research-informed, evidence-based practice for cultural organisations working with children and young people in North West England. During our initial meetings, we decided we would like to understand why the standardised tool that TCT had been employing to capture the impact of their programmes (the Short Warwick–Edinburgh Mental Well-being Scale – SWEMWBS) was not deemed suitable, and if we could design more effective methods which align with the nature of the comedy programme.

The Feeling Funny programme seeks to improve the mental well-being of young people through comedy. There is a plethora of research which emphasises the positive impact of the arts on the well-being of young people; however, this tends to focus on music, drama, dance, singing and visual arts (Lee et al., 2020; Wang et al., 2020; Zarobe and Bungay, 2017). Although rather limited, there is a growing body of evidence which highlights the positive impact that humour can have on an individual’s health and well-being (Crawford and Caltabiano, 2011; Twardzicki and Jones, 2017). A systematic scoping review conducted by Gonot-Schoupinsky et al. (2020) explored the many benefits of laughter, which include a reduction in anxiety, tension, stress and depression, and an increase in self-esteem, hope and energy. It also helps to provide a sense of empowerment and control. Furthermore, it has been suggested that humour can help reduce the stigma associated with talking about mental health (Corrigan et al., 2014). The Feeling Funny programme employs the art form of comedy to provide space for young people to talk about their mental health. An outline of the programme and techniques employed is provided in Box 1.

TCT hoped that through this research project, they would be able to provide concrete evidence to support the case for using alternative methods of evaluation and impact when applying for funding. There is a growing body of evidence which calls for more flexible, creative and participatory methods to be recognised (Cook et al., 2017; Holt, 2004; Dunn and Mellor, 2017; Lee et al., 2020). Indeed, it has been suggested that this type of research can provide a more accurate account of how a programme is working, and therefore the impact it is having on the community (Elmore et al., 2019; Lenette et al., 2019). It is important to recognise the issue of capacity for small charity organisations, as these methods take time to establish, and for people to develop. TCT wanted to use the research not only to highlight to funders the financial implications, but also to share what can be gained by taking a more participatory and creative approach. Overall, TCT hoped that with the evidence gained as part of this research, they would be able to develop a new and improved practice that would truly capture the participants’ journey without the added pressure of clinical scales that take away from the fun and creativity of the workshop programmes.

The project

For many years, researchers have highlighted the importance of working collaboratively with young people to move beyond ‘top-down’ approaches when assessing impact (Alderson et al., 2019; Doucet et al., 2021; Hodges et al., 2014). The benefits of adopting such approaches include the potential for social change, as it empowers communities that are often disempowered, and ‘the potential to strengthen participation in service provision and delivery’ (Smithson and Jones, 2021: 358). Shifting to working alongside rather than ‘on’ young people allows the opportunity to generate and produce knowledge with them (Bradbury-Jones et al., 2018; O’Brien and Dadswell, 2020), as they are placed as the ‘expert’ (Liddiard et al., 2018). Indeed, as argued by Doucet et al. (2021: 2), this ‘allows for the development of conceptual frameworks that value and validate the realities and perspectives of youth’. Yet despite these many benefits, there are still challenges that come with such approaches. Adults working with young people are required to be reflexive throughout the research process and production to acknowledge issues such as inherent power imbalances between the adults and the young people (Kirby, 2020; O’Brien and Dadswell, 2020). Researchers are being encouraged to acknowledge the ‘messiness’ that comes with participatory research (Cook, 2020; Facca et al., 2020).

Although TCT have worked across a range of different educational provisions to deliver their Feeling Funny programme, for the purposes of this paper we focus on how the programme worked within one setting, a youth organisation. The youth organisation offers support to disadvantaged young people outside their educational environment and is based in North West England. The young people involved in this project had encountered a range of difficulties during their educational journey, leading them to disengage from education. All had received a Special Educational Needs (SEN) label, were experiencing issues around their mental health, and were likely to have a range of services involved with their care. Such young people would potentially fall into the category of ‘seldom heard’ (Ní Shé et al., 2019). This term is employed within academic literature to describe a wide range of participants who struggle to have their voices heard (Cook, 2020; Hodges et al., 2014). It was important to be aware of the wider circumstances that surround the young people. For example, they were likely to encounter a wide range of professional adults, and potentially had rather negative experiences with regard to sharing their stories. It is widely acknowledged that having an SEN label attached shapes how young people are responded to by adults (Caslin, 2019; Liddiard et al., 2018). Within educational settings, it is often the adult’s voice which will dominate any discussion surrounding the young person’s needs (Dimitirellou and Male, 2020; Bourke and Loveridge, 2018). Therefore, making the adjustment to being on an equal footing with the adults in the room may feel very unfamiliar (Bradbury-Jones et al., 2018). Spending time with young people to get to know them and to build trusting relationships prior to any data collection taking place was essential, as was making sure that they understood the purpose of the research (Bourke and Loveridge, 2018; Groundwater-Smith and Mockler, 2016).

Research process

During the first session, all the young people were informed of the research and asked if they would be willing to have their experiences captured. The first part of the research was trying to capture the young people’s experiences of taking part in the programme. All the young people were also asked to complete the standard tool at the start and at the end of the programme. Young people were also invited to participate in semi-structured interviews to gain a more detailed insight into their experiences. Five young people agreed to take part in interviews before, during and after the programme. The young people were provided with a series of prompts and were asked to share their responses. It was up to the young people to decide how they would like to respond; for example, they could draw, write or just talk about their experiences. These sessions focused on what the young people hoped to get out of the programme, how the programme was going, and what they felt they gained from taking part in the programme. Following on from this, the young people were invited to form a participatory research group. The focus of this group would be trying to identify any issues experienced when completing the standard tool, to establish why this tool was not considered effective in capturing young people’s experiences of the programme. For the purposes of this paper, we will focus on the experiences of the three young people who formed the participatory research group. These young people took on the role of research subject and co-researcher (Dunn and Mellor, 2017). An overview of this research process is provided in Figure 1.

Figure 1.

Overview of research process (Source: Authors, 2022)

Ethical considerations

Throughout the study, ethical considerations were of the utmost importance. The study gained ethical approval from Liverpool Hope University Ethics Committee and adhered to British Educational Research Association (BERA, 2018) guidelines throughout. In a study of this nature, issues around power relations were significant. To help develop trusting relationships with the young people, the researcher attended all the sessions and participated in some of the games and activities to help build relationships. The data collection techniques developed were discussed with the young people, and flexibility was essential to allow young people to decide how they wanted to participate (Groundwater-Smith et al., 2015; Michail and Kellett, 2015). It was considered important that the participants felt part of, and in some ways took ownership of, the research process. One mechanism employed to achieve this was to ask the young people to decide how they would be referred to throughout the research. Within the findings section of this paper, all the names used are those chosen by the young people themselves. In addition, the study adopted ‘process consent’, with consent being negotiated as an ongoing concern throughout the research process (Heath et al., 2004). The young people were provided with ongoing opportunities to agree to continue or to withdraw at any stage (O’Reilly and Dogra, 2017).

We wanted to work with the young people at every stage of the research process, which included the analysis of data. Throughout the research, ‘data’ took many formats, including the use of Post-it Notes, storyboards, conversations, posters and the standardised tool (Dunn and Mellor, 2017). We discussed the data that had been gathered during the semi-structured interviews and participatory research group with the young people. During our conversations, we were able to identify themes that we felt were important to share as part of the research. Due to the nature of the research, data production and analysis were an ongoing process, and we worked collaboratively with the young people to make meaning of the data (Alderson et al., 2019; O’Brien and Dadswell, 2020; Smithson and Jones, 2021). The young people were given access to all the findings prior to dissemination to ensure that they were happy that their voices had been captured accurately.

For the purposes of this paper, we have chosen to focus on two key challenges encountered by TCT:

1. meeting the expectations of external funders and the standardised tool

2. the development of creative and participatory methods that align with TCT programmes to capture the impact of comedy on young people’s sense of well-being.

Challenge 1: meeting the expectations of external funders and the standardised tool

Like many small charity community-based organisations, TCT are reliant on external funding, and this often comes with expectations of how impact will be captured. It is widely acknowledged that funders tend to prefer quantifiable, statistical data to demonstrate impact (Greenhalgh et al., 2016; Lenette et al., 2019). Determining whether an intervention has had an impact on an individual’s sense of well-being is neither easy nor straightforward. The subjective nature of well-being makes it difficult to make universal claims (Fattore et al., 2019). The difficulties of assessing the impact of community-based programmes are widely acknowledged (Cook et al., 2017; Hokanen et al., 2018). For TCT, their funders had asked them to employ the Short Warwick–Edinburgh Mental Well-being Scale (SWEMWBS).

The standardised tool

SWEMWBS is a widely employed tool with a significant number of studies advocating its use (for example, Rogers et al., 2018). The SWEMWBS is a shortened version of the Warwick–Edinburgh Mental Well-being Scale (WEMWBS) that uses 7 of the 14 items with an alternative scoring system. TCT based their standard tool on the SWEMWBS. (For an example of what the young people are asked to complete, see Figure 2.)

Feeling Funny standard tool, based on SWEMWBS (Source: Authors, 2022)

The tool enables the user to track well-being over a period of time and is used widely within mental health programmes and studies (Haver et al., 2015; Hunter et al., 2015). When using the SWEMWBS, an alternative scoring system called transformational scoring is used to assess impact. A three-point increase between the initial scoring and post scoring is considered as a considerable change in well-being. Due to the tool’s national recognition and success in other community projects, this appeared to be the best choice for TCT. However, it soon became clear that this may not be the most appropriate tool for this programme. Upon its initial use, TCT found that, for a significant number of participants, they had scored lower on their post-programme score. This made it appear as though their sense of well-being had deteriorated since taking part in the programme. These results also did not collate with the data gathered from participants when using more participatory methods. TCT wanted to find out why this was happening, so they ran a series of focus groups with people who had undertaken a course with them. Through discussions with the participants, they suggested a number of reasons for why this might be the case. For example, participants stated that they did not understand the terminology used, such as ‘I feel useful’. Many of the recipients felt that having a form to complete did not fit with the nature of the programme, which is about having fun and being creative. For many, this was an opportunity to escape from more formal mental health interventions, and several participants indicated that external factors would have an impact on their responses to the tool. Interestingly, another comedy-based programme aimed at supporting people experiencing mental health issues encountered difficulties when applying this tool. This was due to the relatively small number of participants. They also questioned what is meant by a ‘meaningful’ transformation regarding participants seeing a three-point increase between their pre and post score (The Comedy School, 2017). The employment of the tool does not allow for an opportunity to gain insight into what participants take away from completing the programme. Having a set of predetermined outcomes potentially misses the opportunity to gain a real insight into how the programme is experienced (Ross, 2017). The first stage of the research process was to explore young people’s experiences of completing the SWEMWBS.

Challenge 2: the development of creative and participatory methods that align with TCT programmes

Aware of the limitations of employing a standardised tool to capture impact, we set out to develop research methods in collaboration with young people that would be aligned with the nature of TCT programmes. There is a growing body of literature which supports the employment of more creative methods, especially when working with children and young people (Brady and Preston, 2020; Caslin, 2019; Hewitt et al., 2018). The importance of young people taking on the role of researcher has also been acknowledged (Elmore et al., 2019; Hopkins et al., 2017). By placing young people at the heart of this project, we hoped to develop approaches that are flexible, creative and responsive to their needs, so that they are truly reflective of their experiences, and enable us to hear their stories (Elmore et al., 2019).

Discussion

This paper demonstrates the difficulties of assessing the impact of a comedy programme on the well-being of young people. Cultural organisations want to ensure that their programmes are having a positive impact, so it is essential that they find ways to capture this. It became clear throughout this project that the standard tool alone is not suitable for a programme of this nature. Although there is evidence which supports the use of this tool (Haver et al., 2015; Hunter et al., 2015), it is not considered appropriate by practitioners or by young people involved with TCT. It is important to recognise that there may be particular features of the programme that mean the tool is considered unsuitable – for example, the length of the programme, the diversity of the groups of young people with which TCT are working, and the numbers of young people who take part in their programmes. However, by employing more creative mechanisms, and more ways of engaging with young people alongside standardised approaches, we were able to gain an insight into the experiences of young people who have participated in the Feeling Funny programme, and the stories they wanted to share.

The findings are based on a relatively small number of participants, where emphasis was placed on gaining a detailed and in-depth insight into how the young people experienced the programme (Silverman, 2005). The study does not seek to make generalisations, and it is important that the reader is aware that this paper only provides a snapshot. The paper seeks to encourage professionals to develop more creative, participatory and flexible research approaches which afford the opportunity not only to hear the voices of young people, but also to develop programmes that are responsive, relevant and led by the needs of the community (Elmore et al., 2019; Lenette et al., 2019). Such research intends to provide an account of the specific situation that gets ‘sufficiently close to its underlying structure to enable others to see potential similarities with other situations’ (Winter, 2000: 1). In addition, although the paper is focused on a small number of young people, it is reflective of issues that TCT have experienced in evaluating all of their programmes. TCT have been aware of these issues, and this was an opportunity for them to capture how the process is experienced by the participants of their programmes. Furthermore, the collaborative nature of this project offered us the opportunity to develop approaches that were led by the participants (Alderson et al., 2019; Doucet et al., 2021; Hodges et al., 2014). It is also important to recognise that a ‘one size fits all’ approach is deemed inappropriate, as it can undermine the experiences of young people (Dunn and Mellor, 2017). Therefore, although the animation video worked well in this setting, this may not be the best approach for other groups that take part in TCT programmes. We are reminded to acknowledge and address the ‘messiness’ that comes with participatory research (Cook, 2020; Facca et al., 2020). One of the biggest challenges we encountered was maintaining contact with all of the young people. However, we feel it is important to recognise the commitment shown by the participatory research group; even though James was not able attend all sessions, he did maintain contact with the team, and he was able to rejoin the group at later stages. With this project, we were able to immerse ourselves in the research process to help build trust and develop relationships in a way that is not possible to achieve with a one-off top-down evaluation (Ross, 2017).

It is also important to consider the financial implications of doing this type of research, and to acknowledge the position of cultural and arts-based organisations in terms of accessing funding. It is widely acknowledged that funders tend to prefer certain methodological approaches which privilege predefined performance indicators (Greenhalgh et al., 2016; Lenette et al., 2019). When we shared the video with TCT funders, the response was very positive; they liked the video, and felt it was a good way to capture the impact of the programme. However, they stated that they would still like the more traditional, quantitative, statistical data alongside this. We do not advocate an either–or approach; rather, we hope to highlight that which may be lost if we do not seek to develop creative and participatory approaches to research. If we are able to highlight what can be gained by taking this approach, we hope to encourage funders to recognise not only the value of this type of research, but also the time and money that need to go into this work.

Through the process of undertaking this project, we learnt that the mechanisms designed to capture the impact of TCT programmes need to be affordable, flexible, accessible, creative and participatory. It is only by adopting these approaches that we will be able to gain a true insight into the impact of the programmes. Through working collaboratively with practitioners, young people and researchers, TCT hope to be able to develop programmes that are responsive to the needs of their recipients (Alderson et al., 2019; Doucet et al., 2021; Hodges et al., 2014). TCT have set up a steering committee to help shape programmes which will include the voices of young people, and the information gained from this project will inform their future research production. There are many uncertainties surrounding arts provision in the UK, and TCT have been hit particularly hard by COVID-19; however, it is hoped that we will be able to build on this work in the future. We understand that change is an ongoing process (Wadsworth, 1998), and we are committed to continue being reflexive and responsive in our work.

Funding

This research was a Cultural Education Research Initiative (CERI) supported by Curious Minds, Liverpool Hope University. The Comedy Trust received funding from Liverpool One Foundation and Garfield Weston Foundation to run the Feeling Funny programme.

Declarations and conflicts of interest