Participatory evaluation for large-scale arts programmes: challenges, adaptations and unexpected shifts in culture

Implementation of Get Talking in Year 1

It was important to both the evaluation team and Appetite that the evaluation was embedded into the programme and produced meaningful findings that could influence the future direction of the programme. In the first year, Appetite’s programme comprised an intensive, six-week Taster Menu, a programme of high-quality arts events, an immersive circus and a selection of ‘pavement cafe’ events in the city centre each weekend. It also included a large-scale outdoor event in a city park, followed by a series of events, including a children’s theatre production and a partnership with libraries encouraging parents to engage their children in creative storytelling. The programme was designed not only to attract new audiences to the arts in the city, but also to offer new inspiration and insight into the new types of art that could be produced within the city. Alongside the programme, each consortium partner recruited a paid Appetite Builder, community engagement workers with direct responsibility for building relationships between local people and Appetite. These Appetite Builders were employed by the consortium organisations to work directly with communities, some of whom were not directly engaged in art or cultural activity, and to support them to access tickets, attend events and reflect on their experiences.

The evaluation and project teams were keen to involve local people with the evaluation. Participation, as a core principle of Get Talking, therefore provided a useful means of involving local people, with an interest either in the arts or in regenerating Stoke-on-Trent, in the Appetite programme by involving them in the evaluation process. Twelve community researchers were recruited to influence and inform the evaluation of the programme, and to gather feedback from audiences from across the city. The team included four Appetite Builders and eight volunteers who had an interest in art, or in working with and strengthening their own communities.

The university team gained ethical approval for the evaluation in April 2013, and they provided full training and support to the community research team. During the initial weeks of the programme, all consortium partners and Appetite staff team members were trained in Get Talking. The training covered the principles underpinning Get Talking and how these related specifically to the delivery of Appetite. It also provided the partners with an opportunity to identify key stakeholders for the project, and to embed a collective understanding both of participatory approaches and of the impact of including community partners in all stages of the process within the consortium.

The Get Talking short course, led by the evaluation team, was delivered one evening a week over the six weeks of the Taster Menu. The sessions provided participants with an opportunity to reflect on all stages of the evaluation research process, from identifying stakeholders and interviewing skills, to analysis of findings and action planning. Delivery of the training programme in this way provided a number of benefits. First, the Get Talking approach became integrated into the Appetite programme and provided community researchers with a live project through which they could test and refine their newly developed community consultation skills. Second, members of the community research team were able to get support from the evaluation team, and from each other, during the training sessions, which was crucial to their well-being and continued engagement with the programme. This was particularly important given the intensity of the data collection period, with events being held every weekend during the six-week Taster Menu period, and the numbers of face-to-face consultations, ranging from one hundred to two hundred at each event. A third benefit of integrating the Get Talking training into the live Appetite Taster Menu activity was that it enabled ongoing review and reflection on the findings as they emerged. However, although the community research team was involved in some analysis, the sheer volume of data meant that most of the feedback collected from the events was analysed by the evaluation team. Throughout the whole training programme, the research team modelled creative methods of engagement, providing the Appetite Builders with tools for engaging community groups and the whole team with training on use of some of the more common creative consultation techniques.

Data collection during Year 1 utilised creative consultation techniques that reflected the themes of the art available to audiences. During the Taster Menu, the evaluation team and community researchers were located in a ‘Travelling Tea Room’, a large and elaborately decorated outdoor tent, which helped to set the creative environment for audiences to engage with evaluators in conversation about the arts. The evaluation team designed bespoke creative tools fitting with the themes of the 14 performances and arts events. These tools included ‘quali-tea’ pots, where audience members were asked to vote on definitions of quality art using sugar cubes (Figure 2a), a six-foot arts wish tree to identify aspirations for art in Stoke-on-Trent (Figure 2b) and acrostic poems to gather audience feedback on a poetry performance. Each tool was designed to be creative and attract engagement. They also provided a means of collecting feedback, whether through written comments or voting. Community researchers were available to support audiences to write their feedback comments to ensure that people with lower literacy skills were not excluded from the process.

Figure 2.

Examples of creative consultation techniques used during Appetite’s Taster Menu (Source: intervention by Nicola Winstanley and Sarah Nadin, photographs by Jenny Harper). (a) Quali-tea pots; (b) Arts wish tree

As well as data gathering, creative techniques were also used to illustrate stories of people’s engagement with Appetite. Evaluators worked with art students at Staffordshire University, and with audience members, community hubs and volunteers, to produce creative case studies (Figure 3). Examples of these included a textile banner, illustrations and videos. Each creative case study was accompanied by a short text to provide further detail.

Figure 3.

Creative case studies (Source: intervention by Nicola Winstanley and Sarah Nadin, photographs by Jenny Harper). (a) Textile banner; (b) Illustrations

Challenges

Get Talking provided a useful foundation for the first year of Appetite’s evaluation – one that secured the role of evaluation in the Appetite programme and created a culture with audiences of debating and discussing the arts in Stoke-on-Trent. However, several constraints experienced during Appetite’s first year resulted in a pragmatic evolution of the evaluation methodology.

Smits et al. (2009) argue that increasing participation in evaluation can increase the team’s capacity to deliver. However, while community researchers, with support from the evaluation team at Staffordshire University, gathered feedback from relatively large audiences in Year 1, the sustainability of supporting them to engage with large audiences over a long period of time was questionable. The community researchers were a team of volunteers and, as such, management of the group was time and resources intensive. The university evaluation team reflected that during the planning stages, insufficient consideration had been given to the additional capacity and resources required to effectively support a team of community researchers to meaningfully engage in a research project of the size of the Appetite evaluation.

Equally, participatory evaluation ‘is and should be without the controls and checks that may protect certain stakeholders’ interests over others, which may be found in some more traditional forms of commissioned research’ (Squirell, 2012: 94), and participation in PAR involves people having ownership and control of the research process (McIntyre, 2008; Buhaenko et al., 2004). The Appetite team and consortium were supportive of the Get Talking approach to evaluation, and they welcomed the creative engagement of audiences through the participatory evaluation. However, as a Creative People and Places programme, Appetite was under increasing pressure for their evaluation to address specific questions, and to include larger sample sizes and extensive demographic data. Ledwith and Springett (2010) point to conditions that create barriers for authentic participation in organisational development. These include short-term funding; the introduction of top-down targets and indicators; some forms of knowledge, usually ‘scientific’, being revered over others; and the continued drive towards evidence-based practice, resulting in linear processes that limit opportunities to learn from dialogue and reflection. Equally, MacDowall (2015: 1) points to the increased dependence of cultural policy in recent years on the production of ‘accurate and relevant data’. It cannot be denied that some of the conditions described by Ledwith and Springett (2010) were present during the latter stages of Year 1, in particular the increased importance placed on participants’ demographic data and the specific needs of funders. This presented an ethical dilemma for the evaluation team. By labelling our approach to evaluation ‘Get Talking’, we were implying that local people would have an element of control over the whole process, and the team understood that participation can, if mismanaged, cause damage to communities and the reputation of organisations.

Perhaps the main challenge for Get Talking as a participatory approach was the scale of the evaluation programme. In comparison, the community research team was relatively small, with many having other family or work commitments. They were therefore unable to engage in all stages of the evaluation, and their activity became restricted mainly to data collection, the listening stage of the Get Talking process. For well-funded projects such as this, there are ethical questions of how much we are willing to ask of a small team of committed, but unpaid, volunteers. While PAR seeks to address ethical concerns of using outside researchers, Minkler (2004) points out that it is often the ‘professionals’ working on a project that are perceived by participants as having the most to gain from collaborative approaches to research, retaining paid posts and producing research outputs. As an evaluation team, we reflected on the ethics of requesting so much voluntary engagement of community researchers. It was clear that a more pragmatic approach was required through which we could offer a more realistic menu of opportunities for participation in the evaluation process (Gratton and Beddows, 2018).

A further challenge for the evaluation was the funder’s requirement for larger sample sizes than the conversational Get Talking approach could provide. As such, the evaluation team adapted the methodology to incorporate larger sample sizes and to supplement the largely qualitative data collection from Get Talking with quantitative and demographic data, reducing the time committed to collecting qualitative data, and increasing capacity for design and analysis of survey data.

Finally, the creative techniques used to evaluate Appetite events were intended as a way of starting a conversation that would be recorded afterwards by the researcher in written format. This approach ensured that details of conversations were captured without being intrusive for the research participant. Semi-structured interviews, however, take significant time with large audiences and this, combined with the need to gather feedback from larger samples, meant that we were relying increasingly on the creative consultation tools as a way of recording findings, reducing the emphasis on conversation as a means of learning. Creative consultation techniques therefore became a recognisable and attractive feature of the Taster Menu evaluation, with audience members seeking out the wish tree and other tools to offer their feedback.

These challenges (and opportunities) led to discussion and critical reflection among the evaluation team at the end of Year 1 of the Appetite evaluation. The university team felt under pressure to continually design creative tools, to attend events to support community researchers and to analyse findings. In addition, feedback from community researchers collected as part of the Get Talking short course indicated that although they enjoyed the engagement with Appetite, the schedule was intensive and exhausting. Appetite’s Critical Friend also played a significant role at this stage in supporting the evaluation team to reflect on the Get Talking process and implementation, how well it addressed the needs of the national evaluation, and the adaptations required to reach the sample sizes required and to maintain the well-being of the evaluation and community research teams. Overall, it was clear that, despite the evaluation team’s commitment to the principles of Get Talking, some pragmatic adaptations were needed for Year 2.

Pragmatic adaptations

During Year 2, the evaluation team introduced pragmatic adaptations to all three of the Get Talking elements – principles, process and creative techniques – driven by the need to deliver an evaluation that meets the needs of the funders, the programme staff and the communities engaged with Appetite.

Get Talking’s use of creative techniques, both to engage people with the evaluation and to tell the story of Appetite through case studies, was one of the most successful and recognised elements of the Year 1 evaluation. However, in Year 1, most of the creative tools and techniques had been created by the evaluation team or students for the case studies. Reduced staffing capacity at Staffordshire University in Year 2 meant that this arrangement was unsustainable. Equally, the artistic quality of the creative tools was inconsistent. Therefore, two socially engaged artists were commissioned to develop the creative techniques and, in many cases, to conduct the consultations, allowing the evaluation to mirror the artistic quality of the programme. The tools created by the artists were of high quality, and they attracted audiences to engage in discussions about the art on offer and, through support from the trained community research team, added capacity to the research team for the collection and analysis of data. Refocusing the evaluation methodology on the creative engagement techniques, and bringing in artists to deliver that element of the work, provided the research team with additional capacity to introduce more traditional research methods, including online surveys and focus groups, increasing the sample size as required by the programme and funders (Figure 4).

Artists’ creative consultation tools (Source: intervention by Nicola Winstanley and Sarah Nadin; photographs by Jenny Harper, 2015). (a) Ribbon wheel; (b) Firegarden light umbrella

Reflecting on the Get Talking model outlined in Figure 1, creative techniques therefore became a significant element of the Appetite evaluation. However, the core principles underpinning PAR took differing levels of priority in light of the challenges, resulting in an adapted version of Get Talking better suited to the needs of a large-scale arts evaluation.

The principles of participation and inclusion remained an important element of the evaluation, and community researchers continued to play a role in the collection of data by supporting delivery of the artist-led creative consultations. However, the evaluation of Years 2 and 3 became far more driven by a sole lead researcher who was more directive in delivery of the evaluations, in line with the Creative People and Places questions, and who ensured a balance of artistic creative consultations and online, post-event surveys to maximise sample sizes and demographic analysis. A crucial change was that the programme was no longer promoted as a community researcher-led project, therefore bringing expectations in line with the realities of working within a large-scale programme with clearly defined evaluation parameters.

The addition of online, post-event surveys was contrary to Get Talking’s inclusive principles, which traditionally avoid the use of techniques that rely on levels of literacy to participate. However, given the continued use of creative consultation techniques at Appetite’s events and performances, people were still offered an opportunity to provide feedback in a less traditional, more conversational way. The use of surveys provided the evaluation team with larger sample sizes and data through which qualitative data could be cross-checked.

However, the research team were acutely aware that, despite the broader reach and improved artistic quality of the creative consultation techniques, the reduced involvement of community researchers raised questions about how participatory the evaluation had become. Cornwall and Jewkes (1995) make the distinction between collaborative and collegiate participation. Our aspiration for the Appetite evaluation was that it would adopt a collegiate approach to research, in which the community research team were equal partners in the research process. However, in Year 2, our methodology evolved to accommodate the needs of the Appetite team and funders to maximise learning within the context of a Creative People and Places programme. Our amended model was a collaborative approach in which ‘researchers and local people work together on projects designed, initiated and managed by researchers’ (Cornwall and Jewkes, 1995: 1669). We were able to use this collaborative approach to value the contribution of local people in the process, within the wider constraints of the broader evaluation framework.

Building a culture of learning through an inclusive framework for critical reflection

Get Talking is underpinned by a cycle of questioning, dialogue, action and reflection. For three years, this provided Appetite with a valuable structure for action learning. With the active involvement of audiences and an ‘army of community researchers’, evaluation became recognised as a team activity, and seen as ‘a form of learning, rather than an opportunity to point blame at [what] hasn’t worked so well’. This built strong relationships and trust, and it enabled Appetite to be more responsive to community needs. In addition, the training offered to all consortium partners at the start of the Appetite programme embedded an understanding of Get Talking at both strategic and operational levels with Appetite. This, in turn, led to a shared commitment from partners to contribute to the reflective cycle, further enabling Appetite to be responsive to emerging findings.

Increasing audience engagement with Appetite

The use of Get Talking for the evaluation had a positive impact on audiences’ engagement with the programme. ‘Get Talking helped facilitate unbiased conversations with local people about what [art] they might like to see’, and the creative engagement techniques adopted ‘helped to fire everyone’s imagination’. For audiences, the evaluation of Appetite remained distinctive in its use of creative and artistic techniques for data gathering and, over the first three years of the programme, it helped to initiate and sustain a conversation about arts and culture among people who had previously had little experience or interest in the arts.

Extending the reach of Get Talking and Appetite

As a result of Get Talking training, the Appetite team, Appetite Builders and consortium partners gained new skills in participatory research and creative consultation techniques. The Appetite Builders used these skills and techniques to reach communities who initially had little experience of engaging with arts and culture, and to enhance their own practice: ‘this was something [Appetite Builders] could take away from the work they did as well’. This enabled them to build and sustain relationships with Appetite’s ‘community hubs’, generating ideas and consulting on options for art in local communities. Consortium members also applied their Get Talking training to other aspects of their work, ensuring that ‘the learning did not just stay with Appetite, but spread further into the city’.

Extending community involvement in Appetite

Community researchers became the ‘foundations on which the [Appetite] programme was built’. Working with community researchers enabled Appetite, and Staffordshire University as evaluators, to understand Appetite in the ‘context of the community’. For Appetite, ‘working with community members as decision-makers was a new thing’, and the use of Get Talking in the first year of the programme provided a framework that Appetite could use to co-produce more of their programme with community and audience members as they moved into Year 2. Appetite delivered an annual strategic programme of large-scale, high-quality, outdoor arts. At the end of the first year, the Appetite team used the principles underpinning Get Talking to inform a framework for community engagement with the strategic programme. They initially recruited two representatives from each community hub to join a group to discuss ideas for the next strategic programme, which became the Supper Club. Founded on participatory principles of co-production and inclusion, this group met monthly, over supper, to discuss new ideas, experience ‘tasters’ of art or visits to other areas of the country to experience new art forms, and to co-produce and evaluate large-scale, Appetite arts events. Now in its seventh year, strategic decisions about the programme are still made through consultation with the Supper Club. This model of co-production is firmly embedded into Appetite, and it has been used to attract funding for expansion of the programme into an adjacent area of low arts engagement.

Research ethics statement

The authors declare that research ethics approval for this article was provided by Staffordshire University ethics board.

Consent for publication statement

The authors declare that research participants’ informed consent to publication of findings – including photos, videos and any personal or identifiable information – was secured prior to publication.