| Summary of the findings |
| The value of public involvement in research has been well documented, but the literature is inconclusive about how to formally evaluate outcomes or ensure it is of a good standard, particularly in trials of complex interventions spanning several years. |
| This mixed-methods evaluation aimed to review and improve the public involvement activity in a complex intervention programme – PROSPER – using the six core standards for public involvement developed by the UK National Institute for Health and Care Research (NIHR). It further sought to assess the effect of public involvement in the programme on individuals, the programme and wider society. This study has tested the utility of using the NIHR standards to evaluate and improve public involvement structures in a complex intervention programme working with older adults with frailty. |
| Key messages | |
| • | The six NIHR core standards for public involvement are a useful tool for improving public involvement (PI) activity in complex intervention programmes over time. |
| • | The impact of public involvement activity in long-term programmes of complex interventions is multi-dimensional. |
| • | Peer research is an asset but requires a relinquishing of control, which is not always comfortable for professional researchers. |
Introduction
Patient and Public Involvement and Engagement (PPIE) is mandated by the National Institute for Health and Care Research (NIHR) (Involve, 2015). The NIHR Centre for Engagement and Dissemination in the UK offers advice about how, when and why to enact PPIE, but there is debate around how best to evaluate and report the outcome of PPIE activities (Crowe et al., 2020). In addition, there is a paucity of public involvement evaluation from studies with a target population of older adults with frailty (Mah et al., 2025).
In 2019, Greenhalgh et al. reported 65 existing frameworks to support PPIE in research. The NIHR standards framework was classified as ‘partnership focused’ and therefore suited to the evaluation of PROSPER, which is a complex, long-term intervention programme, involving multiple stakeholders, iterative development and evolving contexts (Greenhalgh et al., 2019). The standards offered a comprehensive, flexible and principle-driven approach aligned to the funder’s requirements.
PROSPER is an NIHR-funded programme designed to optimise and test personalised care planning to improve quality of life for older people with frailty (Heaven et al., 2020). It comprises four work packages: identifying the target population, optimising the intervention, carrying out a feasibility study and conducting a randomised controlled trial. The PPIE structure operates at multiple levels, both strategic and operational. The bi-annual programme and trial steering committees (P/TSCs) provide independent oversight. Bi-monthly programme and trial management groups guide operations and monthly locality groups review intervention delivery. In addition, selected individuals contribute on an ad hoc basis, for example through role-play training, review of project and dissemination material and analysis. All lay members of the programme have experienced frailty or cared for an individual with frailty. All are aged 65 and over. Figure 1 shows the PROSPER PPIE structure, the proportion of PPIE at each level and information flow.
Public involvement structure in the PROSPER programme
The NIHR standards
The NIHR standards aim to improve PPIE (Crowe et al., 2020). Each standard is a statement of good practice, with further sub-elements describing how it should be evidenced. Box 1 outlines the standards.
The six NIHR standards (Source: National Institute for Health and Care Research, n.d.)
Inclusive opportunities: Offer public involvement opportunities that are accessible and that reach people and groups according to research needs.
Working together: Work together in a way that values all contributions, and that builds and sustains mutually respectful and productive relationships.
Support and learning: Offer and promote support and learning opportunities that build confidence and skills for public involvement in research.
Governance: Involve the public in research management, regulation, leadership and decision making.
Communications: Use plain language for well-timed and relevant communications, as part of involvement plans and activities.
Impact: Seek improvement by identifying and sharing the difference that public involvement makes to research.
The NIHR recognises that high-quality PPIE needs to evolve. Success is not about meeting all the standards all the time, but about improvement over time. We used the publication of the NIHR standards as an opportunity to benchmark our PPIE activities in PROSPER. We asked, ‘What does good PPIE in research look like?’ and ‘What do we need to do to improve going forward?’
The evaluation took place at the end of the feasibility phase, which is work package 3 (WP3). This work was opportunistic and not part of the core programme funding. It did not require ethical approval, according to the Health Research Authority decision tool (www.hra-decsiontools.org.uk).
Aims
Our aim was to describe and evaluate PPIE in the PROSPER programme, and provide recommendations for improvements in work package 4 (WP4). Our objectives were to:
explore multiple levels of potential PPIE effect
map our PPIE onto the six standards set out by the NIHR.
Methods
We undertook a rapid mixed-method evaluation as part of the PROSPER programme which received ethical approval from Yorkshire & The Humber – Bradford Leeds Research Ethics Committee on 18 October 2018 (18/YH/0294).
Two lay members from the programme, CQ and SB, were trained as peer researchers and undertook face-to-face interviews with lay and professional stakeholders. These interviews explored capacity building, influence and effect of PPIE.
Two research staff, AH and NK, met with the peer researchers, SB and CQ, to agree the content and approach of the interviews. Both SB and CQ had developed good communication skills in their previous roles in academia and industry and as public sector governors. However, they did ask for additional training on semi-structured interview techniques. Training focused on transferable skills, the aim of the interviews and question types. The training was interactive and delivered over two workshop sessions by AH and NK.
The team agreed that the interviews should last approximately 60 minutes, with some flexibility if needed. SB and CQ were most comfortable interviewing together, alternating between asking questions and taking notes. They did not wish the interviews to be audio-recorded, as they felt this would be intrusive for themselves and the participants.
Interview questions were provided beforehand to give participants time for reflection. All interviews took place at a mutually convenient time and place. Travel expenses or transport was provided.
All lay members were invited to interview. Members of the locality group were voluntary sector staff involved in the intervention delivery, which we felt were outside of the remit. Three senior representatives of the academic team were chosen to represent the trials team and wider programme management. Interviews were undertaken over a period of nine weeks, from November 2019 to January 2020.
All interview notes were prepared by CQ and verified by SB. Participants were sent the notes so they could edit or add further thoughts. The final version was submitted to AH and NK for preliminary analysis. This was then discussed with SB and CQ so consensual meaning could be agreed. NK also undertook documentary analysis of meeting minutes, to identify issues raised by public members.
Our lay interviewers were remunerated for their time in line with the agreed rates for other public members. One interviewer declined payment. Written consent to participate and the use of data for research and dissemination purposes was received from each participant.
Analysis
A hybrid thematic analysis (Crabtree & Miller, 1992) was undertaken using the NIHR standards as an existing framework alongside inductive themes generated within the data. We adopted a realist lens in seeking to understand how PPIE impacted individuals, wider research and wider society. Data gained from documentary evidence was integrated with interview findings.
Results
Twelve lay members of the programme and three members of the research team were interviewed. The final sample comprised:
the chief investigator
the programme manager
the senior trials manager
two lay programme management group members
one lay trial steering committee member
one lay programme steering committee member
five lay intervention development group members.
Lay members were representative of our population of interest – that is, all aged over 65 years with lived experience of frailty: either themselves or in caring roles. They included five men and four women. All were ‘White British’. Two members were already involved in research; the others were new to research. Invitations to take part in the intervention development group were distributed via community networks. All interested parties were contacted by telephone before being invited to a half-day workshop. The workshop explained the research aims and used participatory methods to assess the candidates’ suitability for the role. Group members were purposively selected after the workshop.
Individuals
Lay members reported several positive outcomes for themselves. They noted their knowledge of frailty had increased and this in turn had validated, or made them reconsider, their own approach to older age.
The PROSPER programme
Although lay members did not think they had much influence, their involvement was key in several areas. By being a ‘critical friend’ and ‘asking the awkward questions’, they helped focus the outcome measures. This led to ‘quality of life’ becoming the primary outcome. Two members of the project management group were instrumental in introducing the research team to wider statutory partners. These statutory partners in turn provided excess treatment costs for the feasibility trial. Also, lay members provided an authentic lived experience which was utilised in role-play training with intervention delivery staff.
NHS research
Some aspects of PROSPER would not have been possible without lay member involvement, including the role-play training, PPIE evaluation and developing the video used to introduce PROSPER to potential recipients. In addition, most lay people felt that, having been involved in PROSPER, they would be strong advocates for NHS research in future, with some already having joined other projects.
Wider society
Participants found wider community effects harder to articulate, but some lay members felt that their experience would lead them to advocate more strongly for services which maintain independence in older age. Some also noted being involved in more community-based groups and possibly being able to transfer their knowledge (from PROSPER) into other areas of voluntary work.
Barriers and facilitators
Each PPIE member was bound by the terms of reference in non-disclosure of the intervention details. Not being able to promote the PROSPER approach was a source of frustration, particularly to those on the intervention development group. Another area of frustration was the lack of opportunity for lay members to work programme-wide. For example, the lay member on the programme steering committee had specific skills that would have been useful in the intervention development group; however, they were asked by the chair not to join both groups, as it was felt it would compromise their impartiality in critiquing the programme.
In addition, one member of the research team felt that the research process inhibited them from making best use of lay members’ input, in the sense that it would have been more useful to have most input at the proposal development stage. However, they recognised that sufficient funding was not available at that stage.
A recurrent criticism from all participants was the lack of ethnic diversity within the PPIE representation (which was also reflected in the trial participants).
On the positive side, it was felt that lay members were treated equally. It was noted that a skilled chair contributed to this, with careful selection of lay members at the outset. It was also felt that relationships were good, and the dangers of tokenism were recognised and avoided. Lastly, it was felt that remuneration for time and expense was fair, but the programme was relatively low cost and this was therefore not an overriding incentive.
Secondary data
Documentary analysis showed the voices of all lay members were evident throughout the programme to a lesser or greater extent, but this did not significantly differ from other professional member input. Lay members across the programme were able to question, critique and validate the programme’s activities, generating reflection and debate within the research team. They were also able to make suggestions. Some of the PPIE impact across the PROSPER programme is illustrated in the input and outputs shown in Table 1.
PPIE impact across the PROSPER programme (Source: Authors – a full table is available on request)
| Group | Lay member input/question | Output |
|---|---|---|
| Programme steering committee | How far ahead of the ‘drop off’ (that is, decline in quality of life and increase in health and social care cost) an intervention should be targeted | It was noted this was a critical question and that the data, while providing a good starting point, would always be open to interpretation. There was a balance to be had between too ‘fit’ (expensive intervention for little return) or too ‘frail’ (low impact and cost saving). |
| Trial steering commitee | The process for married couples participating in the programme | It was confirmed that, currently, members of the same household could be recruited if approached at the same time, but not if someone in the household has previously taken part. This needs further consideration in the definitive trial. There may be a possibility of treating couples as a mini-cluster and randomising together. |
| Programme management group | Whether the local teams would signpost to other services, particularly mental health | It was confirmed that signposting was a key element of the delivery model. |
| Whether ethnicity had been addressed in the feasibility, noting the community assistors used in City (a particular area of the Bradford site) | It was confirmed that strategies to make WP4 more inclusive were being discussed, for example by using researchers with community languages. It was also noted that use of the assistors had been discussed previously, but this was not really practicable. |
The most tangible influence by lay members was in the intervention development group. This group was responsible for reviewing and signing off on all the intervention toolkit materials, including job descriptions, patient-facing information and consent forms. The animation used to introduce the intervention to potential clients was also co-produced by group members, following the feasibility trial.
Mapping onto the NIHR standards
Both the interview findings and documentary analysis show that PROSPER has demonstrated considerable progress towards the NIHR standards for PPIE. The relative strength of current activity is mapped against the standards in Table 2.
Mapping PPIE activity onto the NIHR framework (Source: Authors – a detailed table is available on request)
| UK standard for public involvement | Strength of current activity |
|---|---|
| 1. Inclusive opportunities | ✓✓✓✓✓ |
| 2. Working together | ✓✓✓ |
| 3. Support and learning | ✓✓✓ |
| 4. Governance | ✓✓✓✓ |
| 5. Communications | ✓✓✓ |
| 6. Impact | ✓✓✓ |
Discussion
There are numerous frameworks for evaluating PPIE in research, such as Public Involvement Impact Assessment Framework Guidance for Reporting Involvement of Patients and the Public, Version 2 (PIAAF) (Collins et al., 2018) and GRIPP2 (Staniszewska et al., 2017). We used the NIHR standards as a pragmatic way of quickly identifying a baseline for our involvement activity before embarking on the definitive trial phase. Since the standards’ inception, other projects and programmes have used them to provide both formative and summative evaluations (Capobianco et al., 2023; Seddon et al., 2021).
Key findings indicated that PPIE in the PROSPER programme had both perceived and tangible benefits for individuals, the programme, NHS and wider society. The scope of involvement with the programme afforded multiple opportunities for knowledge exchange, which in turn provoked self-reflection on growing old and frailty. This type of reflection has been noted in other studies (Capobianco et al., 2023). In PROSPER, both the primary outcome measure and excess treatment cost funding were influenced by the involvement of lay members. Many of our lay members have increased their skills and felt empowered to continue to support NHS research through other projects, which are common features in other studies (Karlsson et al., 2023; Lauzon-Schnittka et al., 2022). All felt they would advocate on behalf of NHS research. In terms of wider societal benefits, participants were unsure of these, although they acknowledge the potential usefulness of transferable knowledge and skills.
We were able to strengthen our findings through documentary analysis and use of the NIHR framework. Using the NIHR standards has also helped us to identify areas for improvement. We intended to use this learning to inform the PPIE structure of WP4. The intention was to work with the lay members and work out some of the ‘how to’ in the definitive trial. However, this evaluation took place at the end of 2019, just before the effects of the Covid-19 pandemic took hold.
In the subsequent years, much of the research team’s energies have been spent in trying to keep the trial viable to the funder and sponsor. In addition, some of our public members were classed as vulnerable and advised to shield and/or did not have the resources to engage virtually. It took some time to re-engage with everyone, and not all members have re-engaged. Despite this challenge we continue to work with our public members to strengthen our activities aligned to standards 2, 3, 5 and 6.
Strengths
This work focused on the PPIE experience of older adults with lived experience of frailty, which is underrepresented in the literature. The evaluation took place in 2019. We believe that it was one of the first complex programmes to adopt the NIHR standards outside of the national pilot.
We used the NIHR standards as both a summative and a formative evaluation. The summative evaluation was at the end of our feasibility trial. The formative aspects were to be taken forward in the definitive trial.
We used an inclusive participatory approach involving trained lay researchers, which added authenticity and relevance. It also supported our co-production ethos, including the delegation of control in data collection. We provided bespoke training to our peer researchers, which helped to build capacity, support quality and ethical integrity.
We used the NIHR framework to provide rigour and structure to the evaluation. We also investigated multiple layers of impact for the individual and wider society. We collected primary data from both lay members and the research team. The combined interview and documentary analysis helped strengthen and validate our findings.
Weaknesses
The decision not to audio-record interviews presented some difficulties in contextualising the notes and subsequent analysis. However, the iterative process of participant validation and the consensus meetings did mitigate the effect of this.
All lay contributors were ‘White British’, common in PPIE (Miah et al., 2020), which fails to fully reflect the population of interest. This is acknowledged but remains a significant issue. The peer researchers were also both members of the PPIE structure and may have been influenced by their own experience. Additionally, although lay members led the interviews, the involvement of professional researchers in training and analysis may also have introduced bias into the interpretation of the findings. The small sample size and specificity to one programme limits the relevance of findings to other settings.
Future research and evaluation
The PROSPER programme, initially designed as a five-year initiative ending in 2022, was extended to 2026 due to delays caused by the Covid-19 pandemic. This extension has highlighted the challenges of sustaining meaningful PPIE over a prolonged period. While PPIE is a cornerstone of research co-production, there is limited empirical guidance on how to maintain engagement over long durations without it becoming tokenistic or burdensome, especially in a population of older adults with frailty.
Future research should explore mechanisms for sustaining and refreshing PPIE over time in large, multi-year programmes. This includes investigating models for re-engaging existing contributors, engaging new participants and ensuring continuity and institutional memory without compromising inclusivity or depth of engagement. Understanding the emotional, logistical and practical impacts of extended timelines on lay members is also essential, especially in an older population. Failure to maintain at least some of the PPIE members over the lifetime of a project detracts from the collective memory and experience. The importance of an ‘on-going collective learning experience’ has been noted by other authors (Jones et al., 2024; Karlsson et al., 2023).
PROSPER’s structure, comprising four interrelated work packages, raises important questions about how to design and support PPIE that is both meaningful at the level of individual work streams and coherent across the broader programme. Future studies could usefully examine how PPIE strategies can be tailored to the specific needs and goals of different components within a complex research programme, while also fostering a sense of shared purpose and collaboration across the whole.
Reflections
This rapid appraisal was only possible due to the enthusiasm and flexibility of the lay members and their commitment to the work. Ideally, the evaluation would have been planned to run throughout the process from start to finish and been fully funded.
The peer researchers had a great deal of autonomy. Their shared experiences with interviewees gave them greater insight and understanding of the context. This enabled them to focus questions where they felt it was most important, although the professional researchers also felt that in some instances areas of potential exploration had been missed in favour of probing others. SB and CQ also had the final say in how and when interviews were conducted.
The professional researchers trusted the peer researchers to deliver what was needed for the evaluation. SB and CQ were reassured that they could give open and honest feedback about the process and findings. This was extremely important during the analysis stage, where the professional researchers did not have first-hand experience of the interviews.
Although the lay members had transferable skills and the researchers were confident in their abilities, it soon became clear that more time than was originally anticipated would be needed for training, as is recommended by other authors (Jones et al., 2024; Karlsson et al., 2023).
The professional and peer researchers were disappointed that, due to the Covid-19 pandemic, they were unable to disseminate the findings and conclusions from this evaluation at an in-person event. Instead, the findings and conclusions were shared with all PROSPER lay members via a virtual workshop.
Authors’ contributions
All authors contributed to planning the interviews and devising interview schedules. SB and CQ conducted all interviews together and took notes. CQ wrote up notes, which were corroborated by SB and the participants. AH and NK analysed the secondary data. AH, NK and CQ co-wrote the first draft of the interview findings. AH collated the results and drafted the manuscript. All authors have agreed the final version. SB drafted the Plain English summary.
Acknowledgements
We would like to acknowledge the contribution of all the PROSPER public involvement members and their invaluable contribution to this evaluation and the whole programme.
Open data and materials availability statement
The datasets generated during and/or analysed during the current study are available from the corresponding author on reasonable request.
Declarations and conflicts of interest statement
Research ethics statement
This evaluation falls outside the remit of the NHS REC review according to the HRA guidance. However, written consent for participation and the use of data for research and dissemination purposes (including publication) was received from each participant.
Consent for publication statement
Not applicable to this article.
References
Capobianco,L; Faija,C; Cooper,B; Brown,L; McPhillips,R; Shields,G; Wells,A. (2023). A framework for implementing Patient and Public Involvement in mental health research: The PATHWAY research programme benchmarked against NIHR standards. Health Expectations 26 (2) :640–650, DOI: http://dx.doi.org/10.1111/hex.13676
Collins,M; Long,R; Page,A; Popay,J; Lobban,F. (2018). Using the Public Involvement Impact Assessment Framework to assess the impact of public involvement in a mental health research context: A reflective case study. Health Expectations 21 (6) :950–963, DOI: http://dx.doi.org/10.1111/hex.12688
Crabtree,BF; Miller,WF. (1992). A template approach to text analysis: Developing and using codebooks In: Crabtree,BF, Miller,WFWF(ed s .), Doing qualitative research: Research methods for primary care. Vol. 3 :93–109. Sage Publications.
Crowe,S; Adebajo,A; Esmael,H; Denegri,S; Martin,A; McAlister,B; Moore,B; Quinn,M; Rennard,U; Simpson,J; Wray,P; Yeeles,P. (2020). ‘All hands-on deck’: Working together to develop UK standards for public involvement in research. Research Involvement and Engagement 6 (1) :53. DOI: http://dx.doi.org/10.1186/s40900-020-00229-y
Greenhalgh,T; Hinton,L; Finlay,T; Macfarlane,A; Fahy,N; Clyde,B; Chant,A. (2019). Frameworks for supporting patient and public involvement in research: Systematic review and co-design pilot. Health Expectations 22 (4) :785–801, DOI: http://dx.doi.org/10.1111/hex.12888
Heaven,A; Bower,P; Cundill,B; Farrin,A; Foster,M; Foy,R; Hartley,S; Hawkins,R; Hulme,C; Humphrey,S; Lawton,R; Parker,C; Pendleton,N; West,R; Young,J; Clegg,A. (2020). Study protocol for a cluster randomised controlled feasibility trial evaluating personalised care planning for older people with frailty: PROSPER V2 27/11/18. Pilot and Feasibility Studies 6 (1) :56. DOI: http://dx.doi.org/10.1186/s40814-020-00598-x
INVOLVE. Going the extra mile: Improving the nation’s health and wellbeing through public involvement in research. National Institute for Health and Care Research. https://www.nihr.ac.uk/going-the-extra-mile.
Jones,C. H; Seddon,D; Algar-Skaife,K; Maddock,C; Green,S. (2024). Involving older adults and unpaid carers in the research cycle: Reflections on implementing the UK national standards for public involvement into practice. Quality in Ageing and Older Adults 25 (1) :44–55, DOI: http://dx.doi.org/10.1108/QAOA-03-2023-0019
Karlsson,AW; Kragh-Sørensen,A; Børgesen,K; Behrens,KE; Andersen,T; Langhoff Kidholm,M; Rothmann,MJ; Ketelaar,M; Janssens,A. (2023). Roles, outcomes, and enablers within research partnerships: A rapid review of the literature on patient and public involvement and engagement in health research. Research Involvement and Engagement 9 (1) :43. DOI: http://dx.doi.org/10.1186/s40900-023-00448-z
Lauzon-Schnittka,J; Audette-Chapdelaine,S; Boutin,D; Wilhelmy,C; Auger,A; Brodeur,M. (2022). The experience of patient partners in research: A qualitative systematic review and thematic synthesis. Research Involvement and Engagement 8 (1) :55. DOI: http://dx.doi.org/10.1186/s40900-022-00388-0
Mah,H; Dobson,R; Thomson,A. (2025). The importance of lived experience: A scoping review on the value of patient and public involvement in health research. Health Expectations 28 (2) e70205 DOI: http://dx.doi.org/10.1111/hex.70205
Miah,J; Parsons,S; Lovell,K; Starling,B; Leroi,I; Dawes,P. (2020). Impact of involving people with dementia and their care partners in research: A qualitative study. BMJ Open 10 (10) e039321 DOI: http://dx.doi.org/10.1136/bmjopen-2020-039321
National Institute for Health and Care Research. UK standards for public involvement, https://sites.google.com/nihr.ac.uk/pi-standards/standards.
Seddon,K; Elliott,J; Johnson,M; White,C; Watson,M; Nelson,A; Noble,S. (2021). Using the United Kingdom standards for public involvement to evaluate the impact of public involvement in a multinational clinical study. Research Involvement and Engagement 7 (1) :22. DOI: http://dx.doi.org/10.1186/s40900-021-00264-3
Staniszewska,S; Brett,J; Simera,I; Seers,L; Mockford,C; Goodlad,S; Altman,DG; Moher,D; Barber,R; Denegri,S; Entwistle,A; Littlejohns,P; Morris,C; Suleman,R; Thomas,V. (2017). GRIPP2 reporting checklists: Tools to improve reporting of patient and public involvement in research. BMJ 358 j3453 DOI: http://dx.doi.org/10.1136/bmj.j3453