Evidence Cafés: Overcoming conflicting motivations and timings

Evidence levels and types: What is data, information and knowledge?

Philosophical literature has long debated concepts of knowledge and information, which has led on to the more elusive and complex concepts of wisdom and enlightenment. Expressed simply, evidence can be seen at different levels of data, information and knowledge. Each level could be viewed as a stage of depth in analysis, when increasing levels of generalized interpretation and processing are applied. For example, 14031879 is data; it is a string of digits. Apply a formatting process to the digits to produce 14-03-1879, and the data becomes information: a date expressed in UK DD-MM-YYYY format. Knowledge would emerge from web research into this date, which would identify that this was Einstein’s date of birth.

The word ‘data’ comes from the Latin ‘datum’, a fact or, more interestingly, a starting point. Data are often connected to empirical evidence. While the term goes back centuries, it is not surprising that it became more widely used with the birth and mass consumption of personal computers. Information and knowledge are often described as evidence for change within an organization.

Added to notions of evidence depth are the different sources and types of evidence, from experiential to research, which could be described along a continuum of rigour in their systematic collection and interpretation. However, it is important not to associate rigour with importance, since experiential evidence can reflect highly valuable and influential evidence in national debates. Although these concepts (data, information and knowledge) have been tied strongly to research, they are still widely used in practice frames to describe a variety of types of evidence from experiential/tacit understanding, to procedural and research insights. With the growing importance of evidence-based practice within the police domain, there has been a shift to prioritize quantitative research knowledge (Sherman and Strang, 2004; Sherman et al., 2005) over tacit and experiential knowledge. However, the evidence-based practice movement has complicated this debate further with value statements about the superiority of one methodology (randomized controlled trials) over another, regardless of the reviews of method effectiveness to answer different practice-related questions (Kitson et al., 1998). Evidence Cafés address the tension between a historical tendency to assign greater weight to quantitative research evidence in preference to tacit and experiential evidence by enabling participants to connect up these different notions of evidence.

Facilitating the equitable exchange of evidence

Evidence Cafés provide a means for practitioners and researchers to connect up research evidence with practice-based evidence. The Evidence Café approach to knowledge exchange is built on the principles from the worldwide Café Scientifique movement (Grand, 2015). Evidence Cafés draw on these principles, placing at their core concepts of two-way knowledge exchange rather than one-way knowledge transfer.

In the next section, we describe the methods and methodology, and then present three case studies that show the impact of motivation and timing on the flow of evidence, and the effectiveness of knowledge exchange through Evidence Café methods.

Data collection methods

We collected data at each Evidence Café using a variety of methods:

• audio recordings – transcribed and anonymized

• discussion objects – photographed and/or digitized depending on the discussion object

• flip chart/magic whiteboard notes – photographed and digitized

• online survey – to capture views and perspectives before and after the Evidence Café

• participant notes – either handwritten or using Livescribe recording pen

• Post-it note feedback at the end – reflect on: three things you liked, three things you thought could be improved, three things you would like more of

• still photographs

• Twitter – #WeCops debate run two hours prior to an Evidence Café to identify evidence-based practice themes to feed into the subsequent Evidence Café

• video:

  • camcorder, either set up in a corner of the room on a tripod, or handheld

  • iPad, handheld

  • Flip Video, handheld.

The choice of data recording method varied between Evidence Cafés depending on the participants and the choice of topic.

Developing a discussion object: The evidence typology

Each discussion object was tailored to match the topic of the Evidence Café. For example, in Evidence Café 10, we developed an evidence typology as a discussion object to support participants to identify issues around translating the masses of policing data into useful information that could lead to knowledge and insights into how to improve police practice, as well as integrating different types of evidence from research with policing processes and experience evidence. The evidence typology also allowed participants to classify levels of evidence by increasing depth of analysis:

• data – raw data without meaning, for example, numbers, words

• information – data with attached meaning for ‘who, what, where’

• knowledge – information with stories, insight, know-how for ‘why and how’.

The evidence typology then enabled participants to classify evidence on a continuum of rigour (of equal importance) for types of evidence:

• experiential – for example, stories

• interpretive – for example, expert reports, analytics

• procedural – for example, policies and processes

• research/empirical.

For each evidence type, there is a different level of analysis that has been achieved. Types and levels of analysis link together through the typology (see Figures 1a and 1b).

Figure 1:

(a) Evidence typology types and levels, (b) Evidence typology types and levels

The evidence typology brought together different notions of evidence, enabling participants to find connections between them without allowing any one particular type of evidence to dominate. Participants engaged deeply with the typology; the EBC from Evidence Café 10, who was a high-ranking officer in charge of teams of data analysts, reported that his staff subsequently integrated the evidence typology into their working practices:

There was some really good conversation after the session, and I can see some copies of the typology about the office! (EBC, Evidence Café 10)

During Evidence Café 10, one data analyst commented in the group discussion that they were collecting large amounts of a particular category of raw data, and turning it into reports that were then submitted and filed with no further impact. They found this very demotivating. The EBC, who was the high-level boss of the team, was shocked, having some months previously issued a directive that this unproductive activity should cease. Informed by the Evidence Café, the EBC determined to reissue the directive and take steps to ensure that it was filtered to all levels in the organization. In this example, the meaning-making activity around the evidence typology highlighted an issue that data were being processed into information, but that information was not being interpreted into useful knowledge. The informal and equitable methods of the Evidence Café, where all voices carry equal weight, enabled meaningful communication between a team member of low status and the officer in overall charge.

In later feedback, the EBC from Evidence Café 10 described how that force planned to use the evidence typology to support changing work practices:

In terms of using the material presented during the Café, I will certainly be using it as part of presentational and training material for the development of our ‘organisational learning’ function. This is a development of our current performance/corporate services function to more effectively harness learning in a number of formats be that internal, academic or policy. I see the typology as particularly useful in being able to demonstrate the transition from an older way of working (late-‘90s performance targets environment) into a more holistic arena which triangulates qualitative and quantitative data to provide a compelling narrative in context. (EBC, Evidence Café 10)

In view of the effective way that the evidence typology enabled police participants to classify evidence as different types of data, information and knowledge, it was redeployed as the discussion object in two other Evidence Cafés on different topics, hosted by other police forces. The Evidence Café themes were: (1) how to identify and make use of different forms of data collected around antisocial behaviour; and (2) how to evaluate the success of initiatives to stop young high-harm offenders from reoffending. The evidence typology was effective in both these Evidence Cafés.

The power of the evidence typology as discussion object in enabling participants to unpick concepts of data, information and knowledge in a range of policing contexts prompted us to use it as an analysis framework within which to analyse the evidence from the case studies. We combined this with a grounded thematic analysis of the data we had collected from the Evidence Cafés, and we used the evidence typology to classify the flows of evidence from data to information and knowledge that we identified. The analysis then allowed us to map out the level of knowledge transfer and knowledge exchange between research and practice, as well as between practice and research. We also collected evidence of changed understanding, and of impact on practice and processes, for both the police and academia. This analysis identified a key challenge around conflicting motivations and timings, and their impact on whether or not two-way knowledge exchange occurred.

Case Study 1: Evidence Café 12 – Minimal evidence flow

Evidence Café 12 was the second Café run at Police Force 1. The first Café held with this force (Evidence Café 1) had delivered high levels of two-way flow of evidence, leading to equitable knowledge exchange. In contrast, Evidence Café 12 did not deliver knowledge exchange. Evidence Café 12 was facilitated by an RC and EBC on the topic of social media as an online tool for evidence capture. The EBC was a senior officer with power to effect change, and the participants were technical and specialized experts in combating cybercrime and digital forensics.

Case Study 1: Analysis

Despite engaging interactions with the application, the police participants identified limitations of an online app for court evidence capture, and suggested useful redevelopments that would support police investigations, such as accessing suspects’ private social media details and generating screenshots for police reports. The academic team responded that, as required by their research project, a ‘proof of concept’ (due to be trialled in another police force) had been developed and could not be changed. Although knowledge from practice was offered, the academics were not motivated to change their research, which they regarded as already successful. They did not step outside the academic frame or incorporate this knowledge into their research.

The evaluation of this Evidence Café identified that the academic knowledge was siloed and resistant to change from practice influence, thus not allowing for two-way exchange of understanding. This resulted in the shared research evidence lacking relevance to practice needs and led to a lack of knowledge exchange and a lack of impact on practice.

During Evidence Café 12, it became apparent that the research, as designed and implemented, was not seen as relevant by the practitioners as it did not address their practical issues, so although the EBC had the power to effect change, there was no point of connection between the research and practice.

Figure 3 uses the evidence typology to show how the experiential feedback on the tool offered by participants (Column 1) did not have an impact on research (Column 4). It uses red arrows to illustrate the flow of experiential evidence from practice to academia. The research gaps identified by participants were taken by academics to feed into ideas for future research proposals, but unless authentic knowledge exchange was built into the research, the impact on practice of these future research projects may be limited.

Figure 3:

Evidence flow in Case Study 1

In Case Study 1, a great deal of communication took place between practice and academia, focused on the discussion object of the app running on smartphones or iPads, but this did not result in meaning making or changed understandings. Case Study 1 is an example of an Evidence Café where knowledge exchange did not occur. It has been included to demonstrate how our understanding of the factors that contribute to an effective Evidence Café were built not only on our successful Cafés, but also on the lessons learnt from Cafés that were less successful. Evidence flow was limited, and what flow there was had little impact, as neither participants nor academics left with changed or revised understanding.

Case Study 2: Evidence Café 3 – Knowledge transfer from practice to research

Evidence Café 3 was the first Café run at Police Force 3, with the process being well received by the participants. It was facilitated by an RC and EBC alongside two researching academics (including the principal investigator). The research topic for Case Study 2 was a social technology platform to enable members of the community to gather data that might be of help to the police in policing their community. The platform would also allow the police to gather data in areas of particular interest to them, such as recent criminal incidents, to which members of the community could contribute evidence. The platform was open and public.

Case Study 2: Analysis

The police role in the whole-group discussion resulted in their identification of ethical issues (for example, identifiable actions) in public use of social media for police purposes, as well as a reluctance to change current social media practices.

The timing of this Evidence Café, run at the start of the research project, meant that academics expected participants to contribute ideas for use on the platform. They did not anticipate the barriers that were highlighted by the police that made the platform unsuitable for the police. However, the academics’ level of power within the project (one being the principal investigator), their motivation to engage the police and the timing in the research cycle resulted in this feedback being used to modify the research design. The platform went on to be developed with more community focus. However, while the police participants and EBC were enthusiastic to follow up the redesigned research project, the top-level leadership of the police force did not support further involvement. The EBC in this instance was a front-line officer with less organizational power than the EBC in Case Study 1. Figure 4 illustrates the flow of evidence from practice to research, resulting in an increase in research knowledge as academics collected and acted upon the police feedback on their research design. However, there was no matching flow of research evidence to practice. There was potential for flow of research evidence through further involvement with the redesigned project, but the EBC lacked the power to take this initiative forward.

Figure 4:

Evidence flow in Case Study 2

Case Study 3: Evidence Café 6 – Equitable knowledge exchange

Evidence Café 6 was the first Café run at Police Force 6. The process was very well received by the police force, and subsequent Evidence Cafés were requested to be held beyond the analysis period reported in this paper. Case Study 3 demonstrates two-way knowledge exchange facilitated by an RC in collaboration with three EBCs from the host force and an academic expert (principal investigator) focusing on the topic of demand management. A police officer – referred to as a senior practitioner fellow (SPF), who was seconded from operational duties to work on a research project with the support of an academic – and a PhD student also attended the Evidence Café in order to collect data for their research. The host force EBC had requested that the SPF give a short presentation to share insights on the experience of conducting police research with the support of academia.

The academic was a principal investigator with considerable experience of researching demand in the medical sector. He was motivated by a wish to extend his research into the domain of policing. The EBC was a member of police staff in a management role, tasked with improving how the force coped with demand through a series of cross-disciplinary initiatives. Police participants were front-line officers and staff.

Case Study 3: The process

The Evidence Café began with an academic presentation of research into demand categorization, presenting different ways of classifying the types of demand faced by police forces. This led into the group meaning-making activity around the discussion object, a Home Office table containing five categories of demand, by which participants were invited to classify the types of demand they experienced during their work. This was followed by whole-group discussion facilitated by the academic.

While this was taking place, the SPF collected data relevant to his own policing priorities research in an adjoining room using a Q-board methodology (Stephenson, 1953). Figure 5 shows the Q-board used.

Figure 5:

Q-board as discussion object

The Q-board methodology was used to reflect on policing with limited resources, with participants given 62 tiles categorizing demands on the police (for example, parking offences, child abuse, burglary) that had been previously identified, and prioritizing them on the Q-board as more or less valuable. After the whole-group feedback session, the SPF gave a research presentation that generated great interest as it was presented from a strong police practice perspective. Seeing the level of engagement, the academic and RC opportunistically decided to redeploy the Q-board as a second discussion object. They turned it into a collaborative, reflective activity, with two groups of participants, each with a Q-board and a set of demand tiles. Each group had to come to a collective decision as to where to place each tile, with various participants voicing their views. The engaging nature of the Q-board as a discussion object helped break down barriers of status and role, and the discussions were more revealing than the ultimate placement of the tiles. For example, when handling the ‘parking offences’ tile, most participants would put this down as the least valuable. However, one participant commented, ‘But if you were to ask the Police and Crime Commissioner, he would probably place this as much more valuable, given the pressure from residents’. This was a reference to a tension between the motivation of a Police and Crime Commissioner, who is elected by members of the public, and the police who have to carry out the job, and it is relevant to the choices made about dealing with demand.

The unplanned discussion object was so engaging that participants stayed for an additional 40 minutes over the original duration of the Evidence Café in order to complete the activity. By being responsive to the participants in the room, the academic and RC were able to develop what was originally a research data collection tool into an effective discussion object that was subsequently redeployed in future Evidence Cafés on demand management with another four police forces across the UK.

Case Study 3: Analysis

Both the presentation and the interactive activities focused upon the discussion objects triggered engagement and discussion between participants about the basis for choices made in policing practice and public perceptions. The key output was not the final classification of demand types, but the meaning making that took place supporting the flow of evidence from research into practice. The participants gained a new perspective on their demand, and on successful techniques to manage policing demand. In addition, the interactive activities initiated a two-way flow of evidence between practice and research as academics learnt more about the relevance of the research for policing practice.

The flow of evidence between practice and academia, and the dialogic meaning-making processes, are represented in Figure 6. The red lines show the flow from practice to research (data to information and knowledge) through research data collection feeding into published findings (that is, facilitated Q-board research data collection in a side room – represented by the grey box). The blue lines represent data sharing and dialogic meaning making between police participants, with academics engaging with and supporting the process (purple boxes).

Figure 6:

Evidence flow in Case Study 3

Case Study 3 demonstrates two-way evidence flow leading to effective knowledge exchange with impacts on both research and practice. The academic, EBC and RC (as well as the senior practitioner fellow) all emerged with changed understandings. Data collection informed research practice, with a measurable impact on academia. Police practitioners also learnt new demand classification techniques leading to new police practices at an organizational level. The police force subsequently set up multidisciplinary teams to identify different types of force demand and implement techniques to mitigate undesirable and unnecessary demand, with Evidence Café methods kicking off these initiatives.

This particular Café was the first on demand management with the police, and it demonstrates the importance of a flexible approach to facilitating evidence flow between research and practice.

Enablers: Champions and discussion objects

The role of the practice-based EBCs and academic-based RCs was key to the success of the knowledge exchange. Not only did they ensure co-design of the event and facilitate equitable group work around discussion objects, but they also highlighted the relevance of research to police practice and reshaped research objectives to increase relevance. All Evidence Cafés were jointly introduced by the EBC and RC, and in some, the EBC also gave a presentation about their experiences crossing boundaries between research and practice. This supported meaning making by contextualizing research evidence in practice, and by highlighting connections between the world of policing and the world of academic research. In this sense, the EBCs and RCs acted as a bridge or ‘boundary creature’ (Adams et al., 2013) between the domains of policing and academia.

Research has highlighted that it is not only people that can cross boundaries; objects also facilitate crossing the research and practice boundaries. Boundary objects have been identified as holding an important role as a connection point and a way of mediating evidence, allowing people from differing domains to develop new and shared perspectives. These objects can become embedded in contextual norms, and jargon risks acting as a barrier to traversing contexts. However, when effectively designed, they can support developing new or negotiated joint understanding between people from different worlds who undertake collaborative tasks related to evidence boundary objects (Carlile, 2002; Sapsed and Salter, 2004; Scarbrough et al., 2015; Star, 2010; Star and Griesemer, 1989).

In Evidence Cafés, we developed and reviewed a specific type of boundary object in the form of discussion objects. Boundary objects can facilitate one-way exchange across boundaries asynchronously as well as synchronously. Discussion objects specifically enable equitable co-creation in synchronous evidence exchange. The discussion objects helped practitioners to visualize academic research findings in terms that were relevant to their practice, acting as a bridge between academia and practice across which academics and practitioners could meet on equal terms to realize a shared understanding. Once research was contextualized, practitioners were better able to offer their perspectives and evidence. Thus, discussion objects acted both as a way of making research more accessible, and as a channel through which practice-informed views and evidence from practitioners could contribute to academic research.

The important characteristic of an Evidence Café discussion object, facilitated by the champions, is that it promotes meaningful two-way dialogue among the participants and between the participants and the academic(s). This engages participants who are interested in the topic, as well as breaking down barriers of rank and status, and enabling participants, essentially, to forget the constraints they usually work under and contribute and listen freely and constructively.

Challenges: Competing objectives in motivation and timing

Motivation is a key cause of tension and competing objectives between practice-based and academic research, with practice-based research placing more weight on notions of evidence validity, and academic research assigning importance to evidence rigour and reflexive processes. While both practice-based and academic research are valid, it is often perceived, in both domains, that rigorous academic research evidence is superior and should therefore have more power. To be effective, an Evidence Café needs to give equal weight to both evidence bases. In the weeks running up to each Café, it was important for the RC and EBC to address this tension by matching the objectives of the host organization to those of the academics. Matching objectives does not mean having the same objectives, but rather having objectives that complement each other.

The level of power or authority of champions, academics and participants within both types of organization was identified as important in facilitating motivation to overcome the challenges of competing objectives. For example, within the police, the status of the EBC was a key factor influencing the extent to which evidence and meaning making from the Evidence Café could be successfully integrated with practice. In Case Study 3, the police EBC was a member of police staff with the remit and authority to set up initiatives to address issues of excessive demand, and therefore the impact on practice from Case Study 3 was significant. In Case Study 2, the EBC was a police constable in a force where the leadership team were resetting priorities away from evidence-based practice, thus the impact on practice was limited. The EBC in Case Study 1 was an officer of rank with the authority to drive forward evidence-based practice initiatives, and did so by reusing Evidence Café methods to run Practitioner Cafés within the force. EBCs are key in ensuring sustainable impact on practice from Evidence Cafés, but this cannot be achieved without buy-in and support from their organization and leadership team.

From an academic perspective, the challenges of power on motivations were more subtle but still evident. In Case Study 1, the research champions were academic researchers working on a project overseen and coordinated by a principal investigator who did not attend the Evidence Café. The power they had to initiate technical changes without the principal investigator’s oversight was limited. However, in Case Studies 2 and 3, the academics were both leading and researching on the project, allowing for deeper insights to be taken from and adapted into the research project itself.

Competing objectives of what is considered important or irrelevant evidence can affect motivations in applying the evidence to practice. Evidence Cafés highlight that evidence from both research and practice is of value, but it can be challenging for academia to integrate the two types of evidence without giving more weight to evidence obtained through recognized academic research in preference to rich experiential or procedural evidence developed and shared by practitioners, as illustrated in Case Study 1, where the academics were motivated by disseminating their research. This again highlights the importance of contrasting and competing Evidence Café objectives between the practitioners and academics. For example, within research, if academics have an objective it can blind them to adapting to feedback from practice evidence. In reality, they have the power to change, but the reluctance is more about their motivation to change. Effective research is more about appropriate objectives than about power. In contrast, within practice, the key lies more with power than objectives, as illustrated in Case Study 2, where the EBC lacked the rank and power to effect change. The findings from these Evidence Cafés have identified that with practitioners it is about getting the right people in the room – those with the authority to initiate and carry through change. Within the practical application of any research, it is about changing mindsets, and thus the relevance of the research to practice.

Finally, these Evidence Café findings identified the importance of timing for the Evidence Cafés. The challenge here came in fitting with the research process to enable academics to benefit from practitioner insights. There was also a timing challenge of enabling practitioners to develop a deeper understanding of how current practices may be adapted to align with, and benefit from, new research innovations. Balancing these challenges provides further competing motivations that an EBC and RC need to negotiate and find complementary timings for Evidence Cafés.

Academics are accustomed to presenting their completed research to an audience in a formal setting, at the end of a project. Some academics initially viewed Evidence Cafés as a dissemination activity, to be plugged in at the end of the research to tick a ‘public engagement’ check-box. For example, Case Study 1 highlighted that timing an Evidence Café late in the research cycle is unproductive as it necessarily frames the Café as a dissemination activity. There was no time or space in the Evidence Café process to integrate the evidence flowing from practice into valuable changes for the research project. Thus, although there was potential for knowledge exchange from which both research and practice could benefit, that potential was not realized. This highlights the importance of academics recognizing how an Evidence Café, as an equitable knowledge exchange process, should take place early in the research process. An example of an Evidence Café set at the start of the research design process was Case Study 2, and as such the academics were able to benefit from evidence flowing from practice into research. However, there should also be equity in the timings to benefit practice and the appropriate application of research. Case Study 2 did not effectively balance with competing practice timings to fit with a need for change in the practice context, thus allowing impact upon practice as well as research. While effective balancing of competing motivations in Evidence Café timings are important, this does not mean that all Evidence Cafés placed at the end of a research project will become dissemination activities. Our research highlighted that this very much depends upon the role of the EBCs, RCs and academics, the nature of the research, and how adaptable to changes all of these are. For example, Case Study 3 is an exemplar for many other successful Evidence Cafés that managed to balance competing objectives and timings for both research and practice. This Evidence Café was the first of a series of six Cafés about demand held at different points in the research cycle, within different police forces meeting their own specific practice needs. Part of the research had been fully developed (that is, demand management categorizations) and part was at the start of the research process. All the EBCs and RCs were open to adaptive change to benefit all participants and impact both on research and practice. As such, Case Study 3 is a good example of many other Evidence Cafés that have helped frame appropriate research.