Five years of quality criteria for citizen science projects in Austria: a useful instrument or simply a burden?

Introduction

Currently, we experience the rapid growth of a participatory research methodology that is often described as citizen science (Bonney, 1996; Irwin, 1995), public participation in scientific research (Shirk et al., 2012) or community science (Lin Hunter et al., 2023). Many other terms are in use for this research approach (Eitzel et al., 2017), but in Austria the term citizen science (CS) is widely used by the community and in public. In general, this methodology describes the involvement of lay people in research endeavours, often together with (but not exclusively) professional researchers. Several classifications of CS have been published over the last 10 years that describe how citizens can be involved in research projects (e.g. Haklay, 2013; Serrano Sanz et al., 2014). However, as recent international discussions have shown, an overall and exact definition of CS remains difficult (Haklay et al., 2021a, 2021b; Heigl et al., 2019b). Although there might be good reasons not to strictly define CS (Auerbach et al., 2019), there are also situations where a clear understanding of what constitutes a CS project is essential, for example, for CS associations such as the European Citizen Science Association (ECSA), for funding programmes or for platforms that list CS projects on a regional, national or international scale.

Core elements that characterise most CS projects relate to the scientific rigour, the collaboration between a project team and its participants, often referred to as citizen scientists, communication with the citizen scientists and an interested public, legal and ethical aspects that build a framework for a collaboration on eye-level within a project, and sometimes also specific open science aspects. Although not always explicitly addressed as such, these aspects can be found in several documents related to the good practice of CS (Citizen Science-AG Münster, 2018; Dörler et al., 2022; ECSA, 2015; Hughes et al., 2022; Swiss Citizen Science Principles Working Group, 2022). Most of these principles and criteria have been used to guide the field (ECSA, 2015; Swiss Citizen Science Principles Working Group, 2022) or to evaluate projects and their outcomes (Hughes et al., 2022). While there has been significant research to investigate the outcomes and impact of CS on science, policy and society, to the best of our knowledge, principles and criteria for CS themselves and their effect on projects and portals have not yet been evaluated.

Quality criteria can have multiple effects on projects in general. They can be used to compare certain aspects of different projects with each other (e.g. outreach activities, open science practices) or put an emphasis on specific aspects that are crucial for projects and initiatives (e.g. involvement of citizens in research). Furthermore, they can also influence trust that specific stakeholders such as policymakers, other researchers or potential citizen scientists put in a project. However, quality criteria can also limit creativity or hamper innovation if applied too strictly, or they can exclude certain practitioners. From this perspective it is crucial to understand the influences such criteria have in a given setting.

In 2017, the Austrian CS platform Österreich forscht (‘Austria researches’) started a collaborative process together with project coordinators, citizens and external experts to develop criteria for CS projects to be listed on the platform (Table 1). The 20 co-created criteria were published in 2018 (Heigl et al., 2018).

It is important to point out that the criteria are based on literature, experience from project coordinators and feedback collected via a public consultation. They include elements that apply to scientific work and scientific integrity in general as well as very specific criteria that relate to the participatory nature of CS. Details on the development process can be found in the work of Heigl et al. (2020).

In this paper we analyse how the criteria were perceived by project coordinators who registered on the platform, from the starting point of implementing the criteria on Österreich forscht in early 2018 to early 2023. We put a special focus on those criteria that are specific to the active involvement of citizens in research projects, using both quantitative and qualitative measures to better understand the benefits and possible challenges of the quality criteria. We wanted to study (a) whether there are any differences across projects, type of institutions and research disciplines when applying the quality criteria; (b) what are concrete and perceived benefits and challenges when applying the co-created quality criteria for CS projects; and (c) which criteria triggered reflection processes between platform organisers and project coordinators, as well as within project teams.

Material and methods

Since 2018 the registration process for projects on Österreich forscht requires compliance with the quality criteria, which have been transferred into a set of questions. The quality criteria questionnaire includes 19 open-ended questions corresponding to the 20 quality criteria and is grouped along six areas, namely scientific integrity, collaboration, open science, communication, ethics and data management. Criteria 10 and 11 have been combined into one question in the corresponding criteria questionnaire (‘Where are the results made publicly available so that they can be found, reused, comprehensible and transparent? If there are legal or ethical arguments against doing so, please explain them briefly.’).

Any person who wants to list a project on Österreich forscht is requested to fill in the criteria questionnaire (Heigl et al., 2019a). While most projects are already operating when they apply for listing, some CS initiatives already make use of the criteria questionnaire during their planning stage. For every project the answers to the criteria questionnaire are examined by the platform coordinators. If all the answers are clear and meet the criteria, the project is listed. However, in many cases the responses to the questions in the before mentioned criteria questionnaire are too vague and unclear, and the platform coordinators enter into communication with the applicant in order to get more details on the way certain criteria are understood and are being met or not met by the project. This process can lead to several feedback loops (often via email, but sometimes also in direct calls), where comments on the answers are passed to the applicant and the applicant adapts the project design to be coherent with the criteria or adds missing information to the answers in the criteria questionnaire. The communication between platform coordinators and applicants is documented in digital form either in Track Changes mode in Word files or via email. Together with the classification of projects and information about applicants (institutional background) this documentation forms the basis of our quantitative and qualitative analysis.

In total, 103 criteria questionnaires and the corresponding documented communication were analysed. For the analyses we always used the first version of the criteria questionnaires that was initially handed in by the project coordinators to have a clear understanding of how the criteria and the corresponding questions were understood. Revised criteria questionnaires from the following feedback loops were not analysed, with two exceptions: to illustrate benefits and challenges imposed by the criteria on projects, we compared project descriptions before and after the application process in regards to Criteria 8 and 17. Criterion 8 was chosen because it was the criterion with the most comments and questions; Criterion 17 was the most challenging to implement according to the interviews with project coordinators, since it required a lot of reflection and sometimes also legal and programming expertise.

The 103 criteria questionnaires came from nine different types of institutions (Table 2). We categorised the projects based on the leading institution within the project consortium.

Based on the Austrian Systematic of Research Areas 2012 (‘Österreichische Systematik der Wissenschaftszweige’) (Statistik Austria, 2024), which is based on the OECD Revision ‘Fields of Science and Technology Classification’, the criteria questionnaires came from six different research areas (Table 3).

For quantitative statistical testing we first performed a Shapiro–Wilk test for normality of the data, before we used Kruskal–Wallis tests to check for significant differences between institutional or research field background. If significant differences were found, we conducted a post-hoc Dunn’s test with Holm correction.

The qualitative analysis aimed to understand the benefits and challenges when applying the co-created quality criteria for CS projects and to identify potential requirements to adapt the implementation process or any criteria to better support the community.

For the qualitative analysis all comments from the platform coordinators that relate to unclear description of the project’s criteria compliance were analysed by two researchers using the software MAXQDA. We applied the method described by Mayring (2014, 2019), who has developed the qualitative data coding method since the early 1980s in the tradition of grounded theory, where no specific pre-defined theoretical embedding has been intended. At the centre of the analytical process is the systematic coding of text material itself. Here, we apply it complementary to the quantifying analysis described previously. In the coding process the researchers were independently assigning categories to the data material in an inductive way, exploring the material for any new insights. Thus, the categories also emerged from the data material. In addition, the coding was done deductively alongside the category system, which formed the basis of the 20 quality criteria.

Based on these quantitative and qualitative insights, 10 interviews with project coordinators were conducted by the same two researchers who did the qualitative analysis. The 10 projects were specifically selected to deepen our understanding of one coding category, ‘Delineation of citizen science’, which is explained in more detail in the results section. The semi-structured interviews were performed along a set of guiding questions:

• Can you remember the process of registering with Österreich forscht? What do you remember about it? (Additional triggering questions: How was the registration process for you and your team? Were there any issues and discussions that you can remember?)

• Has the registration process and the specific questions at registration influenced your understanding of CS or the implementation of the project?

The interviews were conducted in an open manner in order to allow for any additional aspects to be addressed. The interviews were all conducted via the online meeting software Zoom during the period November to December 2023. Each of the interviews lasted a maximum of 30 minutes. One of the researchers took on the role of the interviewer while the other researcher was taking notes. Consent was given by each of the interviewees to record the interview for documentation and analysis purposes. Projects were anonymised by assigning a unique code for each project.

By applying a mix of qualitative and quantitative approaches we were able to explore the material in an appropriate way for the purpose of this study and find out more about the benefits and challenges of co-created quality criteria and their practical usage.

Results

Out of the 103 analysed projects, 97 eventually fulfilled all criteria and were listed on Österreich forscht. All of them required some clarifications before being accepted. These clarifications were requested in the form of comments on the individual responses in the criteria questionnaire. The median number of comments for all the criteria questionnaires is five (x_med = 5), meaning that 50 per cent of the projects that applied for listing on Österreich forscht got less than five comments on their criteria questionnaires, and 50 per cent of the projects got more than five comments on their questionnaires. Only six projects did not send any follow-ups after having been contacted multiple times for further clarification of open points and unclear answers in the criteria questionnaire, and therefore could not be listed on Österreich forscht. There was no project which was rejected because it did not fulfil the criteria at the end of the application process. The Shapiro–Wilk test showed no normal distribution in the number of comments to the individual criteria questionnaires (p = 0.015).

Differences across types of institutions and research disciplines

In the quantitative analyses we looked for differences across institutions and disciplines in the number of comments or remarks for the answers in the criteria questionnaire.

No significant difference (p = 0.282) could be found between types of institutions regarding the overall number of comments to the individual criteria questionnaires (Figure 1).

Figure 1.

Boxplot of the mean number of comments per criteria questionnaires regarding institutional background of the projects

We excluded those institution types with less than five criteria questionnaires from further statistical testing due to low statistical power, which were colleges, libraries, foundations and the Austrian Academy of Sciences. Since data was not normally distributed and the sample sizes differed between the institutions, we used a Kruskal–Wallis test for the number of comments in the criteria questionnaires of universities, associations, companies, public authorities and museums, which showed no significant difference over all criteria categories (i.e. research integrity, collaboration, open science, communication and ethics). When checking the different criteria categories, no significant differences could be found for research integrity, collaboration, open science and communication. For ethics, we detected a significant difference (p = 0.004), and a post-hoc Dunn’s test showed a significantly lower number of comments (p = 0.01) in universities (x_med = 0.82) compared to museums (x_med = 2.625). However, no significant differences between museums and any other institutions could be detected.

We saw no significant differences (p = 0.904) in the overall number of comments to criteria questionnaires when looking at the main research domains of the projects (Figure 2).

Figure 2.

Boxplot of the number of comments in the individual criteria questionnaires in regard to the main research domains of the projects (AGRI = agricultural sciences, HUM = humanities, NAT = natural sciences, SOC = social sciences)

Since we did not have enough criteria questionnaires from medical/health sciences and technical research/engineering, we excluded them for further statistical testing. We again used Kruskal–Wallis test to look for significant differences between research domains in the respective criteria categories. We could not detect any significant differences in the number of comments in the respective criteria categories.

The number of comments and remarks to each criterion across all projects shows that Criterion 8 received the most comments by far (Figure 3). This criterion asks for the description of the roles and tasks within the project and where to find this information. In total, 83 comments and remarks were made to this criterion, as most project coordinators forgot to state one of the roles (i.e. the roles of the researchers or the role of the citizen scientists) or to describe where one could find this information. Taking a look at comments exchanged per group of criteria, the open science criteria group received the highest mean number of comments and remarks (n = 36), mainly due to a confusion of what is the data within a project and what is a project result.

Figure 3.

Overview of the number of comments over all projects by each criterion. The red vertical line indicates the median of the number of comments (x_med = 5). Criteria specific for CS are in italics (4, 5, 6, 8, 12, 13, 14)

We found significantly fewer comments to the project coordinators regarding the criteria specific for CS projects in total (Criteria 4–6, 8, 12–14) than for the rest of the criteria (Criteria 1–3, 7, 9–11, 15–20; p = 0.007). In the interviews, however, we learned that project coordinators could mainly remember the communication and reflections on CS-specific criteria, especially Criteria 4–8. For example, the criteria for collaboration (Criteria 4–8) triggered important reflections among some of the project teams and they realised that ‘simply collecting data [without any research aim] is not citizen science’ (Project CK).

Discussion

Our investigation of the quality criteria for CS projects on Österreich forscht and the underlying process indicates that although the process was perceived as demanding, the criteria had a positive impact on the projects. There was no evidence that the process placed projects from specific research domains or from specific institutions at a disadvantage, although we found significantly more comments on the ethics criteria in projects by museums compared to university projects. We could not, however, find any significant differences between other institutions. This difference could be an artefact due to the low number of criteria questionnaires from museums (8) compared to the high number of criteria questionnaires from universities (50). However, it is also possible that ethics in collaborating with citizens is handled differently in universities compared to museums. Whereas universities’ ethics commissions focus on the ethical evaluation of research projects, including the ethical treatment of test subjects and citizen scientists (e.g. University of Vienna Ethics Committee, 2025), museums’ ethics commissions might concentrate on provenance, restitution or on how to ethically handle human remains (ICOM, 2004). Further investigations will be needed to have a closer look at this finding.

In the preamble of the criteria, it is explicitly stated that the criteria do not aim to exclude certain projects (Heigl et al., 2018), which could be confirmed by this study. However, the process of how the criteria are applied seems to be crucial, as can also be seen in the comments by the platform coordinators to the projects and in the feedback from the interviews. The analyses show clear challenges for project coordinators in several areas of the criteria. Supporting project coordinators on how to fulfil the criteria is therefore key to helping them overcome these challenges.

An unexpected result stems from the insight that Criterion 8, which deals with transparent and clear communication of the tasks and roles within a CS project, triggered the most comments and enquiries. Clear and transparent communication is a cornerstone for many aspects of a CS project, since citizen scientists need to understand what they are doing, why they are doing it and what are the results of their engagement in a CS project (Golumbic et al., 2019; Roche et al., 2020; Rüfenacht et al., 2021). However, the information provided by the project coordinators for this criterion was often missing important parts, either on the actors and their roles in the project or where this information can be found. The high number of enquiries for Criterion 8 seems to also be due to the fact that many aspects have been packed into one single question, that is, roles and tasks of participants and the project team and where these roles and tasks are made explicit.

Also, the relatively high number of enquiries and comments to the open science criteria seems counterintuitive at first. In many cases, project coordinators could not differentiate between data and results of their project or there was a misunderstanding on the term ‘open’ (e.g. that displaying reports on a map online that cannot be downloaded is not open). CS is often considered to be open science practice par excellence (Morriello, 2021; Smith et al., 2017). However, as some studies have found out (Groom et al., 2017; Suter et al., 2023), CS projects score only averagely when it comes to open science practices. Reflecting on their review of 42 biodiversity projects, Suter et al. (2023) recommend guidelines that support CS projects on how to implement open science practices. Based on the challenges of implementing open science practices in CS projects, which were identified via the quality criteria questionnaire, Österreich forscht started a working group on open science to provide hands-on training for CS project coordinators.

An important outcome of this study is the insight that the criteria also helped project coordinators to distinguish CS from conventional forms of research or science communication. Such delineations have been discussed intensively in parts of the CS community (Auerbach et al., 2019), as some fear that too narrow definitions and interpretations of CS can curtail innovation and diversity in the field if not applied with sensitivity, and that innovative forms of CS might be excluded. For example, new forms of citizen involvement could be considered not complying with very strict and narrowly defined criteria on the requested depth of collaboration within a project. But as the results of our investigation indicate, the criteria support diversity in participation rather than hinder innovation, as they trigger reflections about the contributions of citizens to the research process and their value from being involved. Since the active participation of citizens in the research process is the cornerstone of CS (Haklay, 2013; Serrano Sanz et al., 2014; Wiggins & Crowston, 2011), these reflections also have a direct effect on the degree of involvement of the participants in a project.

We observed not only a reflection on the degree of involvement of citizens in projects, but also an increased awareness of inclusion and transparency. Both are important aspects in CS projects that have been mentioned in previous studies (Eleta et al., 2019; Ozolinčiūtė et al., 2022), and also some of the main principles of Responsible Research and Innovation (RRI). RRI stresses the importance of a continuous engagement of citizens and civil society during the innovation process to negotiate the value of its outcomes (Von Schomberg, 2013), which aligns well with CS practices.

One aspect of transparency in particular, namely informed consent, stood out in our analyses, as several projects were guided into being transparent about their use of personal and research data. To ensure informed consent, projects need to make participants aware of what kind of data are used in which way and shared with whom, so every individual can make an informed decision whether to participate or not, which is important in every research project that involves people, and of course also in CS (Quigley et al., 2021), since in many projects technological tools, often from third parties (e.g. for collecting data), are used. Researchers don’t have full control over data collected via these tools anymore, and subsequent use of the data after the project’s end may be unknown during the project (Schmietow, 2016). This also relates to clear communication about the roles and responsibilities in projects that have already been discussed earlier.

Next to transparency and informed consent, inclusivity is a much-discussed topic within the CS community (Ockenden, 2007; Pandya, 2012). By asking projects explicitly to define which demographic groups they unintentionally exclude, the criteria trigger reflection on this topic and a reflection on the engagement restrictions and how some of them might be overcome in the future.

Through the personal interviews, some recommendations to the criteria process as a whole could also be derived. Several project coordinators said that the process of adapting to the criteria was time consuming, but useful. Time constraints, especially in CS projects with a tight timeline, are often a challenge. Therefore, the process should be optimised to allow project coordinators to adapt to the criteria more quickly, without losing the reflective potential the criteria now hold. Furthermore, the criteria itself need to be re-evaluated periodically, to avoid hindering innovation in the field of CS.

Conclusion

The Austrian quality criteria for CS projects on Österreich forscht clearly had an influence on CS projects, as our investigation showed. They increased diversity of involvement methods, awareness for ethical practices and open science, and supported a more transparent communication within several projects. Concerns about them being a barrier for smaller or citizen-led projects or preventing innovation within the field had been voiced (Auerbach et al., 2019), while at the same time national platforms across Europe have been looking for a similar approach to ensure an upright and transparent registration process for CS projects (Dörler et al., 2022). With this study, which critically reflects on the implementation process of quality criteria on a national CS platform over a period of over five years, we see the appreciation for the process prevailing over the initially expressed concerns. Through a considerate support-system approach, the criteria could in fact help projects in Austria to become more transparent, inclusive and clear, without excluding specific stakeholder groups or research domains.

Authorship statement

All authors have contributed in designing the study. DD and FH did the quantitative analyses, BK and TS conducted the interviews and analysed the qualitative data. All authors contributed in writing, editing and graphic design, and approved the submitted version.

Data and supporting information

All quantitative data used during this investigation are deposited on Zenodo and assigned the following DOI: https://doi.org/10.5281/zenodo.13645724. Results of the personal interviews with project coordinators are not published due to privacy reasons.

Declarations and conflicts of interest