Bringing Formative Evaluation to Fiocruz – A participatory training for Brazil’s largest health research institution

Estimated Time
15 minutes
Resource Type
Blog
TIPC Region
Latin America Hub
Published
2024

Researchers from the University of Campinas (Unicamp) in Brazil held a participatory training session with key employees from Fiocruz, Brazil’s largest public health research institution, to introduce them to Formative Evaluation, using the Theory of Change canvas as a basis. In this blog post, leading researchers Dr. Janaina Pamplona da Costa from the TIP network of coaches and her colleague Dr. André Sica de Campos reflect upon their experience with adopting these tools for TIP.

A COLLABORATIVE EVALUATION PROJECT

Dr. Janaina Pamplona, Dr. André Sica de Campos and their team from UNICAMP conducted an evaluation project with Fiocruz, Brazil’s largest public health research institution. Fiocruz is the research arm of Brazil’s Ministry of Health and key component of Brazil’s health system which is comprised of three entities: the Ministry of Health, the public health care sector and Fiocruz, offering research support.

During the pandemic, Fiocruz was very involved in mitigating the impacts of COVID-19 through taking action in six key areas: Conducting core research, facilitating the technology transfer to bring the AstraZeneca vaccine to Brazil, developing, producing and delivering COVID testing kits, supporting crisis communication, producing and delivering online content aiming at the training of health professionals and assistance to vulnerable populations in Favela da Maré – a low income community adjacent to its laboratory and management head-quarters.

The collaboration between Fiocruz and the UNICAMP research team had a two-fold purpose: Evaluate Fiocruz’s response and impact in the six key areas they were working on and, since Friocruz did not have an in-house evaluation team, deliver training to leading Fiocruz coordinators, planners and managers. The training was aimed at getting key people to grasp a basic understanding of impact evaluation, start talking about evaluation and for them to become multipliers and start growing in-house evaluation capacity.

For this training, Dr. Pamplona da Costa and her team used elements of the Formative Evaluation approach as part of an ex post evaluation exercise. The team deemed the Formative Evaluation approach as suitable since they were keen to facilitate a participatory approach to evaluation which fit well with the overall ethos of the Brazilian public health system and the work culture of Fiocruz.

THE THEORY OF CHANGE CANVAS AS A GATEWAY TO FORMATIVE EVALUATION 

The team developed a short university-style training session for a group of 15 Fiocruz employees, consisting of coordinators of each key area, project managers and high-level managers and planners, reporting to Fiocruz´s President. For one stage of the training, which was spread out over two mornings and one afternoon, the Theory of Change canvas as presented in the MOTION handbook was used as a basis for organising a logic model for evaluation.

Rather than using the online canvas, Dr. Pamplona de Costa asked participants to reflect upon key Theory of Change questions using an offline spreadsheet which they could use as a consultation document during and after the training. Participants were invited to use the questions as a guide, and go back to discussions later on.

 “Innovation is often related to the Oslo Manual concept of innovation but we wanted to tell them that they can think about innovation from another perspective, such as transformative innovation or responsive innovation. It brings a different view on what innovation is about, how we think about it, how to evaluate it, how it happened over time during the pandemic.”, reflected Dr. Pamplona de Costa.

After the training, each area wrote their own report, evaluating their actions. The training and the spreadsheet in particular helped them to reflect on how to prepare this report.

OPPORTUNITIES AND LIMITATIONS OF USING THE THEORY OF CHANGE TOOL 

The spreadsheet worked nicely for people to engage with these questions, and it ultimately facilitated a reflective discussion about each element of the Theory of Change.

“The tool was very useful for them to learn about evaluation and gain a self-reflexive and transformative innovation perspective of their own actions.”, concluded André Sica de Campos

However, it was also noticeable that the questions did not resonate with all participants equally well. For example, representatives of the two more technology focussed areas (technology transfer and testing kit) did not find the guiding questions suitable. The questions seemed to appeal more to representatives working in the areas of communication, training to health agents and assistance to vulnerable populations. Hence, a better fit was apparent with the service providing areas rather than product related outputs.

In conclusion, the Unicamp team was positive about using elements of the Theory of Change tool because it made participants think about evaluation in a much broader perspective, related to a wider range of issues. This would be less so with a very straightforward evaluation approach.

ABOUT

Dr. Janaina Pamplona da Costa is Associate Professor and currently Head of Department at the Department of Science and Technology Policy (University of Campinas – UNICAMP, Brazil). She is a faculty member at the Postgraduate Programme in Science and Technology Policy (UNICAMP) where she convenes courses on Methodology, Social Studies of Science and Technology, and Geography of Innovation. She also teaches a course on Science and Technology Studies for undergraduate students. She is the current President of the Latin American Association for the Social Studies of Science and Technology – ESOCITE LA. Her research interests are networks, governance and alignment, geography of innovation and transformation in peripheral contexts.

Dr. André Sica de Campos is Associate Professor and Course Convenor in the Public Administration bachelor degree (University of Campinas – UNICAMP, Brazil). André teaches modules on Public Sector Marketing and Communication; Governance of Public Policies; and Economics of Public Sector to undergraduates. He is also responsible for post-graduate modules on Methodology and University Systems to UNICAMP´s Management and Science and Technology Policy programmes.

Citation

Pamplona da Costa, J, Sica de Campos, A, Witte, J (2024) Bringing Formative Evaluation to Fiocruz – A participatory training for Brazil’s largest health research institution. Available at https://tipresourcelab.net/resource/bringing-formative-evaluation-to-fiocruz-a-participatory-training-for-brazils-largest-health-research-institution/ 

Leave your feedback on the resource

Your email address will not be published. Required fields are marked *