Reflexive practice in Transformative Innovation Policy: Six lessons for action researchers

This blog is an edited extract from an article exploring TIPC experiences of implementing formative evaluation, highlighting the skills, attitudes and values required from evaluators to strengthen trust, participatory engagement and a learning mindset in experiments. 

Between 2019 and 2021, TIPC worked with partners to develop a formative approach to the evaluation of Transformative Innovation Policies at the Swedish Innovation Agency (Vinnova) and in the Adaptive Cities Through Integrated Nature-Based Solutions (ACT on NBS).

This involved assessing the changes associated with or leading to socio- technical transitions (Molas-Gallart et al., 2021), based on an understanding of how transformation of socio-technical systems happens according to a specific type of sustainability transition theory known as the Multi-Level Perspective (Rip and Kemp, 1998; Geels, 2002; Geels and Schot, 2007). In TIPC’s formative evaluation approach, the Multi-Level Perspective is used as a baseline, and the generic theory of change for the development of specific theories of change tailored to individual programmes and interventions.

Evaluation in support of social learning

A key goal of evaluation is to inform and provide developmental feedback for system transformations (Ofir and Rugge, 2021), and to support social learning built on such feedback. Indeed, social learning is critical for changing socio-technical systems (Schot and Steinmueller, 2018); it occurs when a heterogeneous set of actors share their knowledge and assumptions in an interactive process aimed at creating new knowledge, generating trust among the actors and leading to joint action (Pahl-Wostl, 2006).

To support social learning, evaluation practice takes on specific characteristics and requires values and attitudes that are different from those required for other kinds of evaluative practice.

The following lessons emerged from the Vinnova and ACT on NBS projects on the role of reflexive practice, and values and attitudes required of action researchers for formative evaluation.

Lesson 1:
Take time to build trust

We have found that clearly expressing each other’s assumptions and expectations and the intervention at the start of the process is a factor of success.
(Vinnova team member)

In Vinnova’s engagement, we did not spend enough time at the beginning of the engagement to get to know each other and to understand each other’s (TIPC’s and Vinnova’s) perspectives and methodologies. This happened because the TIPC team felt pressed by the need to deliver tangible outputs, so we focused on defining a theory of change and did not pay enough attention to clarifying mutual expectations.

As a result, the amount of time needed to work together, the mission-oriented methodology used by the Vinnova team and the roles of each of the TIPC and Vinnova members were not initially clear. This led to misunderstandings and frustrations, but, progressively, through the interactions and conversations, the group gained a deeper understanding of each other’s expected contributions, context and similarities and differences in their understanding of how policy could support transformation.

Lesson 2:
Surface diverse goals and expectations

The following quote from a Vinnova team member expresses the relevance of dedicating time to share and agree on goals and expectations.

When I think of setting up a similar project in the future or in another context, I think one of my main takeaways is that it would be necessary to spend more time talking at the beginning of the project. Within the group, you need to allow for everyone to clearly speak out about expectations and assumptions. What are the assumptions about each other, and what do we as researchers and practitioners, as well as individuals, expect when it comes to working with each other?

In Act on NBS’s engagement, we followed a different approach: we invested two months (February and March 2020) in getting to know each other’s context and expectations on the engagement. These previous interactions were helpful in designing the workshops, in which we insisted on the importance of aligning mutual expectations.

The virtual challenge provoked by COVID-19 added a layer of complexity to this. We engaged with experimental methodologies, built up a team and did it all through online tools. Having a safe and respectful space to talk about the process has been fundamental for us to enable co-creating processes:

Having the opportunity, in a safe space, to have a serious conversation around what we are doing while we are doing it and to build up trust. To come to that point has taken some time, with some excursions outside of the map that we intended to be within, but I think I have learned a lot.
(Vinnova team member)

Lesson 3:
Engage critically with theoretical premises and assumptions

Our formative evaluation approach has a strong theoretical background based on transitions theory. We have realised how this distinctive feature makes our approach meaningful for participants, allowing them to reflect on their intervention and, eventually, to reorient it.

The Vinnova case required critical examination of these theoretical inputs and assumptions, and engagement with other perspectives. When the TIPC team started its engagement, Vinnova was designing its intervention following an experimental strategic-design approach (Hill, 2012). According to this approach, a way of delivering ambitious change is a continuous design of a prototype that can be tested and refined over time (Young, 2010), without pre-defining specific desired outcomes (Gaziulusoy and Erdogan Oztekin, 2019). Instead, we started our engagement attempting to build a theory of change and then deriving the transformative outcomes from it.

This resulted in a clash between the two different perspectives, which brought tensions and difficulties. As one of Vinnova’s participants recalled:

From the broader perspective of the mission-oriented work and its design-oriented approach, it has turned out to be complex and somewhat “unfitting” to map up a ToC in the “classical way” [input-output-activities-actors-outcomes] at this point in time. The mission-oriented work is not based on a pre-defined map; rather, the work is to build up along the way, with an overall direction [the missions] as the guiding principle.
(Vinnova participant)

To address this initial differences, we focused on the objectives and practices laid out in the previous two sections: developing trust and mutual understanding through the generation of safe spaces, and the use of a variety of techniques in a flexible way. We also abandoned the idea of mapping up a theory of change and focused, instead, on identifying the four most relevant outcomes that the food policy team wanted to achieve.

Lesson 4:
Question prevailing assumptions

As we have described, the two engagements became sources of learning both for the project participants and ourselves, but both processes were challenging. A Vinnova team member described in some detail the problems we faced:

One problematic, and commonly occurring, aspect of this is the way we as practitioners tend to relate to researchers as consultants […] researchers are viewed as experts who are supposed to help the practitioners to solve specific issues. Practitioners, who tend to be busy and focused on practical solutions, thus enter the exchange with a “what’s-in-it-for-us?” mentality, and expect the researchers to deliver tools and advice in a similar way as consultants would. In the case of TIPC, I think this generally occurring tendency was a bit further enhanced by the ambitious (and indeed partly selling) tone and professional format of communications material of the consortium. Actually, in the same way, the high expectations on Vinnova (in general and within TIPC) might partly derive from the agency’s capacity and strategy regarding communications and self-proclaimed profile. Another, quite contradictory, aspect of the imbalance in expectations, lies in the simultaneously existing assumption that researchers do not know the reality of the practitioners as well as the practitioners do themselves.

In this way, the practitioners questioned our role as evaluators and brought us to a “discomfort zone”. To deal with this situation, we had to acquire some of the values and attitudes that Patton (2020: 123) highlights as relevant for transforming the field of evaluation.

The experimental ethos and the reflexive practice we have developed have been key to questioning our assumptions and aligning expectations. For the TIPC team, one main assumption we questioned was the need for a “classical” theory of change to conduct the formative evaluation. As we have described before, in the case of Vinnova, we abandoned this idea and focused instead on the identification and monitoring of desired outcomes.

Lesson 5:
Embrace empathy, humility and openness to new ways of thinking

These attitudes go hand in hand with the evaluation practices supporting social learning. Building trust and mutual understanding, as well as recognising the essential role of intermediaries, requires putting aside the “expert” ego and engaging with the practitioners with a high degree of empathy.

It also requires an openness to new opportunities, ideas and ways of thinking, and to the valuing of multiple perspectives.

In the two cases presented in this paper we used a combination (“bricolage”) of techniques. In the Vinnova case we combined our approach with the design thinking approach that was being used by the practitioners looking for and finding common points, and in the ACT on NBS case we tried to integrate the participants’ perspectives and ways of thinking with our Multi-Level Perspective and the use of Transformative Outcomes.

Lesson 6:
Adapt to ambiguity and emergence

The COVID-19 pandemic forced us to learn to cope with ambiguity and unfamiliar situations. We were rapidly required to work remotely, not only with our partners but also within the TIPC team. This was a totally “unfamiliar” situation.

Another “discomfort” came from the characteristics of the cases we addressed. We are mainly trained as researchers rather than facilitators. Yet, our role as evaluators in this approach required the application of facilitator skills: many of our tasks were those of a facilitator.

Also, we were unfamiliar with the two contexts within which we worked and the communities we engaged with. We had never met any of our partners, nor had we worked with their institutions. Taken together, we were often outside our comfort zone, and yet the combination of flexibility, openness and humility and the individual and collective reflection we engaged in after each interaction helped us adapt and work comfortably.

Due to the experimental character of the engagements, we had to be adaptive, agile and cognitively nimble. The bricolage of methods and the flexibility in using the theory of change are examples of these attitudes.

Challenges like the difficulties in applying our theory-based approach required adaptations that, while preserving the theoretical foundations of our approach, led to diverse ways of setting up the evaluation activities. We combined our approach with others and played facilitating and mediating roles to make our interactions more dialogical without losing theoretical depth.

Conclusions

Throughout the two engagements, we were challenged in our role as evaluators.

We had to develop different values and attitudes from those common in evaluation practice. We questioned prevailing assumptions, and exercised empathy and humility without difficulty, but opening to new opportunities, ideas and ways of thinking and valuing multiple perspectives became more difficult because of our reliance on a specific and rather complex theoretical framework.

We also found it difficult to be comfortable with ambiguity and unfamiliar situations, and with adaptability and the ability to be cognitively nimble. Some discomfort was present in the two engagements, but we learned how to deal with it and become more relaxed in situations where disagreements and differences in practices and culture emerge.

Adaptability has been a must throughout the engagement, but, given our theory-led approach, there were limits to the extent to which we could become “cognitively nimble”. We did, however, combine different theoretical foundations when applying our approach to adapt it to the needs and concepts held by the participating practitioners.

Although Patton (2021) does not mention reflexivity as a key attitude, it became very relevant in our context. A reflexive practice was crucial in both engagements, helping us overcome challenges and enabling social learning.