In December 2021, the MOTION project, a collaboration of TIPC and EIT Climate-KIC, came to an end. During its two years of working closely with partners from three EIT Climate-KIC projects – ACT on NBS, SATURN and SuSMo – MOTION supported capacity building and monitoring, evaluation and learning (MEL) activities to enhance their transformative potential. The first year consisted of developing a “Transformative Theory of Change” for each project (Figures 1, 2 and 3). Drawing on TIPC’s module-based methodology, partners engaged in several guided interactions to connect each project’s inputs, core activities and outputs with “transformative outcomes” that help leverage their desired impacts. The Transformative Theories of Change made explicit the logic and anticipated change process of each project, facilitating reflections with partners about how to improve it, unveiling bottlenecks, contextual dependencies and transformative potentials.
Figure 1: SuSMo’s Transformative Theory of Change: four change pathways centred around stakeholder engagement
Figure 2: SATURN Transformative Theory of Change
Figure 3: ACT on NBS Transformative Theory of Change
In MOTION’s second year, we developed a monitoring, evaluation and learning (MEL) plan for each of the three projects, based on their Transformative Theory of Change (ToC). The first step consisted of discussing with partners what their MEL expectations were – for instance, learning more about a specific change-pathway within their ToC or monitoring a given Transformative Outcome – and collectively identifying which indicators could be used to that end. In SuSMo, partners chose to focus on three pathways: Stakeholder Engagement (SE), Data for Impact Evaluation (DIE), and Policy, Regulation and Procurement (PRP) and to use indicators associated to four Transformative Outcomes, namely networking (to monitor the SE pathway), circulation of knowledge and tools, learning/unlearning and changes in expectations (all three to monitor DIE and PRP). In SATURN, the partners chose to focus on two Transformation Outcomes, namely, circulation and upscaling. These were chosen based on the previously identified (with the aid of the transformative ToC) strengths of the SATURN project, as well as the areas in which partners saw a need for further improvements. In ACT on NBS, the partners opted for three pairs of Transformative Outcomes (or “Transformative Outcome pathways”): (1) networking and learning; (2) circulation and replication; and (3) upscaling and institutionalisation.
To facilitate the discussion and identification of potential indicators for SuSMo, the MOTION team prepared a scheme (Figure 4) that illustrated how the stakeholder engagement network could be monitored and the tools being developed through the other two pathways (DIE and PRP) were associated to the Transformative Outcomes.
Figure 4: Conceptual scheme to identify indicators for the selected SuSMo pathways: Stakeholder Engagement (network), and Data for Impact Evaluation and Policy, Regulation and Procurement (changing practices)
When identifying and selecting transformative change dimensions to monitor, it is important to aim for a balance between their relevance for achieving impact and feasibility of monitoring them through indicators. These characteristics often are at odds with each other. Indeed, SuSMo partners identified this trade-off, when asked which dimensions were more relevant and which were easier to monitor (Figure 5).
Figure 5: The trade-off between impact relevance and monitoring feasibility of change dimensions associated with SuSMo’s selected pathways
Given this difficulty, the MOTION team proposed to base the evaluation of learnings related to the selected pathways and Transformative Outcomes on a survey, which was to be answered by the SuSMo and SATURN partners. When designed appropriately, surveys contribute to the formative monitoring, evaluation and learning approach that transformative change projects or programmes require. In contrast to traditional surveys that often seek to characterise the status of a system at a given point in time (think, for example, of periodic socio-economic surveys used to depict a neighbourhood or city, and which often aim to deliver quantitative indicators), surveys used in the context of a formative process seek to trigger reflection and learning among respondents. This type of survey is bidirectional: it creates a dialogue between the respondent/participant and the survey taker. In a formative MEL process, surveys are therefore a tool for information or knowledge gathering, which forms the basis for reflection, learning and adaptation. In contrast, for ACT on NBS, an alternative approach was developed, consisting of the aggregation of qualitative data/information collected through set of interviews, allowing participants to dive deeper into the most relevant concepts and learnings.
In the case of the survey, its design (or “protocol”, in the technical jargon) consisted of the presentation of several statements about the three pathways (SuSMo) and the Transformative Outcomes (SuSMo & SATURN) that were key to achieving their desired impacts. After evaluating the statements through a scale ranging from “strongly disagree” to “strongly agree” (examples in Figures 6-9), respondents were prompted to justify their choices. This was the key mechanism for inducing reflection and learning. Taking the survey yet was an intermediary step in the MEL process. More important was the process of feeding back the synthesized results to respondents. For SuSMo, this was done through a workshop facilitated by the MOTION team, and in the case of SATURN, survey results were synthesized in an online video (Figures 10-11).
Figure 6-9: Examples of SuSMo’s MEL survey results which were fed back to respondents in an online workshop
Figure 10-11: : Examples of SATURN’s survey results which were synthesized in an online video distributed to respondents
Finally, the MOTION team held a concluding MEL-process workshop with SuSMo and SATURN and two rounds of final interviews with ACT on NBS partners. These activities were organized to provide project partners with a reflexive space for further reflections on the overall implementation process, achieved learnings, and to discuss ambitions beyond the project’s lifespan. At the SuSMo workshop, partners discussed plans for disseminating tools they’ve developed – for instance, guidelines for shared mobility evaluation and policy, procurement and evaluation.. The intention was to formalize this plan through a Memorandum of Understanding between SuSMo partners, which would include further testing and improvement of the tools. One barrier to achieving permanent impact that was discussed by the partners in the workshop was that the field of shared mobility was rapidly and constantly changing, so that the guidelines could become obsolete before being fully tested. Therefore, one lesson was that continuous monitoring and evaluation had to take place and become a fixed element in the process, , from tool development to deployment. The approach for monitoring and evaluation had to be flexible, depending on whether the tool was more practical (as the one for evaluating the sustainability of shared mobility) or more theoretical (as in the case of defining the principles for PRP of shared mobility). A second lesson unveiled during the workshop was that future engagements should not focus on past achievements but on the anticipation of trends to update and improve the tools. To this end, two goals were discussed by partners: (a) the establishment of a “collective memory” repository for the data and materials collected and created through SuSMo, which was to (b) form the basis for a shared mobility observatory. Regarding the establishment of a shared mobility network, SuSMo was able to bring together diverse stakeholder at the city level, but a challenge for the future would be to maintain the interest of stakeholders – for instance, by showcasing shared mobility success stories based on the SuSMo tools –to keep the network cohesive.
At the SATURN workshop, the elements for improvement on circulation gravitated around (a) knowledge and experience collection and synthesis; (b) external knowledge and experience accessibility; and (c) knowledge and experience sharing among partners and with stakeholders. Across these aspects, the need for more human resources and capacity within the project was highlighted as key for improving on them. Furthermore, more effective use of target dissemination and communication activities such as the use of social media for attracting attention and interest was considered important for improving on the selected Transformative Outcomes in the future. The elements for improving on upscaling centred around: (a) engagement of external actors; (b) valorisation of project results by external stakeholders and (c) the uptake of project results. Important aspects for improving across these elements were the active engagement of challenge owners (e.g. municipalities) from the beginning of the project with a view on co-developing outputs and results. More resources and strategic emphasis on stakeholder engagement activities, which would help to develop lasting relationships with SATURN and thereby increase the interest and demand for project outputs and results, was yet another aspect for improving on the Transformative Outcome upscaling in future projects.
During the concluding ACT on NBS interviews, partners reflected about the extent to which the outcomes proposed for the MEL phase, chosen due to their higher transformative potential based on the Transformative Theory of Change, were achieved. Results showed a good level of achievement of the four monitored outcomes, but also highlighted implementation challenges in the short term. For example, in terms of ‘capacity building and technical assistance for greater uptake of Nature Based Solutions (NBS) (Networking and Learning Pathway), ACT on NBS achieved good results, but faced two major challenges: (1) although partners gained experience that was useful to develop future business plans, it was difficult to commit resources for the co-funding of capacity building events in cities; (2) longer projects (than ACT on NBS) in continuous interaction with cities are required to increase the number of participating cities and to build lasting capacities, especially with less-experienced cities, which might require knowledge transfer and support from the more experienced ones. The same was also expressed for outcomes related to the ‘Upscaling and Institutionalisation Pathway’. To achieve institutionalisation (e.g., policies incentivising bankable Nature Based Solutions projects) a more integrated network of cities implementing Nature-Based Solutions is required, with higher capacities and know-how transferred from more advanced to less advanced cities. MOTION project results also showed that the Transformative Theory of Change constituted a very practical tool during the last stages of ACT on NBS in 2021, in the context of the whole interactive process, as expressed by (anonymous) interviewees:
“The collaboration with MOTION has been like a rescue […] not only through the ability of the Transformative Theory of Change to trace back our work with municipalities, but also through the systematisation of knowledge transfer on transformative change.”
“Very useful for honing and translating our initial ideas about what an innovation ecosystem should be doing, a bit more targeted with goals and objectives that you have to articulate around the Transformative Outcomes.”
“Thanks to the collaboration with MOTION project we [ACT on NBS partners] are in discussions on whether and how to continue collaborating together through a future partnership.”
The information gathered throughout the whole MOTION interaction will inform the co-production of an enhanced Transformative Theory of Change for ACT on NBS, that can be used to provide future initiatives with higher transformative capabilities for scaling-up urban Nature Based Solutions.
At the end of the day, MOTION’s interactions with SuSMo, SATURN and ACT on NBS resulted in a successful partnership. MOTION piloted and further developed the TIPC methodology, incorporating and applying the Transformative Outcomes conceptual framework. Conversely, the partnership lead to two key contributions from MOTION to SuSMo, SATURN and ACT on NBS: (a) the systematization of each project’s specific process (activities, outcomes, outputs and impact) in a Transformative Theory of Change, which allowed project partners to see how they fit in and contribute to the system they seek to transform, and (b) based on this Transformative ToC, partners were able to draw insights and second-order learning lessons on how to disseminate the outputs/outcomes from the projects and how to approach future (similar) projects so as to improve their transformation potential.
The MOTION Blog Series
Blog 1: Moving Applied Research Online During the Corona Crisis: The MOTION Experience
Blog 2: MOTION capacity building: How to develop a theory of change for systems transformation? Our training session at the International Sustainability Transitions Conference (IST) 2020
Blog 3: A Narrative About the Transformation to Sustainable Land Use Management: SATURN’s Portfolio of Actions
Blog 4: When Transformative Outcomes meet a Theory of Change: making policy experiments more transformative with MOTION
Blog 5: Monitoring, Evaluation and Learning for Transformative Change Projects: Lessons from MOTION’s Collaboration with SuSMo, SATURN and ACT on NBS
Blog 6: MOTION Citizen Engagement Towards Sustainable Cities
Blog 7: Outcomes and reflections of the MOTION project
