This short introduction by Alejandra Boni and Jordi Molas-Gallart outlines the need for a formative approach to evaluation, and presents five principles that underpin this approach.
Formative evaluation for transformative outcomes
The Transformative Innovation Policy Consortium has set about the task of developing an approach to the evaluation of transformative policies and experiments consistent with its policy principles. This methodology, initiated by TIPC partner, INGENIO, encompasses a ‘formative evaluation’ theory of change for transformative outcomes.
Why do we need a formative approach?
Firstly, TIP projects are ambitious and operate within complex environments and systems. It is not only a matter of developing a technology, or deploying it in a specific context. Transformation requires organisational, institutional and cultural changes. Many of these changes take place beyond the location and period of the initial project and intervention. If we wait to assess a policy until its potential impacts are starting to emerge, it may be impossible to attribute such impacts to the policies under assessment. Therefore, the lessons derived from the analysis will be obtained too late to be used in an effective learning process. We need to address attribution and timing problems
Secondly, evaluation practice is contingent on the nature and scope of the activity to be assessed, the resources and the evaluation capability available, the point in the policy process at which evaluation is introduced, and the governance structure within which the activity under assessment is conducted. TIPC activities are conducted in very different contexts, and involve a wide variety of international experiments across multiple geographical hubs.
Although we cannot develop a single set of tools for general applicability, TIPC has defined a conceptual framework and a generic approach that can tackle attribution and timing issues that can be adapted to local circumstances.
Principles of formative evaluation
The following five principles underpin this formative evaluation approach:
(1) It is formative and inclusive. Formative evaluation refers to a long-established set of principles and practices that aim to improve policy design and implementation in close collaboration with all agents participating or directly benefitting from these tasks.
(2) It focuses on the identification and monitoring of transformative outcomes. The Transformative Innovation Policy Consortium has developed a set of twelve ‘transformative outcomes’ that are traceable and attributable to the policy interventions under consideration, and that can be used as indicators of whether and how the intervention is triggering processes that are likely to be transformative in nature.
(3) Learning objectives are at the heart of the approach. Monitoring whether and how such outcomes are emerging from the interventions becomes a step in a process of learning that will steer policy development and implementation towards the transformative goals.
(4) The approach uses flexible theories of change. The theory of change within formative evaluation identifies the transformative outcomes that the intervention is seeking and the ways in which the interventions are expected to lead to such outcomes. As the intervention evolves, the lessons learnt from the evaluation process can lead to the theory of change being revisited and adapted.
(5) Finally, it is based on a nested, multi-level approach. Several, interrelated theories of change can be used to address different levels of policy activity: projects that operate at niche level, programmes involving a variety of transformative projects, or policy mixes deploying tools across different policy sectors. At different levels of policy activity, we expect the most relevant transformative outcomes pursued to also be different. The design of these theories of change is informed by transitions theory.
Boni, A., Molas-Gallart, J. (2020) Principles of formative evaluation. Available at https://tipresourcelab.net/resource/principles-of-formative-evaluation