Evaluation

http://evi.sagepub.com

Evaluation

DOI: 10.1177/1356389004043136

2004; 10; 77 Evaluation

Petri Virtanen and Petri Uusikylä

Evaluation

Framework for Understanding Micro–Macro Conversions in Programme

Exploring the Missing Links between Cause and Effect: A Conceptual

http://evi.sagepub.com/cgi/content/abstract/10/1/77

The online version of this article can be found at:

Published by:

http://www.sagepublications.com

On behalf of:

The Tavistock Institute

can be found at:Evaluation Additional services and information for

http://evi.sagepub.com/cgi/alerts Email Alerts:

http://evi.sagepub.com/subscriptions Subscriptions:

http://www.sagepub.com/journalsReprints.navReprints:

http://www.sagepub.com/journalsPermissions.navPermissions:

at PENNSYLVANIA STATE UNIV on April 16, 2008 http://evi.sagepub.comDownloaded from

Exploring the Missing Links between

Cause and Effect

A Conceptual Framework for Understanding

Micro–Macro Conversions in Programme

Evaluation

PETRI VIRTANEN

Net Effect Ltd, Finland

PETRI UUSIKYLÄ

Net Effect Ltd, Finland

Programme evaluation has become a widely applied mode of systematic

inquiry for making judgements about public policies. Although evaluation, as

a form of systematic inquiry, has provided feedback information for policy

makers, it still too often produces banal answers to complex and multi-

dimensional societal problems. In this article, we take a close look at the

ontological premises, conceptions of causality, and relationships to rational

theories of action of different programme evaluation paradigms. There is a

paradigm crisis in evaluation resulting from differences over assumptions

about causality. Evaluation paradigms clearly provide research strategies, but

more particularly they map causal links in contrasting ways. Traditional

cause-and-effect logic disregards the fact that programme effects are always

brought about by real actors rather than constructed ideal actors. A new

interpretation of causes and effects is needed, which would strengthen the

core ideas that lie behind the now widely applied and consolidated realistic

evaluation tradition.

KEYWORDS: causality; evaluation paradigms; micro–macro conversion;

programme evaluation

Introduction

Programme evaluation has become a widely applied mode of systematic inquiry

for making judgements about public policies. Evaluations have been used:

• ex ante – testing the coherence and applicability of the policy;

Evaluation

SAGE Publications (London,

Thousand Oaks and New Delhi)

DOI: 10.1177/1356389004043136

Vol 10(1): 77–91

06 043136 (ad/t) 2/4/04 3:18 pm Page 77

at PENNSYLVANIA STATE UNIV on April 16, 2008 http://evi.sagepub.comDownloaded from

• ex nunc – assessing the implementation and intermediate results of the

policy; and

• ex post – analysing the ﬁnal results and outcomes of the policy.

Although evaluation, as a form of systematic inquiry, has provided crucial

feedback information for policy makers, it still too often produces banal answers

(‘that we already knew but now it has been proven’) to complex and multi-

dimensional societal problems.

Are evaluators not sufﬁciently qualiﬁed to capture the true dynamics of public

policies or understand the cause-and-effect chains of those policies? Or is the

reason for symbolic, ‘nice to know’ evaluations, that policy makers want easy

answers or results that support already-made decisions? Or is the whole concept

of programme evaluation a ‘mission impossible’, i.e. is it simply impossible to

measure the causal effects of public policies? Our answer is partly all of these

but none alone. To produce a satisfactory answer we need to explore the onto-

logical premises of programme and project evaluation as well as their assump-

tions about causality.

This article ﬁrst discusses the epistemic nature of causality in programme

evaluation starting from the original ideas presented by Francis Bacon and David

Hume centuries ago. Second, we highlight some axioms of programme manage-

ment theory and thereafter discuss the relationship between these postulates and

– in ‘paradigmatic’ terms – three alternative models of public policy evaluation.

Then, third, these foundations are critically challenged with some of the core

arguments of social theory of action and its commentaries, mainly those of

Coleman (1990). Fourthly, linking all of the previous approaches and starting

points together, a new conceptual model for programme evaluation is presented

in order to encapsulate the ideas introduced in this article.

The Strong and Weak Postulates of Causality Assumption

in Programme Evaluation

It is necessary to clarify our starting point. First, we are not suggesting anything

revolutionary in terms of combining science with evaluation. On the contrary, in

our view, scientiﬁc reasoning and evaluative reasoning currently go ‘hand in

hand’. Second, we are aware of limitations in the typology presented in this

section.

Despite this, we present a conceptual model to map the differences in certain

programme evaluation paradigms, acknowledging that not every possible

perspective is included in this analysis. The contrast of goal-bound and goal-free

evaluation paradigms has also been made before by many others. One might

argue that these two possibilities do not differ signiﬁcantly in terms of causality

analysis. Indeed, they both require the evaluator to write statements about

effects (either intended or not), which implies that s/he relies on sound cause-

and-effect reasoning. Only the experimental design requires that the evaluator

sticks to the initially intended effects in order to carry out the analysis of impacts.

Most approaches to causality analysis, including quantitative and qualitative

Evaluation 10(1)

06 043136 (ad/t) 2/4/04 3:18 pm Page 78

at PENNSYLVANIA STATE UNIV on April 16, 2008 http://evi.sagepub.comDownloaded from

approaches and mixed methods, apply to either intended or unintended effects

as well as intended and unintended cause-and-effect mechanisms.

Science does not just describe the world; it also gives us explanations of how

and why things are as they are. This starting point is essential for trustworthy

programme evaluation practice and for understanding the logic of programme

evaluation. In addition, many argue that scientiﬁc and philosophical theories

should relate to that which can be observed, measured and recorded, and should

provide explanations in terms of causal laws. The reverse side of this is specu-

lation. To understand the philosophy that lies beneath this kind of thinking, a

short look back into the history of science is required.

Francis Bacon (1561–1626), and his Novum Organum (published 1620), paved

the way for modern science. Bacon’s method rested on two key pillars: obser-

vation (the scientist records the world using the data of sensory experience); and

induction (the scientist then generalizes from a whole collection of particular

instances to draw general conclusions) (e.g. Ladyman, 2002: 18–29). However,

the classic discussion of the problem of induction and causality is to be found in

the work of David Hume (1711–76). Hume related induction to the nature of

causation and the laws of nature. According to Hume, causal relationships

connect ideas that have – at ﬁrst sight – no logical relationship. Hence the basic

idea is that scientiﬁc explanation is causal explanation. Whatever is being

explained is the explanandum and that which does the explaining is the explanans

(Van Fraassen, 1998: 269–70; Vedung, 1997: 166–9; Pawson and Tilley, 1997: 30–4;

Mohr, 1995: 266–9).

According to the seminal work by Popper (1980: 59–60), to give ‘. . . a causal

explanation of an event means to deduce a statement which describes it, using

as premises of the deduction one or more universal laws, together with certain

singular statements, the initial conditions.’ As Popper underlines, the initial

conditions describe what is usually referred to as the ‘cause’ of the event in

question and the prediction describes what is usually referred to as the ‘effect’.

Thus, according to Popper, the principle of causality is the assertion that any

event whatsoever can be causally explained, i.e. it can be deductively predicted.

As a whole, it would be tempting to use the terms coined by Kuhn (1962) to

describe the situation regarding major trends in evaluation methods and method-

ologies. Are the current methods and methodologies used in various evaluation

activities, forms of what Kuhn calls ‘normal science’? Are the problems of

modern evaluation practices, puzzle-solving activities, where rules for solving

puzzles are quite strict and determined by various guidelines, standards and

methods? Are the current evaluation procedures conservative in the sense that

methodological innovations are not produced? Considering the role of causal

reasoning in different evaluation paradigms brings us face-to-face with a

paradigm crisis. In our view, a new understanding is needed for causal interpre-

tation of cause and effect, and this kind of reasoning would – at least to some

extent – strengthen the core ideas which lie beneath the currently widely applied

and consolidated realistic evaluation tradition (Pawson and Tilley, 1997).

The arguments stated herein do not offer new theory, but rather an alterna-

tive view. Very little conﬁrming research or literature is currently available to

Virtanen and Uusikylä: Exploring the Missing Links between Cause and Effect

06 043136 (ad/t) 2/4/04 3:18 pm Page 79

at PENNSYLVANIA STATE UNIV on April 16, 2008 http://evi.sagepub.comDownloaded from

support the argumentation. Therefore, demands for falsifying our argument will

need to be dealt with in the future. As Popper argued, theories can be ranked

according only to their degree of falsiﬁability and thus by a true measure of their

empirical content. This article is located somewhere in the realm of discovery

rather than the realm of justiﬁcation.

Evaluation Paradigms and Programmes

There is debate about causality across different evaluation paradigms. Three

evaluation paradigm interpretations will be considered here to highlight and

consolidate the authors’ views about causality. The ﬁrst paradigm represents

‘goal-free’ evaluation, the second, ‘goal-bound’ evaluation, to apply the well-

known classiﬁcation of Michael Scriven (1981), and the third, the realist

paradigm.

Bearing in mind all the above-mentioned restrictions on evaluation paradigm

typologies, consider the ﬁrst approach: Guba and Lincoln’s magnum opus,

Fourth Generation Evaluation (1989). For Guba and Lincoln, evaluation is funda-

mentally a social, political and value-oriented enterprise, and mainly because of

this, they do not treat evaluation as a scientiﬁc process. Fourth generation evalu-

ation moves away from the existing ‘. . . measure-oriented, description-oriented

and judgement-oriented’ paradigm into something new, which is probably most

easily captured by the term ‘negotiation’ (1989: 7–8).

Consider the reverse of measurement-, description- and judgement-oriented

evaluation. The result is non-measurable-oriented, non-description-oriented and

non-judgement-oriented evaluation. Polemically, it would be tempting to

summarize this approach as one of bias, speculation and indifference. Guba and

Lincoln do not devote much space to deﬁning the nature of the programme or

project to be evaluated or their goals (see Guba and Lincoln, 1989: 71–4). They

focus instead on the views of various stakeholder groups, alongside which the

evaluation agenda should be developed. Co-operation is the key word and it

must be invoked in every evaluation. Guba and Lincoln draw attention to causal-

ity and generalization, since ‘conventional’ evaluation methodologies, according

to them, do not take into account contextual factors except by physically and

statistically controlling them.

In 1989, Guba and Lincoln’s ‘constructivism’ offered a fresh approach to the

evaluation community, challenging the prevailing positivist paradigm. By under-

lining the role of negotiation, putting the emphasis on hermeneutic methodol-

ogy and stressing the importance of relativistic ontology, they blew fresh air into

the discussion of evaluation tools, techniques, methods and methodologies. But,

ultimately, their classic is a child of its era. Reading it 14 years after its publi-

cation, the harsh attacks on quantitative methods are irritating – one might even

assume that the attacks were Guba and Lincoln’s main point. The result, in

methodological terms, was a severely one-sided interpretation of how evaluation

should be carried out. Alternative models were judged negatively.

But what was Guba and Lincoln’s object of attack? This question leads us to

the second paradigm interpretation. We refer to it as ‘normal’ (evaluation)

Evaluation 10(1)

06 043136 (ad/t) 2/4/04 3:18 pm Page 80

at PENNSYLVANIA STATE UNIV on April 16, 2008 http://evi.sagepub.comDownloaded from

science; this prevailed during the 1960s, 1970s and 1980s, prior to Guba and

Lincoln’s contribution. The ideas of Evert Vedung (1997) and Peter Rossi and

colleagues (1999) are used here to exemplify the goal-bound tradition of

programme evaluation. Vedung (1997: 223–5), for example, considered a

programme to be an entity understood by means of intervention theory consist-

ing of causes (hypotheses underlying intervention logic), intervention per se

(programme), planned conversion (expected results), intended output (in

concrete, measurable terms), intended immediate outcomes, intended inter-

mediate results, and intended ultimate (societal) outcomes. The difference

between this approach and that of Guba and Lincoln is much more than

semantic. According to this paradigm it is important to map inter-linkages

between outcomes (or effects), results, outputs, activities, inputs and objectives.

This approach has recently been labelled result-chain thinking (Uusikylä and

Valovirta, 2002). In the goal-bound approach, the evaluator aims to map causal

interpretations between objectives, inputs, outcomes and effects.

Finally, the third paradigm, which derives from realist evaluation, tries to

combine the elements of goal-free and goal-bound approaches. Pawson and

Tilley (1997: 63–4) argue that social programmes are ‘. . . undeniably, unequivo-

cally, unexceptionally social systems. They . . . comprise the interplays of indi-

vidual and institution, of agency and structure, and of micro and macro social

processes.’ There seems therefore to be much to learn from investigating the

‘social nature’ of programmes. Pawson and Tilley (1997) suggest that an evalu-

ator should focus on certain themes in carrying out evaluation assignments,

including embeddedness, mechanisms, contexts, regularities and change.

The key idea in Pawson and Tilley’s reasoning is the connections between

contexts (C), mechanisms (M) and outcomes (O). This is what they call the CMO

principle. As we understand this logic, an evaluator should focus on the inter-

vention but this alone is not adequate. S/he should also construct the hypotheses,

implement data-collection strategies and analyse data in terms of CMO conﬁgur-

ation. This means in practice that an evaluator tries to understand what might

work for whom in certain circumstances, uses multi-method data-collection oper-

ations and data analysis, and ﬁnally consolidates his/her ﬁndings with the context

in which the programme operates (thus answering the question: what worked for

whom in what circumstances?).

Pawson and Tilley’s conception comes close to the classic ideas of causality

presented by David Hume almost 400 years ago. Hume further examined the

concept of causality and stressed the importance of contiguity – the relationship

of being connected in space and time (see Hume, 1963). Table 1 lists the main

methodological differences between the alternative approaches to programme

evaluation.

Micro–macro conversion will now be considered and the ideas of James

Coleman will form a conceptual bridge to develop a novel conception of causal-

ity in programme evaluation. In addition, conceptual ideas, originally developed

by Kaufmann (1987), related to the intermediate sphere between the micro and

macro, will be linked to ideas presented by Coleman to formulate a better frame-

work for understanding the logic and usefulness of programme evaluation.

Virtanen and Uusikylä: Exploring the Missing Links between Cause and Effect

06 043136 (ad/t) 2/4/04 3:18 pm Page 81

at PENNSYLVANIA STATE UNIV on April 16, 2008 http://evi.sagepub.comDownloaded from

Evaluation 10(1)

Table 1. Main Approaches in Programme Evaluation

‘Goal-free-approach’ ‘Goal-bound approach’ Realistic approach

Philosophical base Phenomenology Positivism Realism

Author example Guba and Lincoln, Vedung, 1997 Pawson and Tilley, 1997

1989 Rossi et al., 1999

Mohr, 1995

Disciplinary base Sociology; Economic theory Social policy

Anthropology Political theory Sociology

Focus of Describe programme Judge worth of To identify and analyse

methodology holistically and programme mechanism-context-

from perspective Identify causal links outcome

of the participants relationships

Methodology Ethnography, case Experimental and Context  mechanism

studies; participant quasi-experimental  outcome

observation; designs

triangulation Cost–beneﬁt

analysis

Causality Reluctant to map out Primary concern An issue on the

interpretation the connection research agenda,

between causes special emphasis on

and effects the CMO-framework

Variables Emerging in course Predetermined as Predetermined and

of evaluation input–output emerging

Control or Not necessarily Yes Not necessarily

comparison

groups

Participants role in Active None Active

carrying out

evaluation

Evaluator’s role Interactive Independent of Observing

programme

Political pressures Describe Controlled in Describe and explain

(internal/external) design

Focus of evaluation Present holistic Render go/no go To understand and

report portrayal of decision report CMO-

programme in conﬁgurations

process

06 043136 (ad/t) 2/4/04 3:18 pm Page 82

at PENNSYLVANIA STATE UNIV on April 16, 2008 http://evi.sagepub.comDownloaded from

The Use of Meta-theories to Find Causal Explanations in

the Implementation of Programmes

Alternative approaches to programme evaluation vary with regard to their treat-

ment of programmes and the action taken to implement the activities of the

programme. For example, economic models might be said to be heavily under-

socialized explanatory models of societal reality, i.e. they focus on programmes and

their societal goals while ignoring the societal context and the actions of individuals

through which programmes are implemented.

This may, in extreme cases, lead to

false and mechanistic interpretations of the results and impacts of programmes.

Alternatively, the constructivist (and partly also the realist) approach to evalu-

ation represent over-socialized interpretations of programme reality. These

approaches tend to neglect the programme goals and focus rather too much

attention on negotiations between stakeholders and consensus building. Thereby

they seem in constant danger of lapsing into wholesale relativism. The behav-

ioural approach falls somewhere between the under- and over-socialized models.

Although the behavioural approach examines action and human behaviour, this

has been undertaken rather pre-deterministically. In practice, the result is a

model that takes into account the programme objectives and the results of an

experimental study while saying nothing about those mechanisms that transfer

programme objectives into programme outputs.

One of the main criticisms of mechanistic cause-and-effect models is that they

neglect the fact that an action occurs in context and thus its success depends upon

the way an action is performed by a speciﬁc actor in a speciﬁc situation (as the

realist school of evaluation has emphasized). Kaufmann (1987) formulated the

following two chains of logic long before the emergence of the realist evaluation

tradition, formulated by Pawson and Tilley in 1997.

Figure 1 describes the fact that traditional cause-and-effect logic is undermined

by the fact that the effects of a programme are always caused by real actors rather

than constructed ideal actors. A programme always exerts inﬂuence through

social action and its course follows from actions that are based on interpretations

of the expected logic of the programme. Actions are always socially constructed

rather than objectively derived from abstract programme logic. Any programme

always interferes with a pre-existing intervention ﬁeld where social actors react

to produce ﬁnal outcomes or effects.

Figure 2 elaborates the model further. It begins with the assumption that

programmes are socially constructed, i.e. a certain group of actors (with certain

intentions and their best understanding of society’s needs) develop programmes.

These will be implemented by other actors who might – and most often do – have

their own interpretations of programme logic and its situational validation. There

will then be a third set of actors who try to adapt themselves to the expected goals

and objectives (based on their own subjective understanding) and ﬁnally imple-

ment the programme. From this logic of programme implementation constructed

by multiple sets of actors, we are dealing with several, rather than one, interpre-

tations of a programme. We therefore need a more precise methodology to

explore how social programmes are turned into real outcomes and effects.

Virtanen and Uusikylä: Exploring the Missing Links between Cause and Effect

06 043136 (ad/t) 2/4/04 3:18 pm Page 83

at PENNSYLVANIA STATE UNIV on April 16, 2008 http://evi.sagepub.comDownloaded from

These issues regarding the relationship between programmes and actions

resemble the classic micro–macro (or agency–structure) problem in modern

sociological theory. This debate revolves around the following issues:

• How do structures determine the actions of individuals?

• How are structures created?

•What limits, if any, are there on the capacity of individuals to act indepen-

dently of structural constraints?

These all relate to the central issue of the limits on human agency. In terms of

programme evaluation, the related questions are:

• How do programmes determine the activities of projects (or the individuals

related to it)?

• How are programmes created?

•What limits are created by programmes to the capacity of projects (and

individuals) to act independently?

A Conceptual Model for a Research Agenda

In existing programme evaluation models there is a gap between programme-

level results and project-level results. This is a problem of aggregation:

programme-level results cannot simply be aggregated from project-level results,

although many programme evaluators tend to believe they can be. This is similar

to what Fischer (1995) has called mapping out situational validity in programme

evaluation.

Evaluation 10(1)

FALSE PROGRAMME

LOGIC

A PROGRAMME

initiates

A CASUAL PROCESS

that produces

EFFECTS

ELABORATED PROGRAMME

LOGIC

A PROGRAMME

interferes with

AN INTERVENTION FIELD

whose reactions produce

EFFECTS

Figure 1. The True and False Social Intervention Logic of a Programme (adapted from

Kaufmann, 1987: 10-11)

06 043136 (ad/t) 2/4/04 3:18 pm Page 84

at PENNSYLVANIA STATE UNIV on April 16, 2008 http://evi.sagepub.comDownloaded from

This aggregation problem is demonstrated by an example from the European

Social Fund (ESF), applying the model proposed by James Coleman (1990) and

previously developed ‘at the idea level’ by Kaufmann (1987). The positivist or

behavioural evaluation approach often implicitly assumes that the introduction

of a socio-economic programme leads to a particular expected change in society,

e.g. increased employment or regional competitiveness. Also the evaluation of

impacts follows the same logic (e.g. as expressed in evaluation guidelines – such

as the MEANS Handbook series – adopted and supported by the European

Commission).

The major problem for explanations of system behaviour based on actions and

orientations at the level below that of the system is how to move from the lower

level to the system level. For example, in economics there is microeconomic and

macroeconomic theory; and one of the central deﬁciencies in economic theory is

Virtanen and Uusikylä: Exploring the Missing Links between Cause and Effect

ACTORS

A PROGRAMME

develop

OTHER ACTORS

(with their intentions and

definitions of the situation)

(with their intentions and

definitions of the situation)

which is implemented by

for

whose reactions produce

BENEFICIARIES

EFFECTS

(with intentions and on the

basis of a defined situation)

Figure 2. The Socially Embedded Course of Programme Intervention (adapted from

Kaufmann, 1987: 11–12)

06 043136 (ad/t) 2/4/04 3:18 pm Page 85

at PENNSYLVANIA STATE UNIV on April 16, 2008 http://evi.sagepub.comDownloaded from

the weakness of the linkage between them, a weakness papered over with the

idea of ‘aggregation’ and with the ubiquitous concept in macroeconomic theory

of the representative agent (Coleman, 1990: 6–7). Employment programmes (at

the document level) often indicate a series of programme activities which aim to

enhance employment effects without specifying the intervention logic of these

activities, i.e. the process that transforms cause into effect and outcome. Figure

3 leaves questions unanswered regarding the kind of proposition being demon-

strated and in particular, what unit or units were involved in the proposition (i.e.

the macro-level proposition is not sufﬁciently detailed).

The single proposition can however be split into three:

• an independent variable characterizing the individual;

• both independent and dependent variables characterizing the individual;

• the independent variable characterizing the individual and the dependent

variable characterizing the system.

Figure 4 shows a way of representing such multi-level systems of propositions.

The upper horizontal arrow represents the baseline macro-level proposition.

There are then three connected arrows. The ﬁrst begins from the macro-level

proposition and goes down to a lower level (individual level). This arrow (1)

represents the transformation function that converges the objectives of the ESF

programme (macro-level) into a local- (or regional-) level understanding of the

societal goals that the programme serves. Thereafter, local actors (through

bargaining and exchange of information) transform these internalized programme

values into concrete project ideas and local priorities (2). Local projects

(micro-level) formulate their project ideas so that projects meet local needs and

articulate programme goals (such as innovative approaches to local employment,

customer orientation, quality principles etc.). The third lower-level arrow (3)

represents the turning of individual or project values into action (i.e. projects or

their stakeholders start acting differently, providing new innovative services etc.).

These results and experiences can then be aggregated to local results (4) (inno-

vations, increased employment etc.). The last arrow (5) is the most important

and interesting, because it transfers the local project results into programme (i.e.

systemic) results. In concrete terms it veriﬁes the aggregation of changes taking

place regarding individual-level behaviour which then results in local- (or

regional-) level changes (i.e. increased employment, better competitiveness in

the region etc.) which in turn can then be aggregated into programme results.

Evaluation 10(1)

ESF Objective 3

Programme

Increased

employment

Figure 3. Macro-level Proposition: ESF Objective 3 Programme Increases Employment

06 043136 (ad/t) 2/4/04 3:18 pm Page 86

at PENNSYLVANIA STATE UNIV on April 16, 2008 http://evi.sagepub.comDownloaded from

Derived from Figure 4, the practical research questions relevant to this particu-

lar evaluation are brieﬂy summarized in Table 2. The questions listed in the table

would be speciﬁed further during the evaluation. This set of questions is, in other

words, the basic set of questions, from which the actual questions asked should

derived.

Discussion

This article aims to highlight the complexity and multi-dimensional nature of

modern society and that this is difﬁcult to capture with mechanistic evaluation

approaches. Evaluators, therefore, need specialized knowledge to calibrate their

evaluation studies and design experiments.

The debate on causality in programme evaluation has not been as active as it

might have been, considering the challenges facing programme evaluations. In

the European Union (EU) this is already true in the ongoing programming

period (2000–6) of Structural Funds, even without the changes in Structural Fund

framework that will follow the enlargement of the EU. As the number of

Virtanen and Uusikylä: Exploring the Missing Links between Cause and Effect

ESF Objective 3 Programme Increased employment

IIV

II III

MACRO

LEVEL

MESO

LEVEL

MICRO

LEVEL

Programme

created with

certain set of

actors that share

common understanding

of the programme goals

Local actors transform

programme goals into goals

reflecting local priorities

through the involvement of

local networks

Individuals in various

projects transform

programme goals and values

into project goals and values

These goals are

transformed into

activities and outputs

that benefit clients

Local innovations

shape local modes

of activities

Local results are

transformed into

macro effects

Figure 4. Case Example on Macro- and Micro-level Propositions: Impacts of the ESF

Programme on Employment (based on the ideas presented in Coleman [1990: 8] and

Kaufmann [1987])

06 043136 (ad/t) 2/4/04 3:18 pm Page 87

at PENNSYLVANIA STATE UNIV on April 16, 2008 http://evi.sagepub.comDownloaded from

countries increases and the funding to be allocated reduces, it is becoming

increasingly important to produce trustworthy programme evaluations that link

causes and effects of structural interventions. This article has elaborated on

traditional cause-and-effect evaluations towards a multi-level research agenda,

introducing the programme–project–programme logic chain. Further empirical

studies need to be undertaken to test whether our multi-level logic offers a better

understanding of the real value-added of an individual programme.

There are at least two prerequisites for the application of a multi-level evalu-

ation logic. First, there should be a clear understanding of what are the causal

mechanisms that generate certain programme effects, i.e. working hypotheses on

expected programme effects. Second, programme planners should have a contex-

tual understanding of those situational conditions that either enable or hinder

Evaluation 10(1)

Table 2. Concrete Research Agenda Including Research Questions to Be Tackled

(Based on Figure 4).

Set of questions (reﬂecting Programme evaluation research theme

the numbers presented (examples of evaluation questions)

in the Figure 4)

Relation 1 How was the programme planned and its goals set?

What was the level of consensus in mapping out key priorities at

the programme level?

Relation 2 What is the local structure for implementing the programme?

Who are the key actors in the network and what is the volume

and quality of connections within the network?

What is the local focus in implementing the programme? What

themes and issues are tackled in particular?

What kind of selection criteria for projects are applied?

Relation 3 What is the competence of the project managers and project

teams?

What is the human, organizational and client capital of the

projects?

How do the projects develop global objectives, mission

statements and expected results, and design their activities?

How efﬁciently are the projects carried out?

What is the relationship between expected activities and actual

activities?

How does the project self-monitor its success?

Relation 4 How do the developed local innovations function in local

settings?

What is the logic of the local innovation process and who are the

key actors in developing new ideas?

Relation 5 How are the local innovations being transferred into wider

contexts to be utilized?

How can utilization and dissemination aspects be strengthened?

06 043136 (ad/t) 2/4/04 3:18 pm Page 88

at PENNSYLVANIA STATE UNIV on April 16, 2008 http://evi.sagepub.comDownloaded from

the transformation of programme activities into effects and impacts. This is very

important in relation to the dissemination of results and/or models caused by

programmes.

Finally, the question of causality should be linked with the motive and use of

evaluation information. Figure 5 describes four alternative (ideal) models of evalu-

ation in terms of how explicitly causality has been taken into consideration and

how the model of evaluation serves the aims of enhancing public-sector account-

ability. Accountability in this sense can be considered as a macro-political concept

that monitors the degree of success of political and administrative attempts to

provide public services to citizens and govern legitimately (Uusikylä, 1996).

In Model 1 – Scientiﬁc Inquiry – causality has been strongly linked with the

evaluation design but lacks an emphasis on accountability. This hinders the utiliz-

ation of evaluation results and their systematic application to system-building. In

Model 2 – Explorative Inquiry – the situation is contrary: the emphasis on

accountability is strong and explicit but it is difﬁcult to distinguish the causal

effects of public intervention. In these circumstances, evaluations become more

descriptive and qualitative. In Model 3 – Symbolic Evaluation – causality and

accountability are implicit, for example, and evaluations serve a symbolic

purpose rather than the true pursuit of learning. Finally, in Model 4 – Trans-

parent Democracy – both causal assumptions are clear and the emphasis on

Virtanen and Uusikylä: Exploring the Missing Links between Cause and Effect

Accountability in public interventions

Causality in

evaluation

Strong

Weak

Scientific

inquiry

Symbolic

evaluation

Transparent

democracy

Explorative

inquiry

Figure 5. The Differing Emphases Placed on Causality and Accountability within Four

Models of Evaluation

06 043136 (ad/t) 2/4/04 3:18 pm Page 89

at PENNSYLVANIA STATE UNIV on April 16, 2008 http://evi.sagepub.comDownloaded from

accountability is high. This model represents the use of evaluation as a true

democratic control. It resembles what Fischer (1995) calls ‘societal-level vindi-

cation’ in a diffuse and complex system of modern governance.

Note

1. ‘Under-socialized’ is a term borrowed from Granovetter (1985) who classiﬁes socio-

logical and economic theories according to their emphasis on agency and structure.

References

Coleman, J. (1990) Foundations of Social Theory. Cambridge, MA: Belknap

Press/Harvard University Press.

Fischer, F. (1995) Evaluating Public Policies. Chicago, IL: Nelson-Hall Publishers.

Granovetter, M. S. (1985) ‘Economic Action and Social Structure: The Problem of

Embeddedness’, American Journal of Sociology 91(3): 481–510.

Guba, E. G. and Y. S. Lincoln (1989) Fourth Generation Evaluation. Thousand Oaks, CA:

Sage.

Hume, D. (1963) An Enquiry Concerning Human Understanding. Oxford: Oxford

University Press.

Kaufmann, F.-X. (1987) ‘Prevention and Intervention in the Analytical Perspective of

Guidance’, in K. Hurrelmann, F.-X. Kaufmann and F. Lösel (eds) Social Intervention:

Chances and Constraints, pp. 3–20. Berlin and New York: Walter de Gruyter.

Kuhn, T. S. (1962) The Structure of Scientiﬁc Revolutions. Chicago, IL: University of

Chicago Press.

Ladyman, J. (2002) Understanding Philosophy of Science. London: Routledge.

Mohr, L. B. (1995) Impact Analysis for Program Evaluation. Thousand Oaks, CA: Sage.

Pawson, R. and N. Tilley (1997) Realistic Evaluation. London: Sage.

Popper, K. R. (1980) The Logic of Scientiﬁc Discovery. Cambridge: Unwin & Hyman.

Rossi, P. H., H. Freeman and M. Lipsey (1999) Evaluation. A Systematic Approach. New

York: Sage.

Scriven, M. (1981) Evaluation Thesaurus. Newbury Park, CA: Sage.

Uusikylä, P. (1996) ‘Agency Discretion and Public Accountability’, in Hugo van Hassel

(ed.) New Trends in Public Administration and Public Law. EGPA Yearbook 1996.

Budapest: EGPA and Center for Public Affairs Studies.

Uusikylä, P. and V. Valovirta (2002) Osaoptimoinnista tulosketjuihin. Helsinki: Valtio-

varainministeriö [From Partial Optimizing to Result-Chains]. Helsinki: Ministry of

Finance. Available only in Finnish.

Van Fraassen, B. C. (1998) ‘The Pragmatics of Evaluation’, in E. D. Klemke, R. Hollinger,

D. W. Rudge and A. D. Kline (eds) Introductory Readings in the Philosophy of Science,

pp. 264–77. New York: Prometheus Books.

Vedung, E. (1997) Public Policy and Program Evaluation. New Brunswick, NJ: Transac-

tion Publishers.

PETRI VIRTANEN is currently senior partner and Chairman of the Board at Net

Effect Ltd, a Helsinki-based private consultancy, and he is involved in evaluation

and development projects commissioned by the Finnish ministries, the European

Commission and the World Bank. Previously, Dr Virtanen was a professor of

Evaluation 10(1)

06 043136 (ad/t) 2/4/04 3:18 pm Page 90

at PENNSYLVANIA STATE UNIV on April 16, 2008 http://evi.sagepub.comDownloaded from

social policy at the University of Tampere (Finland), 1996–7 and head of unit at

the Ministry of Finance, 1997–2000. He is a board member of the Finnish

Evaluation Society. Please address correspondence to: Net Effect Ltd, Mikonkatu

13 A 34, FIN-00100 Helsinki, Finland. [email: [email protected]]

PETRI UUSIKYLÄ is currently senior partner and Managing Director at Net

Effect Ltd, and he is involved in several evaluation and development projects

commissioned by the Finnish ministries, local government and the European

Commission as well as the World Bank. Previously, Petri Uusikylä was a senior

adviser at the Ministry of Finance and an assistant professor at the University of

Helsinki. Please address correspondence to: Net Effect Ltd, Mikonkatu 13 A 34,

FIN-00100 Helsinki, Finland. [email: [email protected]]

Virtanen and Uusikylä: Exploring the Missing Links between Cause and Effect

06 043136 (ad/t) 2/4/04 3:18 pm Page 91

at PENNSYLVANIA STATE UNIV on April 16, 2008 http://evi.sagepub.comDownloaded from