EVALUATION: DEFINITIONS, METHODS AND MODELS
25
evaluation
1
is focused on the relative degree to which a given product effec-
tively meets a previously specified goal, while goal-free evaluation
2
measures
the effectiveness of a given product exclusively in terms of its actual effects
- the goals and motivations of the producer are ignored. Each approach has
relative advantages and disadvantages. On the one hand, goal-bound evaluation
is ordinarily more cost-effective than goal-free evaluation; on the other hand,
measuring effectiveness entirely in terms of the degree to which stated goals
are met can have at least two undesirable consequences: (a) since effectiveness
is, on this model, inversely proportional to expectations, effectiveness can be
raised simply by lowering expectations, and (b) deleterious or otherwise un-
wanted effects, if any, are left out of account, while unintended benefits, if any,
go unnoticed.
Economic models, on the other hand, test whether program’s productivity,
effectiveness and utility have been satisfactory in terms of expenses. Cost
analysis is currently a somewhat controversial set of methods in program
evaluation. One reason for the controversy is that these terms cover a wide
range of methods, but are often used interchangeably. Whatever position an
evaluator takes in this controversy, it is good to have some understanding of the
concepts involved, because the cost and effort involved in producing change is
a concern in most impact evaluations (Rossi & Freeman, 1993).
• Cost allocation is a simpler concept than either cost-benefit analysis or
cost-effectiveness analysis. At the program or agency level, it basically
means setting up budgeting and accounting systems in a way that allows
program managers to determine a unit cost or cost per unit of service. This
information is primarily a management tool. However, if the units measu-
red are also outcomes of interest to evaluators, cost allocation provides
some of the basic information needed to conduct more ambitious cost ana-
lyses such as cost-benefit analysis or cost-effectiveness analysis;
• Cost-effectiveness and cost-benefit studies are often used to make broad
policy decisions, the terms might be used interchangeably, but there are
important differences between them: by definition, cost-effectiveness
analysis is comparative, while cost-benefit analysis usually considers only
one program at a time. Another important difference is that while cost-
1
Tyler (1949) is the first one to propose goal-oriented and objectives-based models in order to describe
whether students have met their goals or not, underlying how this model can be very useful in providing
information about how to handle new strategies and reform old
ones
2
For an introduction to the distinction between goal-free and goal-bound evaluation methods as well as a
thorough review of their respective strengths and weaknesses, see Michael Scriven, "Evaluation Perspectives
and Procedures," in W. James Popham, ed., Evaluation in Education: Current Applications (Berkeley, CA:
McCutchan Publishing Corporation, 1974).