Program logic models

Program logic also goes under several other names: program theory, logic modelling, program logic models, theory of action, and theory-based evaluation. If you're searching for information about it on the Web, you'll need to look under all those names. The "programs" here are programs of planned action in general - usually not computer programs, radio programs, or TV programs.

Program logic (as I'll call it here) seems to be an extension of the Logical Framework Approach, which was developed in the 1960s for international development projects, and is now widely used for that purpose. (Too widely, perhaps, given its rigidity.)

Around the 1980s came program logic modelling, also known as program theory. Till recently, this approach has been used most in the evaluation of health and social welfare programs. Later offshoots, not covered here, serve the increasing focus on accountability in public service work, such as Results-based Measurement, widely used by the Canadian government. Program logic models, on which this page is focusing, are produced in a wide variety of different forms, with this common basis:

A program is defined as a sequence of objectives.
Usually the designers begin with an desired social impact in mind, and work back from that goal to determine what must be done to accomplish it.
To achieve that impact, some intermediate goals (outcomes) must first be reached.
To attain those outcomes, the program must produce some outputs.
To produce the outputs, activities are required.
And in order to do the activities, inputs or resources are needed.

A simplified form of the sequence can be shown thus:

The diagram is very abstract, so let's create an example to show how the model can work: a radio program designed to educate people about how to prevent malaria. (This is a very simple example, with only a single chain. Real examples are a lot more complex.)

The inputs are the money, labour, and facilities needed to produce the program.
The activity is producing and broadcasting the program.
The output is that some people hear the program.
The outcome is that some of those listeners act on the program's advice.
The impact is that malaria in that area is reduced.

The power of logic modelling is that each step in the chain can be evaluated in a way that's most appropriate at that point.

Did the inputs (money etc.) really produce the activities (the broadcast)? That question can be answered by auditing.
And did the activities produce the output (an informed audience)? An audience survey could answer that question.
Did the output produce the outcome (how many listeners acted on the advice)? The audience survey could answer that, too.
To measure the impacts, public health statistics could be used, perhaps comparing the area where the program was broadcast with a matching area where it was not.

As you move to the right along the causal chain, it gets more difficult to provide clear answers - e.g. if malaria really was reduced, was it because of that radio program, or for various other reasons?

But the principle is that you specify what you hope will happen, in the form of a chain, and then test each link in the chain to work out what really did happen. That's program logic modelling in a nutshell. At first it seems like a ridiculously cumbersome way of stating the obvious, but its value in practice lies in that step-by-step testing. If the program didn't work as intended, you can detect the weak links in the logic chain.

To confuse the uninitiated, the labels "impacts" and "outcomes" for the long-term and short-term objectives are sometimes reversed. Also bear in mind that the division of results into two types is arbitrary: there can also be medium-term effects.

The above diagram is represented as a chain, implying that no stage can occur until all previous stages are complete. In that sense, it resembles a chain scenario (or sequence of possible future events): the main difference is that program logic embodies one overriding intention, while a scenario does not. The problem with this type of thinking is that (though its truth seems obvious) the world doesn't always work like that. This "hierarchy of effects" approach, as it's sometimes called, just doesn't correspond with reality. For example, there's the famous AIDA model of marketing. This means that A (awareness) happens first - people become aware of a product. Then I happens: they're interested in it. Then it's D: they Desire it. Finally, they Acquire it. It makes sense, doesn't it? The trouble is that a lot of stages are often skipped. A few years ago we did a study of why students choose to do postgraduate study at a particular university. We interviewed 19 students, and not a single one followed the "obvious" AIDA model.

Given these types of problem, thinking about "impacts" and "outcomes" is being superseded by an approach known as the outcomes hierarchy. Instead of just two levels, with impacts following from outcomes, there's a whole network, showing that effects can occur through different paths. Here's an explanatory report on outcomes hierarchies.

But getting back to the above diagram, and setting aside for the moment the complexities of real-world outcomes, the diagram's simplicity conceals its power - in two ways:

For those planning a program, to specify such a chain is a useful exercise. It forces them to clarify their theory of action.
After the chain has been agreed, evaluation becomes much simpler (conceptually, at least) because each step of the chain can be evaluated using the most relevant data for that stage, specifically...
- Did the inputs (money and people's time, etc) result in the planned activities being performed? This can be evaluated from work-time logs and financial data.
- To what extent did those activities produce the planned outputs? Data from the program can provide that detail.
- To what extent did those outputs result in the planned direct outcomes? This area is the familiar territory of evaluation: it may involve surveys, experiments, etc.
- And to what extent did those direct outcomes (if achieved) produce the desired broad outcomes? This too is a standard evaluation question, but usually the most difficult to answer. Official statistics are often used at this point.

A more complex example is a project I worked on over several years, with public broadcasters in Vietnam. The activity was the introduction of live radio and talkback programs on provincial radio stations. The inputs were funding and training expertise, provided by a Swedish aid agency. The activity was training local radio producers in the principles of live and talkback radio. The outputs were trained producers. The direct outcomes were the new programs they broadcast. The desired broad outcome (by the funding agency) was improved democracy.

(In fact, this project, like much international development work, used the Logical Framework approach, but I have re-expressed it as a program logic model.)

Notice that the above logic model was described from the Swedish point of view. Though the Vietnamese view of the inputs and activities would be much the same, the later stages could be seen quite differently. It would be plausible to view the end-goal of the Vietnamese radio network as survival in the face of the increasing popularity of television - achieved through an intermediate goal of a new, slicker style of radio programming. From a broader point of view still, it would be possible to see the end-goal as educating the Vietnamese public in the possibilities of consumer choice. This in turn could lead to greater transparency in government and business, democratic involvement with the political process, and increased awareness of the power of consumerism - not quite the same as democracy.

The crucial point - of which this Vietnamese case is only one example - is that from each actor's point of view, the logic model would be somewhat different. When a funder and the recipient have different logic models, they might both commission evaluations, which might not concur on the program's achievements.

The further to the right in the sequence, the more difficult it becomes to relate inputs to outputs, or causes to effects. (This is nearly always the case with logic models.) To begin at the left: did providing the resources result in the desired activities? Simple clerical work could answer that question. But whether the outputs of the project - regular live radio programs with talkback - actually led to increased democracy in Vietnam is much more elusive to measure. As this project was only one of many factors in the westernization of Vietnam, how could its contribution to enhanced democracy in Vietnam be accurately assessed? At the right-hand end of the chain, definitions and assumptions become crucial: a small change in the definition of an end-goal might result in a large change in the evaluated success of a project. Sue Funnell's (1997) cascading approach to program logic could help here: a project may not produce its intended broad outcomes unless specific interventions are used to assist in their accomplishment. Implication: produce the model before beginning the activities.

In practice, a program will usually involve multiple activities, with multiple inputs and outputs, all intended to work on multiple target groups towards a broader goal. For example, how would those talkback radio programs contribute to democracy? There might be several different types of impact from a regular talkback program. (Here, the output is defined from the Vietnamese point of view: i.e. the production of a regular radio program.)

Members of the public would listen to the talkback programs, and perhaps hear others questioning or criticizing government or business actions.
They would come to accept the principle that actions affecting all citizens are accountable to all citizens.
This would embolden some of the listeners to do take part in talkback programs themselves.
Officials would know about the program, and perhaps think twice about going ahead with some project of dubious democratic merit, for fear of being criticized on air. (They might also use such programs in numerous other ways.)
Elected representatives would promote some of the themes mentioned on the programs, in an attempt to be returned to office in the next election.

In those five ways, outputs might achieve short-term or long-term goals. Each result can be enumerated using the most appropriate methodology, e.g.

Criterion to be evaluated Method of evaluation
Program audience size, and listeners' awareness of content A survey, designed to measure the program's audience
Acceptance of principle that official actions are accountable to all citizens Qualitative research, e.g. consensus groups
Listeners take part in talkback programs Radio station keeps records of the number of people calling talkback program
Officials hesitate to make undemocratic decisions "Before" and "after" unattributed interviews with officials
Elected representatives take up themes mentioned on the programs Content analysis of parliamentary speeches and government news releases

But the question of how far those types of activity (even when enumerated) contribute to the enhancement of democracy is very much a political one. The answer to that question would be strongly influenced by the definitions used in the final stage of the logic model.

Limitations of program logic in practice

Program logic has limitations: some intrinsic, and others arising from the context in which it is normally carried out...

No account is taken of unintended consequences - though of course every program has side-effects.
Program logic focuses on a single issue - as intended by the program's managers.
It combines goal hierarchy and time sequence, and thus lacks a way of showing the effects of feedback.
It does not address issues of power, control, and participation. Its model is that experts "intervene" to do something to a passive and unintelligent "target group."
Program logic does not cope with conflict - because it addresses only the program planner's needs. (Imagine applying it to the US conquest of Iraq, to see how it can be a tool for oversimplification.)
Doubtful generalizability, or the "black box problem" - an unclear connection between outputs, outcomes, and impacts.
The assumption of omnipotence - that "because we have a plan, we must succeed in implementing it."

Despite those limitations, program logic is a very powerful method for evaluating the success of any complex activity. Most of the limitations can be overcoming by involving a wide range of stakeholders in building the models, and by having multiple models, with branching.

Program logic - an introduction

Limitations of program logic in practice

Further reading