UNESCO: United Nations Educational Scientific and Cultural Organization

The Organisation

THE ORGANIZATION

1. Basic Ideas
  • Aims of evaluation

  • What is Evaluation?

  • Three Qualities Seven Points of View

  • Cost of Evaluation


  • AIMS OF EVALUATION

    Programme specialists, right from the moment a programme or activity is being planned, must have the necessary information to answer questions such as:

  • What change is the programme or activity directed at? What are its objectives?

  • How is it intended to achieve them? What activities will be carried out and what means will be used?

  • What human and financial resources are to be used?


  • Evaluation then takes on its full meaning when answers have to be given to questions such as the following:

  • Is what was planned really being carried out? Are the expected results being achieved?

  • Are the results producing the desired changes?

  • Is the programme being implemented in a suitable way?


  • These latter questions are crucial in order to have a programme that responds to the needs of a group of persons, a country, etc.

    Evaluation thus has the following aims:

  • to provide organizations or individuals in charge of programmes, providers of funds, and organizations or individuals benefiting from the programmes, with an opportunity to make choices, learn from experience and provide explanations;

  • to determine the importance of a programme, taking the opinion of those benefiting from it as a basis;

  • to modify a programme so that it conforms more closely to the needs to be met in a particular social, political or economic circumstance.


  • Evaluation achieves these aims by:

  • collecting the data needed to answer the questions mentioned above;

  • processing and analyzing the data to identify particular trends and facts and draw conclusions as to whether the programme has been properly implemented and whether it should or should not continue in its current form.


  • An evaluation, whether simple or complex, can be justified only if it makes it possible to come to a decision, at an appropriate moment, whether:

    a) to continue in full the programme or activity being evaluated;

    b) to modify the programme or activity; or

    c) to discontinue the programme or activity.

    The most important criterion by which the quality of an evaluation process should be judged is thus its USEFULNESS FOR DECISION-MAKING. In other words, one should avoid programme evaluation for its own sake.


    WHAT IS EVALUATION?

    Programme evaluation can be defined as a systematic operation of varying complexity involving data collection, observations and analyses, and culminating in a value judgement with regard to the quality of the programme being evaluated, considered in its entirety or through one or more of its components.

    Evaluation means arriving at a value judgement on the basis of measures (qualitative or quantitative) considered to be valid and reliable, which compare the actual results of a programme with its anticipated results. Even where evaluation is concerned with assessing intangible situations, which are difficult to measure, it must, to be credible, be based on data gathered in a rigorous and objective manner.

    The first step in programme evaluation is to define the subject of the evaluation. To that end, a definition must be made of what a programme and what a subprogramme is.

    A programme is a coherent, organized and structured whole, composed of objectives, activities, means.

    An objective is the statement of a result to be achieved. Each general objective can be broken down into more precise statements, which then determine the activities, people and means necessary for the success of the programme.

    Objectives dictate the line to be followed and serve as markers by which the results and their impact can be assessed. Only if objectives are stated in a strict and unambiguous manner can a value judgement be made about the running of a programme.

    Stated more precisely, an objective is a concrete and realistic statement of a desired outcome, encompassing qualitative and quantitative indices (what?) limited in time (when?) and space (where?), and which must satisfy a specific need (why?) of particular individuals or of a particular group (who?).

    The activities, means and individuals concerned are the ingredients that give the programme its functional characteristics:

  • activities: what will be done to achieve the objectives;

  • means: all the resources and tools considered necessary to attain the objectives;

  • individuals: the persons involved at UNESCO Headquarters, such as directors and programme specialists, and in offices away from Headquarters, as well as non-governmental organizations (NGOs), and national partners such as National Commissions, permanent delegations, and so on.


  • These three elements are usually defined in the biennial work plans.

    NB : The concept of activity should not be confused with that of result; activity means whatever is done to bring about the desired change (e.g., training sessions), while the result refers to the change brought about (e.g., the observation that participants in training sessions have gained knowledge).


    THREE QUALITIES, SEVEN POINTS OF VIEW

    Programme evaluation makes it possible to ‘observe’ a programme and evaluate the balance among the following three qualities:

  • Quality desired : Objectives

  • Quality provided : Results

  • Quality perceived : Judgement


  • This observation can be carried out in an exhaustive way from seven points of view, which are like observation "windows":(a) pertinence (b) coherence (c) opportunity (d) impact (e) durability (f) effectiveness and (g) efficiency

    Each observation window gives rise to a particular set of questions. With the exception of a detailed programme evaluation such as may be carried out by a commission of inquiry, a single evaluation rarely explores all seven windows. It is up to the decision-maker to choose the windows to observe through and the programme elements to be evaluated.


    Pertinence

    Pertinence concerns the extent to which the programme objectives directly correspond to the needs to be met. In cases where needs have not been clearly identified, or where they have changed and may no longer be as pressing, a list of needs, in order of priority, must be drawn up. If the needs remain unchanged, the evaluation will determine how closely they are linked to the programme objectives and, if necessary, the objectives will be more clearly stated. The degree of pertinence should be described; it must never be expressed as a percentage.


    Opportunity

    A programme may be entirely pertinent with regard to needs that have to be met, but its opportunity must be assessed in relation to the circumstances in which it is being carried out.


    Impact

    Programme results themselves have consequences. Evaluating the impact means establishing to what extent a programme’s results are having a positive or negative effect on the needs the programme was designed to satisfy and are creating an environment favourable to the programme’s continuation.


    Durability

    Effectiveness is concerned with the quantity and quality of the programme’s immediate results, whereas durability is an evaluation of how long those results and their consequences will last. Durability is of particular significance when one is looking for medium- and long-term results.


    Effectiveness

    Evaluating effectiveness means judging the degree to which a programme is achieving its objectives. Effectiveness is judged by comparing the objectives with the results directly attributable to the programme, with regard to both quantity (e.g. number of persons trained in relation to the target number) and quality (e.g. examination results in comparison with the desired outcome).

    Effectiveness is always expressed in terms of a percentage since the value is obtained by relating two similar elements - expected results and actual results - to each other.


    Efficiency

    Efficiency is evaluated by answering one or other of the following three questions:

  • Are the best possible results being attained with the activities, means and participants available?

  • Could the results be improved by organizing or managing the activities, means, resources and participants differently?

  • Is it possible to reduce the quantity or quality of the activities, means, resources and participants without affecting the quality of the results?


  • Efficiency is not expressed as a percentage because the value is obtained by relating two different elements to each other - goods and services produced in comparison with resources used.


    Coherence

    Coherence concerns the degree to which the various elements (activities, means, resources, individuals) join together to form a whole aiming to achieve the stated objectives. Certain programmes may encompass a large number of activities and specific objectives. Owing to various constraints, including the need to save time and resources, the evaluation often has to be restricted to a limited number of aims or activities.


    COST OF EVALUATION

    Evaluation costs should be provided for when the budget for the programme is being estimated and should be treated as an integral part of programme design and implementation.

    Experience shows that between 0.5 and 1.0 per cent of the budget for a particular programme should be allocated to general evaluation needs. Considering that evaluation constitutes an investment, this is an infinitesimal percentage of the budget.

    Three Qualities, Seven Point of View
    Programme evaluation makes it possible to ‘observe’ a programme and evaluate the balance among the following three qualities:

  • Quality desired : Objectives

  • Quality provided : Results

  • Quality perceived : Judgement


  • This observation can be carried out in an exhaustive way from seven points of view: (a) pertinence (b) coherence (c) opportunity (d) impact (e) durability (f) effectiveness and (g) efficiency.

    The diagram below gives a good illustration.
     More
    An objective the statement of a result to be achieved
    The following table is a very simple tool for checking whether the statement of an objective responds, explicitly or implicitly, to five key questions. More
    Example of elements needed to attain an objective
    Table showing the elements needed to attain an objective (example). More
    Europe and North America Latin America and the Caribbean Africa Arab States Asia Pacific