Evaluation principles DAC Principles for Evaluation of Development Assistance

The evaluation environment - 1 Your political considerations (some aspects; are they important for us?) – How are evaluations looked upon in general? – Sensititvity of the subject? – Likely responses or lack of responses by govt Partner country environment – Involvement (in preparations); organisational set-up – Range of possible responses

The evaluation environment - 2 Inside your organisation – Evaluation policy What are evaluations expected to achieve? Who decides on evaluations? Rules for when and how evaluations are carried out Is there an evaluation dept or decentralised handling? Who owns the evaluations? Management response procedures?

Independence ORGANIZATIONAL INDEPENDENCE. Direct line of reporting between evaluation unit and the management or board of the institution. Evaluation unit located outside operational staff and line management function. Evaluation unit removed from political pressure and able to operate without fear of repercussions. Recruitment and promotion of staff based on merit. BEHAVIOURAL INDEPENDENCE. Ability and willingness of the evaluation unit to produce strong and uncompromising reports. Transparency in reporting. AVOIDANCE OF CONFLICTS OF INTEREST. Extent to which stakeholders are consulted in evaluations to ensure against evaluator bias. Extent to which there are procedures to prevent staff from getting involved in evaluations of activities where they or their close associates have, have had, or in the future may expect to have a substantial interest. PROTECTION FROM EXTERNAL INFLUENCE. Extent to which the evaluation unit is able to decide on the design, scope, timing and conduct of evaluations without undue interference. Extent to which the evaluation unit has control over staff hiring, firing and promotion. Extent to which the employment of evaluation staff is not jeopardized for reasons other than competency and need for evaluation services. Extent to which the evaluation unit can fulfil its obligations unimpeded by restrictions on funds. Extent to which the judgements of evaluators regarding the contents of reports is protected from overruling by external authorities.

Objectivity v. Insight – 1 External or internal? The distinction between external and internal evaluation is closely associated with notions of independence, impartiality and bias. An evaluation is not just any assessment of the merits of an activity, but one that aims to be as objective and impartial as possible. The requirement for objectivity and impartiality is always strong where evaluations serve a purpose of accountability, and it can be equally strong when learning and the creation of new knowledge is the purpose.

Objectivity v. Insight Expert or participatory?

a. Appraisal: A critical assessment of the potential value of an undertaking before a decision is made to implement it. b. Monitoring: Managements continuous examination of progress achieved during the implementation of an undertaking to track compliance with the plan and to take necessary decisions to improve performance. c. Review: The periodic or ad hoc often rapid assessments of the performance of an undertaking, that do not apply the duce process of evaluation. Reviews tend to emphasize operational issues. d. Inspection: A general examination that seeks to identify vulnerable areas and malfunctions and to propose corrective action. e. Investigation: A specific examination of a claim of wrongdoing and provision of evidence for eventual prosecution or disciplinary measures. f. Audit: An assessment of the adequacy of management controls to ensure the economical and efficient use of resources; the safeguarding of assets; the reliability of financial and other information; the compliance with regulations, rules and established policies; the effectiveness of risk management; and the adequacy of organizational structures, systems and processes. g. Research: A systematic examination designed to develop or contribute to knowledge. h. Internal management consulting: Consulting services to help managers implement changes that address organizational and managerial challenges and improve internal work processes.

The evaluation managers roles Promoter of evaluation use through evaluation acapcity building Evaluation broker, engaging all stakeholders in planning... Commissioner of evaluations Providing peer review and support to evaluators (PACT, South Africa)

From literature, managing evaluations is increasingly gaining recognition as an important practice which, until recently was invisible (Baizerman,M., & Compton, D.W. 2009, p8). Various roles managers should play in the evaluation process have been suggested including, as noted by Bell (2004, p.602) developing rational proposals, clarifying the evaluation mandate, monitoring interim progress and ensuring product quality and usefulness. Bell goes on to state that valid and useful evaluations depend as much on effective management as on elegant design (p.603). Rita Sonko-Najjemba, Addis Berhanu: Redefining the role of evaluation managers to foster improved evaluation quality and use. PACT South Africa, October 2012

However there seems to be varying perceptions about the role managers should play in managing contracts and the skills needed for the job (Newcomer K,E p.202). Dibella (1990, p177) suggests that the evaluation managers role should be to bridge the orientation of evaluators and the expectations of program managers as evaluation users. He proposes that: basically, the evaluation manager becomes a buffer between the self interests of the user, who may seek or hope for particular results, and the evaluator who may emphasize a research orientation rather than applicability or relevance of recommendations.

The environment 1 The evaluation manager The consultant The evaluation

The consultant The evaluation manager The evaluation

The immediate environment - Conceptualising Idea stage Checking against plan, strategy and policy Finding out about reasons why, and for what Most influential is probably your own organisation The evaluation manager The evaluation

The immediate environment – Forming up Terms of Reference Interested parties, stakeholders consulted Checking against budget Timing important Procurement options Formal decision on design & requirements The evaluation manager The evaluation Interested parties

The immediate environment – Procurement &Inception Procurement Options – Direct procurement (handpicking) – Tender, fixed price; on quality – Tender, both quality and price Interested parties, stakeholders consulted Checking against budget Timing important Procurement options Formal decision on design & requirements The consultant The evaluation manager The evaluation

Purpose of the DAC quality standards

DACs quality standards

Purpose, planning and design