Sección o Parte de un Documento 1995 es 382.3 B584L(58739) Corden, W. Max Coediciones NAFTA AMERICA LATINA LIBERALIZACION DEL INTERCAMBIO NEGOCIACIONES COMERCIALES TRATADOS ZONAS DE LIBRE COMERCIO FREE TRADE AREAS LATIN AMERICA TRADE LIBERALIZATION TRADE NEGOTIATIONS TREATIES NAFTA Una zona de libre comercio en el Hemisferio Occidental: posibles implicancias para América Latina Incluye Bibliografía En: La liberalización del comercio en el Hemisferio Occidental - Washington, DC : BID/CEPAL, 1995 - p. 13-40 Proyecto Apoyo al Proceso de Liberalización Comercial en el Hemisferio Occidental http://hdl.handle.net/11362/1510 2014-01-02T14:51:16Z 2014-01-02T14:51:16Z Made available in DSpace on 2014-01-02T14:51:16Z (GMT). No. of bitstreams: 0 Previous issue date: 1995 POLÍTICA COMERCIAL Y ACUERDOS COMERCIALES TRADE NEGOTIATIONS COMERCIO INTERNACIONAL E INTEGRACIÓN INTERNATIONAL TRADE AND INTEGRATION Texto March 2014 Development Account Project 10/11AH Improving poverty measurement: building national statistical capacity in Caribbean countries to estimate purchasing power parities DEVELOPMENT ACCOUNT PROJECT 10/11AH Improving poverty measurement: building national statistical capacity in Caribbean countries to estimate purchasing power parities Final evaluation report Prepared by: Richard M. Chiwara This report was prepared by Richard Chiwara, an external consultant, who led the evaluation. Mr Chiwara worked under the general guidance of Alejandro Torres Lépori, Chief of the Programme Planning and Evaluation Unit within the Programme Planning and Operations Division of the Economic Commission for Latin America and the Caribbean (ECLAC), and Nurit Bodemann-Ostow, Programme Officer of the same unit, who provided strategic and technical guidance, coordination, and logistical support. The evaluation also benefited from the assistance of Maria Victoria Labra, Research Assistant, Natalia Rodriguez, Team Assistant, and Alejandra Reyes, Programme Assistant, of the Programme Planning and Operations Division of ECLAC. The evaluation team of the Programme Planning and Evaluation Unit is grateful for the support provided by project partners at ECLAC and the Department of Economic and Social Affairs (DESA), all of whom were represented in the Evaluation Reference Group (ERG). Warm thanks go to the programme managers of the Statistics Division of ECLAC for their cooperation throughout the evaluation process and their assistance in the review of the report, in particular Pascual Gerstenfeld, Chief of Division, Salvador Marconi, Statistician, Giovanni Savio, Statistician, Claudia de Camino, Statistics Assistant and Alejandra Ovalle, Statistics Assistant. The team also extends its gratitude to the programme managers of the ECLAC subregional headquarters for the Caribbean, including Diane Quarless, Chief of Office, Dale Alexander, Computer Information Systems Asssistant and Sinovia Moonie, Statistics Assistant. In the same regard, the assistance provided by Romain Zivy, Economic Affairs Officer in the Office of the Deputy Executive Secretary of the Commission, is gratefully acknowledged. Lastly, the team would like to express its appreciation of the support provided by colleagues of the Capacity Development Office of DESA, including Martin Kraus, Programme Officer, Mikael Rosengren, Programme Officer, and Carlotta Tincatti, Evaluation Expert. All comments on the evaluation report by the Evaluation Reference Group and the evaluation team of the Programme Planning and Evaluation Unit were considered by the evaluator and duly addressed in the final text of the report, where deemed appropriate. The evaluator’s responses to all these comments have been documented in the evaluator’s response table. Moreover, as part of the follow-up to this evaluation, the ECLAC management response to the evaluation will be made publicly available. The views expressed in this report are those of the author and do not necessarily reflect the views of the Commission. Copyright © United Nations, March 2014. All rights reserved Printed at United Nations, Santiago, Chile 2014-38 3 CONTENTS Page Executive summary ........................................................................................................................................... I. Introduction ............................................................................................................................................ II. Evaluation methodology......................................................................................................................... III. Project description.................................................................................................................................. IV. Evaluation findings ................................................................................................................................ V. Lessons learned ...................................................................................................................................... VI. Conclusions ............................................................................................................................................ VII. Recommendations .................................................................................................................................. Annex 1 Documents reviewed..................................................................................................................... Annex 2 Individuals interviewed................................................................................................................. Annex 3.A Analysis of stakeholder responses to the electronic survey .......................................................... Annex 3.B Analysis of beneficiary responses to the electronic survey........................................................... Annex 4 Schedule of planned and implemented activities .......................................................................... Annex 5 Results-based performance matrix................................................................................................ Annex 6 Evaluation terms of reference ....................................................................................................... 5 6 9 10 15 29 30 31 33 34 35 36 41 42 43 Tables Table 1 Table 2 Table 3 Table 4 Table 5 Table 6 Table 7 Project results framework ............................................................................................................. Implementing partners .................................................................................................................. Attendance at project meetings and workshops by country.......................................................... Beneficiary assessment of effectiveness ....................................................................................... Activities planned and implemented............................................................................................. World Bank funding of project activities...................................................................................... Budget delivery (cumulative) ....................................................................................................... 14 15 18 20 26 27 28 Figures Figure 1 Figure 2 Figure 3 Figure 4 Stakeholders’ perceptions of the project’s relevance.................................................................... Beneficiary perceptions of project usefulness............................................................................... Mixed views on effectiveness among implementing partners ...................................................... Budget allocation by activity area................................................................................................. 17 19 20 27 Boxes Box 1 Box 2.A Box 2.B Box 3 Box 4 Box 5 What is evaluation?....................................................................................................................... The Bahamas ................................................................................................................................ Grenada......................................................................................................................................... The national accounts challenge ................................................................................................... Extracts from project document on implementation arrangements............................................... The role of CARICOM (Progress report: July 2010 to December 2012, p. 9) ............................. 7 21 21 23 24 25 Diagrams Diagram 1 Diagram 2 Diagram 3 Stakeholder map ........................................................................................................................... Project problem tree...................................................................................................................... Project sustainability..................................................................................................................... 10 13 28 4 ACRONYMS CARICOM CARTAC CDO CISP COICOP CPI DESA ECCB ECCU ECLAC ECOSOC ERG FGD FISIM GDP HBS ICP MDG(s) ME MORES NSO(s) OECD-DAC PIPS PPBME PPEU PPOD PPP RBM RSWP SCCS SEPD SNA SPD TAM(s) ToR TWG UNEG UNSC VAT Caribbean Community Caribbean Regional Technical Assistance Centre Capacity Development Office (DESA) Caribbean Integration Support Programme Classification of Individual Consumption According to Purpose Consumer price index Department of Economic and Social Affairs Eastern Caribbean Central Bank Eastern Caribbean Currency Union Economic Commission for Latin America and the Caribbean Economic and Social Council Evaluation Reference Group Focus group discussion Financial intermediation services indirectly measured Gross domestic product Household budget survey International Comparison Programme Millennium Development Goal(s) Monitoring and evaluation Model Report on Expenditure Statistics National statistical office(s) Organisation for Economic Co-operation and Development-Development Assistance Committee Price index processing system Programme Planning, the Programme Aspects of the Budget, the Monitoring of Implementation and the Methods of Evaluation Programme Planning and Evaluation Unit Programme Planning and Operations Division Purchasing power parity Results-based management Regional Statistical Work Programme Standing Committee of Caribbean Statisticians Statistics and Economic Projections Division System of National Accounts Structured product description Technical assistance mission(s) Terms of reference Technical working group United Nations Evaluation Group United Nations Statistical Commission Value added tax 5 EXECUTIVE SUMMARY Caribbean countries did not participate in previous International Comparison Programme (ICP) rounds, and they faced challenges in producing and reporting comprehensive and comparable price statistics and national accounts in accordance with the System of National Accounts 1993 (1993 SNA) framework. Project 10/11AH of the Economic Commission for Latin America and the Caribbean (ECLAC) responded to the request by the United Nations Statistical Commission (UNSC) to bridge the capacity gap of national statistical offices (NSOs) in the Caribbean in order (a) to ensure that the region was represented in the 2011 round of the ICP, (b) to improve the measurement of poverty in the region in accordance with internationally comparable methodologies and (c) to enable the estimation of purchasing power parity (PPP) by all Caribbean countries on a sustainable basis. The project was implemented from July 2010 to December 2012 with a budget of US$ 401,000 funded by the Development Account. Pursuant to the Regulations and Rules Governing Programme Planning, the Programme Aspects of the Budget, the Monitoring of Implementation and the Methods of Evaluation (ST/SGB/2000/8), the Programme Planning and Evaluation Unit (PPEU) of the ECLAC Programme Planning and Operations Division (PPOD) commissioned the evaluation to determine as systematically and objectively as possible the relevance and effectiveness of the project in the light of its objectives and expected accomplishments. The 2011 round of the ICP was led globally by the World Bank with ECLAC as the focal point for the Latin America and Caribbean region. Project 10/11AH contributed to overall achievement of the objective to “compile the purchasing power parity for the Caribbean based on ICP methodology”. The evaluation found that the project was focused on very pertinent issues and addressed a significant capacity gap in the Caribbean, and beneficiary countries also confirmed the relevance and usefulness of the project. Although there were vast disparities between participating countries in terms of their statistical and technical capacities as well as the status of their statistical infrastructure, they harmonized their consumer price indices (CPIs) through the use of a common classification of consumer goods and services. All 20 participating countries compiled their gross domestic product (GDP) by expenditure type as per the ICP requirements. The project established partnerships and collaboration between ECLAC, the Caribbean Community (CARICOM), the Caribbean Regional Technical Assistance Centre (CARTAC), the Eastern Caribbean Central Bank (ECCB) and the World Bank. However, the project document did not sufficiently address the issue of project governance and management, and this led to conflicting expectations among project stakeholders. Budget delivery was very satisfactory, with 99% of the allocated budget delivered by the end of the project. Over 77% of the total budget was allotted to substantive project activities including fellowships and grants (46%) and consultants and expert groups (31%). Four key lessons emerged from the project implementation, namely the need to (i) give equal attention to national accounts and price statistics, (ii) harmonize data collection protocols and methodologies, as well as primary data collection timeframes, (iii) assign clear roles and responsibilities to project partners and stakeholders and (iv) develop a clear sustainability plan and exit strategy. 6 On the basis of the evaluation findings and lessons learned, the following five recommendations were made. Recommendation 1: ECLAC should establish a project steering committee to provide strategic governance and oversight for its projects. Recommendation 2: ECLAC should develop technical capacity in its subregional office in Trinidad and Tobago for more effective support to Caribbean countries. Recommendation 3: ECLAC should develop a clear sustainability plan in the project design. Recommendation 4: ECLAC should conduct specific capacity evaluation at the end of each capacity development project. Recommendation 5: ECLAC should explore ways to enhance application of the project’s technical activities as a poverty measurement and policy support tool at the national and regional levels. I. INTRODUCTION 1.1. Evaluation context 1. The Development Account is a programme of the United Nations Secretariat that was originally established in 1997 and has since funded 256 projects for a total of US$ 152.9 million.1 In its latest approved biennium, the Development Account allocated US$ 28.4 million in funding for 52 projects implemented by 10 entities of the Executive Committee of Economic and Social Affairs.2 The project “Improving poverty measurement: building national statistical capacity in Caribbean countries to estimate purchasing power parities” was approved under the seventh tranche of the Development Account for the 2010‐2011 biennium, and was implemented by the Statistics Division and the subregional headquarters for the Caribbean of the Economic Commission for Latin America and the Caribbean (ECLAC). 2. The ECLAC mandate derives from Economic and Social Council (ECOSOC) resolution 106 (VI), whereby the Council established the Commission for the purpose of contributing to and coordinating action towards the economic and social development of the region and reinforcing economic relationships among the countries of the region as well as worldwide. In 1996, by virtue of ECLAC resolution 553(XXVI), the Commission was instructed, inter alia, to collaborate with member States on a comprehensive analysis of development processes geared to the design, monitoring and evaluation of public policies and the resulting provision of operational services in the fields of specialized information, advisory services, training and support for regional and international cooperation and coordination.3 3. The “Programme for Reform” presented by the Secretary-General to the General Assembly in 1997 shifted United Nations planning, budgeting and reporting from a focus on processes to a focus on results, “with the aim of shifting the United Nations programme budget from a system of input accounting to results-based accountability. […] The Secretariat would be held responsible for, and judged by, the 1 See [online] http://www.un.org/esa/devaccount. 2 DESA, the five United Nations regional commissions, UNCTAD, UNEP, UN-HABITAT and UNODC. 3 ECLAC Programme of Work 2014-2015. 7 extent to which the specified results are reached.”4 In subsequent reports, the Secretary-General further stressed the need for an improved monitoring and evaluation system to allow the Organization to better measure the impact of its work.5 The operating procedure for the evaluation function within the United Nations Secretariat was outlined in the Secretary-General’s bulletin entitled “Regulations and Rules Governing Programme Planning, the Programme Aspects of the Budget, the Monitoring of Implementation and the Methods of Evaluation” (ST/SGB/2000/8). 4. This evaluation was a discretionary internal evaluation managed by the Programme Planning and Evaluation Unit (PPEU) of the ECLAC Programme Planning and Operations Division (PPOD). The evaluation was undertaken as part of the general evaluation function to support and inform the decision‐making cycle at the United Nations Secretariat in general and ECLAC in particular and within the normative recommendations made by different oversight bodies endorsed by the General Assembly. 5. This report presents the findings and conclusions of the final evaluation undertaken by an independent evaluator over a period of 16 weeks from 1 August to 28 October 2013. The report has seven chapters. Chapter 1 introduces the evaluation, including its context and rationale, evaluation purpose and objectives and the scope of evaluation. Chapter 2 contains a discussion of the evaluation methodology and approach, including stakeholder analysis and limitations. Chapter 3 describes the project that was evaluated in detail. It includes a review of the development context in the Caribbean and the project’s logic model and results, monitoring and evaluation (RME) framework. The evaluation findings are presented in chapter 4, being based on the evaluation criteria for assessing the project’s relevance, implementation, effectiveness, efficiency and sustainability. Chapters 5, 6 and 7 contain the lessons learned, conclusions and recommendations, respectively. 1.2. Purpose and objectives of the evaluation Box 1 WHAT IS EVALUATION? “…determine as systematically and objectively as possible the relevance, effectiveness and impact of an ongoing or completed programme, project or policy in the light of its objectives and accomplishments.” Source: ECLAC Evaluation Policy and Strategy, p. 8. 6. The purpose of the evaluation (box 1) was to support and inform the decision‐making cycle at the United Nations Secretariat in general and ECLAC in particular. The evaluation was undertaken in accordance with General Assembly resolutions 54/236 of December 1999 and 54/474 of April 2000, which endorsed the Regulations and Rules Governing Programme Planning, the Programme Aspects of the Budget, the Monitoring of Implementation and the Methods of Evaluation (PPBME).6 In this context, the General Assembly requested that programmes should be evaluated on a regular, periodic basis, covering all areas of work under their purview. 4 United Nations General Assembly, “Renewing the United Nations: A Programme for Reform” (A/51/950), July 1997, p. 19. 5 United Nations General Assembly, “Strengthening of the United Nations: An agenda for further change” (A/57/387), September 2002. 6 ST/SGB/2000/8, articles II, IV and VII. 8 7. The specific objectives of the evaluation were to: (a) Analyse the design of the project as well as the relevance of its stated goals to the thematic area and region within which it operated, with particular emphasis on the needs of its beneficiaries. (b) Assess the project’s level of efficiency in implementing its activities, including its governance and management structures and use of resources. The evaluation also examined the level of coordination among implementing partners and the project’s strategic partnerships. (c) Take stock of the results obtained by the project, including the extent to which it achieved its objectives and, as far as possible, the initial impact attributable to it. (d) Assess the sustainability of the project, in the context of financial, political, institutional and technical dimensions. The evaluation also aimed to identify prospects for scale‐up and replication of its activities and outputs to other countries. (e) Identify best practices and lessons learned in the project implementation process that could serve to inform future development projects on the topic, undertaken either by ECLAC or by other United Nations entities. 1.3. Scope of the evaluation 8. The evaluation’s unit of analysis was the project “Improving poverty measurement: building national statistical capacity in Caribbean countries to estimate purchasing power parities”, including the design and implementation of planned activities as well as the results achieved. The evaluation covered the project cycle from July 2010 to December 2012 and included inputs from all stakeholders. 9. The evaluation focused on the project results to assess the extent to which expected accomplishments were realized, and took in the factors that contributed to the results, whether intended or unintended, including: how well the activities were designed and implemented; what outputs were delivered; how processes were managed; what monitoring systems were put in place; and how the project interacted with national partners and key stakeholders. Project results were assessed on the basis of the following definitions:7 • Output. A final product or service delivered by a programme or project to end users, such as reports and publications, which a project is expected to produce in order to achieve its expected accomplishments and objectives. • Outcome. The measurable accomplishment or result (intended or unintended, positive or negative) of a programme or project. In ECLAC usage, “outcome” is synonymous with accomplishment and result. • Impact. The overall effect of accomplishing specific results. Impact is the longer-term or ultimate effect attributable to a programme or project, in contrast with an expected accomplishment and output, which are geared to the biennial time frame. 7 ECLAC Evaluation Policy and Strategy, October 2013. 9 II. EVALUATION METHODOLOGY 2.1. Overall approach 10. The evaluation methodology was agreed with the commissioners of the evaluation on the basis of the inception report drafted by the evaluator to outline the evaluation plan, approach and schedule of deliverables. The following four-step approach was adopted: (1) Secondary data collection. Desk review of official project documents, including annual progress reports and plans. The list of documents reviewed is in annex 1 to this report. (2) Primary data collection. A data collection mission was fielded to Santiago, Chile, from 9 to 13 September 2013, coinciding with a meeting of project beneficiaries. The evaluator undertook individual and telephone interviews with 32 individuals, including project managers, partners and beneficiaries. Four countries (Bahamas, Grenada, Saint Kitts and Nevis and Saint Vincent and the Grenadines) were selected for in-depth case studies. The list of individuals interviewed is in annex 2 to this report. Two electronic surveys were also administered to implementing partners and beneficiaries. (3) Data analysis. Quantitative and qualitative analysis of the data was undertaken to extract information linked to the evaluation criteria. The response rate for the electronic surveys was 33% (implementing partners 36% and beneficiaries 32%). The analysis of survey results is in annex 3 to this report. Preliminary findings were presented to the evaluation commissioners and Evaluation Reference Group (ERG) for review and comment. This report incorporates their comments. (4) Draft report and final report. A draft report of the evaluation findings, lessons learned and recommendations was submitted to the evaluation commissioners and shared with the ERG for comments. This report incorporates those comments. 2.2. Stakeholder mapping 11. The original project concept note had 7 participating member countries, a number that was subsequently increased to 20 following recommendations from the Caribbean Community (CARICOM) and other partners. 12. Each of the participating member countries appointed two focal persons, one for national accounts statistics and another for price statistics. Other stakeholders included the project implementing partners and facilitators, project beneficiaries and strategic partners (diagram 1). 10 Implementing partners - ECLAC (HQ and subregional) Diagram 1 STAKEHOLDER MAP Participating countries Facilitators - CARICOM - CARTAC - ECCB Other United Nations system partners - World Bank Donor/Coordinator - Development Account - DESA CDO Focal points for national accounts Focal points for price statistics Project beneficiaries - National statistical offices 2.3. Limitations 13. The interviews with beneficiaries were conducted at the back of a stakeholder meeting in Santiago. For those countries that had separate focal points for national accounts and price statistics, the evaluator was able to meet only one of them, as most of the beneficiary countries were represented by one individual at the meeting. 14. There was a low response rate (33%) to the electronic survey, and therefore the findings of the beneficiary survey should be seen as indicative rather than representative of the views of all stakeholders and participating countries.8 However, the information was triangulated with document reviews and other individual interviews to eliminate any inconsistencies in the report. III. PROJECT DESCRIPTION 3.1. The development context in the Caribbean 15. The first Millennium Development Goal (MDG1) of eradicating extreme poverty and hunger defined its first target (1A) as, “to halve, by 2015, the proportion of people living on less than one dollar a day, measured at purchasing power parity (PPP)”. It was clear, therefore, that the content and accuracy of the discourse on the achievement of MDG1 depended heavily on the accuracy and harmonization of global and regional PPP estimates. 16. The Caribbean did not have many statistical indicators on poverty, and the data that were available were not always fully comparable across countries. This lack of information hampered public policy and hindered access to international funding for poverty eradication programmes and projects. One of the major reasons for this situation was that the Caribbean did not participate in the 2005 round of the International Comparison Programme (ICP).9 8 Five stakeholders responded out of a total of 14 surveyed, and 25 beneficiaries responded out of a total of 78 surveyed. 9 The ICP is a global statistical initiative for estimating purchasing power parities (PPPs) that allow for the correct comparison of prices, expenditure, income and other monetary aggregates across countries by converting them into a single unit of measurement. 11 17. Through its resolution 2006/6,10 the United Nations Economic and Social Council (ECOSOC) expressed concern that (a) there still existed in many countries a lack of adequate data to (i) assess national trends in the context of monitoring progress towards the realization of all the internationally agreed development goals, including the MDGs, and (ii) inform and monitor the implementation of national development policies and strategies; and (b) that in many countries where data did exist, there was a lack of capacity for effective use of the data. Owing to their specific geographical and structural characteristics, many of the countries in the Caribbean faced challenges in establishing well-developed national statistical systems, particularly when it came to producing and reporting comprehensive and comparable price statistics and national accounts in accordance with the System of National Accounts 1993 (1993 SNA) framework. 18. A significant number of Caribbean countries did not have complete sets of national accounts; for example, some of the countries did not compile complete expenditure-based estimates of gross domestic product (GDP) and very few of them were able to produce supply-use tables. It was also noted that some of the countries were using production-based estimates, but expenditure on GDP was usually incomplete. The production estimate of GDP provided a benchmark and the difference between that estimate and the actual recorded expenditures were treated as a balancing item. In addition, many of the countries that were able to compile GDP by expenditure did not produce expenditure data for all basic headings. 19. With regard to the SNA, the recommended accounting system was the accruals basis, whereby transactions were recorded at the time that the economic value was created, transformed, exchanged, transferred or extinguished. However, many Caribbean countries recorded government revenues and expenditures using cash accounting whereby transactions were recorded only at the time of occurrence. This created a challenge for the calculation of internally consistent purchasing power parity (PPP) in the region because of the divergence of the reference period of price surveys and the transaction period. 3.2. Regional responses and strategies 20. By definition, an individual country cannot produce PPP comparisons with other countries by itself. The ICP was a joint effort whereby countries worked together to determine what would be priced and when. The essence of the ICP was the comparability of results between countries, strict adherence to time schedules, and a common understanding of data-sharing and confidentiality requirements. No other statistical programme required so much cooperation between national, regional and international organizations. 21. Some important lessons were generated from previous ICP rounds. At the beginning of the 2005 round it was decided that the Global Office would provide the framework for establishing how the products to be priced for the consumption components of GDP would be selected and defined. The responsibility for selection of the products and their specifications was given to each region and its participating countries. Individual regions therefore developed their product specifications, and the respective representatives of national statistical offices (NSOs) jointly agreed on the list of products to be priced and their specifications. The approach enabled participating countries to review each other’s prices and ensure that the most appropriate products were being priced using a common framework. 10 “Strengthening statistical capacity”, E/2006/INF/2/Add.1. 12 22. The 2005 round also indicated the need to establish a data quality framework that provided clear standards for estimating PPPs. It was likewise important for the relationship between regional coordination and national participants to be based on formal institutional arrangements such as memoranda of understanding (MOU) and to secure financial support early enough to develop an effective plan of activities that included reviewing and harmonizing price indices and national accounts programmes, as well as developing country-specific execution plans that took into account their respective capabilities and statistical infrastructures. 23. In this connection, CARICOM undertook projects to build capacities in its member States, including harmonization of statistics and implementation of the 1993 SNA. The Caribbean Regional Technical Assistance Centre (CARTAC) was established by the CARICOM Council of Ministers of Finance and Planning to provide technical assistance and training in core areas of economic and financial management at the request of member countries. Part of its work programme in statistics included technical assistance with national accounts and price statistics. 24. Caribbean member countries also collaborated with regional and international partners within the framework of the economic statistics programme under the Barbados Action Plan. At its thirty-third meeting, the Standing Committee of Caribbean Statisticians (SCCS) resolved to form a Technical Working Group (TWG) on National Accounts/Short Term Indicators, which was formally established at the Economic Statistics Meeting held in Guyana in April 2009. The mandate of the TWG was to assist member countries with the implementation of the Barbados Economic Statistics Action Plan within the context of interventions proposed under the Caribbean Integration Support Programme (CISP) and the Regional Statistical Work Programme (RSWP). 25. The ICP therefore provided a framework for estimating PPPs, thereby enabling accurate comparison of prices, expenditure, income and other monetary aggregates across countries by converting them into a single unit of measurement. However, since the Caribbean did not participate in the 2005 round, the United Nations Statistical Commission (UNSC), at its forty-first session, urged ECLAC to ensure that as many Caribbean countries as possible were involved in the 2011 round of the International Comparison Programme. 26. This project responded to the request by the UNSC and aimed to bridge the capacity gap affecting NSOs in the Caribbean in order to (a) ensure that the region was represented in the 2011 round of the ICP, (b) improve the measurement of poverty in the region in accordance with internationally comparable methodologies and (c) enable the estimation of PPPs by all Caribbean countries on a sustainable basis. 3.3. Project strategy and theory of change 27. The project strategy was to combine advocacy for increased coordination and cooperation among NSOs, and to enhance their technical capacities. On their part, participating countries were to designate two national experts (one in national accounts and one in price statistics) as focal points with sufficient decision-making authority to implement technical recommendations derived from project interventions. 28. The strategy aimed to provide an enabling political and legal environment for NSOs, particularly with regard to adequate budget support from their countries, in order to turn around the downward-facing arrow (diagram 2) by addressing the negative challenges and turning them into positive statements. 13 Diagram 2 PROJECT PROBLEM TREE Insufficient coordination and cooperation among NSOs Limited technical skills and insufficient resources at NSOs Lack of harmonized price data collection systems and survey frameworks Incomplete set of national accounts to estimate GDP values at basic heading (1) Difficulty in producing comparable statistics (2) Inability to report as per ICP guidelines (3) Lack of PPPs to assess trends and progress towards MDGs 3.4. Results framework 29. The 2011 ICP round was led by the World Bank as a worldwide statistical operation to produce internationally comparable price and volume measures for GDP and its component expenditures based on purchasing power parities (PPPs). To calculate the PPPs, the ICP holds surveys every five years to collect price and expenditure data for the whole range of final goods and services that comprise GDP, including consumer goods and services, government services and capital goods. 30. The ICP is organized by region. There are six regions, and responsibility for the ICP within each is shared between national and regional agencies. ECLAC was the focal point for the Latin America and Caribbean region. Individual national statistical offices carry out data collection in their respective countries, with regional agencies providing guidance and coordinating data collection and data validation. The regional agencies also put together and finalize the regional comparisons. Responsibility for ensuring that the regional comparisons can be combined in a world comparison and then combining them rests with the Global Office. 31. Within the overall framework of the 2011 round led by the World Bank, ECLAC was therefore the coordinator for ICP activities in the Latin American and Caribbean region. However, as Caribbean countries had not participated in the previous 2005 round, they faced greater challenges in adapting to ICP processes and methodologies, and it was in this context that the project was developed to build national capacities so that Caribbean countries could participate in the 2011 round. The overall project objective was “to bridge the capacity gap of national statistical offices in 20 Caribbean countries to establish common data collection protocols related to purchasing power parity and to produce and analyse national accounts in order to increase poverty information in the Caribbean”. 32. To achieve this objective, the project sought to contribute to four expected accomplishments (table 1). 14 Table 1 PROJECT RESULTS FRAMEWORK Expected accomplishment (outcome) EA1. Adoption by Caribbean countries of common data collection protocols, definitions and classifications for a set of statistics on prices and National Accounts that constitute a basis for estimating purchasing power parities according to the guidelines of the International Comparison Programme (ICP). EA2. Strengthened technical capacity to produce and analyse representative and internationally comparable consumer price indices (CPIs) according to international classifications, standards and recommendations. EA3. Strengthened technical capacity to produce and analyse National Accounts by components of expenditure in line with the System of National Accounts 1993 (SNA 1993) and related international classifications, standards and recommendations. EA4. Strengthened regional network of experts and practitioners and increased interchange of experiences, best practices and methodologies on the production and use of price statistics and National Accounts. Indicators of achievement IA 1.1. Number of countries having defined a core list (or basket) of final consumption products and services for the computation of harmonized CPIs at the basic heading level and shared its results with all participating countries by considering the project’s recommendations. IA 1.2. Number of countries having developed a diagnosis of the structure of GDP by type of expenditure according to ICP requirements to develop weights for the calculation of PPPs. IA 1.3. Number of countries using a common classification of goods and services correlated with the classification of household final consumption expenditure in GDP. Number of countries having adopted an implementation strategy to achieve the same result. IA 2.1. Number of countries having adopted conceptual and methodological improvements in the compilation of their CPIs that are in line with international standards and the project’s recommendations. IA 3.1. Number of countries having adopted conceptual and methodological improvements in the compilation of national accounts that are in line with the SNA 1993. IA 4.1. Number of countries taking part in an interactive network of technicians responsible for CPI and National Accounts of the Caribbean countries. IA 4.2. Number of participants reporting benefits from the network’s activities and exchange of experiences to improve their production and use of price statistics and national accounts. Source: Project document. 3.5. Project implementation arrangements 33. The project was implemented in 20 countries: (1) Antigua and Barbuda, (2) the Bahamas, (3) Barbados, (4) Belize, (5) Dominica, (6) Grenada, (7) Guyana, (8) Jamaica, (9) Montserrat, (10) Saint Lucia, (11) Saint Kitts and Nevis, (12) Saint Vincent and the Grenadines, (13) Suriname, (14) Trinidad and Tobago, (15) Anguilla, (16) Bermuda, (17) British Virgin Islands, (18) Cayman Islands, (19) Turks and Caicos Islands, and (20) Aruba. Initially, the plan was for the project to be implemented in 7 countries, but this later increased to 20 following a recommendation by the Caribbean Community (CARICOM) and other strategic partners. 34. The project had a total allocated budget of US$ 401,000 delivered over a period of two and a half years from July 2010 to December 2012, one year ahead of the planned completion date of December 2013. Table 2 below shows the key implementing partners and their respective roles and responsibilities. 15 Table 2 IMPLEMENTING PARTNERS Implementing agency Statistics and Economic Projections Division of ECLAC ECLAC subregional headquarters in Trinidad and Tobago CARICOM Caribbean Regional Technical Assistance Centre (CARTAC) Eastern Caribbean Central Bank (ECCB) Roles and responsibilities Overall technical coordination and execution of project activities Coordination of logistics related to technical assistance missions (TAMs), workshops and meetings Technical expertise in national accounts and price statistics Technical support Technical support Source: Section 7 of project document, “Implementation arrangements”. IV. EVALUATION FINDINGS 4.1. Relevance 35. This section provides an assessment of the extent to which the intervention was suited to the priorities, policies and needs of key stakeholders, including the beneficiary countries, implementing partners and donors. Finding 1. The project was highly relevant to the national priorities of member countries and well aligned to the institutional mandates of their respective national statistical offices 36. At the time of project conceptualization in 2010, it was generally thought that only a limited number of countries would have the ability or be in a position to participate in the 2011 ICP round, because they had fairly advanced statistical infrastructure or because they had the financial and technical capacity. However, following the first round of consultative visits to Barbados, Guyana and Trinidad and Tobago, it became evident that no Caribbean country should or could be excluded from the 2011 ICP round on account of lack of resources or preparedness. By its very nature, the ICP is inclusive, and leaving out some of the countries would therefore bias regional results, which would serve no useful purpose. 37. It was further recognized that Caribbean regional price levels were fairly high, partly because most goods were imported, so that regional and country GDP per capita was significantly lower when measured on a PPP basis. It was therefore critical for all countries to participate in the ICP round in order to identify any differences in treatment and coverage of national accounts between countries, and to propose ways of overcoming these differences. Price statistics 38. Prior to the project, most countries only produced a consumer price index (CPI), and in most cases the products in the basket were not based on the Classification of Individual Consumption According to Purpose (COICOP); and although the weights were based on a household budget survey (HBS), in some cases this was also outdated. In addition, the number of products in the basket and price quotes was very small, amounting to only between 200 and 400. Very few countries were able to produce 16 a list of specifications, and these sometimes varied by outlet. The heads of the CPI sections usually ignored the variance of products in outlets (for example, white or brown rice) and relied on the discretion of price collectors to always price the same one.11 39. The country assessments also revealed that even when a list was available, the coding to COICOP basic headings was often deficient, and since the specifications were also vague, they did not match the global core list. In Saint Lucia, for example, the base years were outdated, nomenclatures did not follow the COICOP and price indices seemed to move very little, which was an indication that few prices were collected on a regular basis. Although CARTAC had been working with member countries of the Eastern Caribbean Currency Union (ECCU)12 to rebase their CPIs, move to the COICOP and implement the Price Index Processing System (PIPS), the ECCU had not yet reached a harmonized CPI. National accounts statistics 40. The assessments also found that all the countries estimated GDP in terms of production, and demand side estimates were much weaker. With the exceptions of Barbados and Jamaica, household expenditure was estimated residually.13 Only a very few countries allocated financial intermediation services indirectly measured (FISIM) to final demand. The measurement of accommodation rents was also very weak, in terms of both prices and national accounts. Most member countries did not know how to estimate rent properly or did not have the resources to collect price observations, and the situation was further complicated by the huge quality gap in average housing between the countries (with Bahamas at the upper end, for example, and Guyana at the lower end), and it was difficult to determine whether rent statistics should be captured through prices or through quantity indicators. 41. Previously, only a very few NSOs had prices for some building materials and wage rates, and none had price indices for construction machinery and equipment. In addition, most of the countries did not deflate their GDP expenditure estimates. As a result, they had no experience in collecting the type of prices that the ICP requires for these two aggregates. For machinery and equipment, some countries used the United Kingdom system (Barbados), while others used the United States and Japanese systems (Jamaica). Due to their size, some of the countries did not have “local” importers of machinery and equipment other than motor vehicles. 42. Clearly, therefore, the ICP round was focused on very pertinent issues and addressed a significant capacity gap in the Caribbean. A majority of the stakeholders who responded to the electronic survey indicated that the project was highly relevant (figure 1).14 A majority of beneficiaries rated the project as very relevant to their poverty measurement priorities. Those rating the project as being of little or no relevance (16%) observed that there was insufficient explanation of how the data were to be used. 11 Gervais, G. (2011), “Mission report: Assessment of national accounts and price statistics of selected Caribbean countries for the 2011 ICP round”. 12 The ECCU countries are Anguilla, Antigua and Barbuda, Dominica, Grenada, Montserrat, Saint Kitts and Nevis, Saint Lucia and Saint Vincent and the Grenadines. 13 Ibid. 14 The stakeholders included implementing partners and facilitators: ECLAC, CARTAC, CARICOM, the World Bank and ECCB. 17 Figure 1 STAKEHOLDERS’ PERCEPTIONS OF THE PROJECT’S RELEVANCE Project had synergy with other initiatives 67% Project responded to capacity gaps 100% Project was aligned to institutional mandate 100% Project was relevant to national priorities 100% 0% 20% 40% Yes No Don’t know 60% 33% 80% 100% Finding 2. The project was appropriately aligned to the United Nations programme of work for 2010-2011, and in particular the mandates of ECLAC and the Development Account 43. The 2010-2011 Strategic Framework and Programme of Work of the General Assembly (A/63/6/Rev.1) gave ECLAC a mandate for its economic and social development programme in Latin America and the Caribbean. The project objective and activities were consistent with the objectives of the “Statistics” and “Subregional activities in the Caribbean” subprogrammes, which were: • to foster the use and incorporation of accurate and timely statistical information in economic and social policies in countries of the region (Subprogramme 10), and • to promote and further strengthen the development process within the Caribbean and enhance the subregion’s cooperation with Latin American countries (Subprogramme 12). 44. In particular, the project’s objective was in line with the following expected accomplishments of these subprogrammes: Subprogramme 10 Expected accomplishments (a) Progress in the implementation of the strategic plan 2005-2015 of the Statistical Conference of the Americas, and (b) Increased technical capacity of ECLAC member States to monitor economic, social and environmental trends and to formulate evidence-based policies. Subprogramme 12 Expected accomplishment (a) Strengthened capacity of policymakers and other ECLAC stakeholders in the subregion to formulate and implement economic and social development measures and improve integration in the Caribbean as well as between the Caribbean and the wider Latin American region. 45. As for the Development Account, its projects aim at achieving development impact by building up the socioeconomic capacity of developing countries through collaboration at the national, subregional, regional and interregional levels by providing a mechanism for promoting the exchange and transfer of skills, knowledge and good practices among target countries within and between different geographic 18 regions. The project objective was aligned with the first three of the five thematic clusters into which projects funded by the Development Account fall: (i) Statistics, (ii) Governance and institution building, (iii) Trade, economy and finance, (iv) Social development and (v) Sustainable development, environment and natural resources. Finding 3. Project beneficiaries found the issues and recommendations of the project very useful and relevant to their work 46. The evaluation noted that member countries were fully committed to participating in the ICP 2011 round. The NSOs had designated two national experts, one for national accounts and one for price statistics, as focal points to implement technical recommendations derived from workshops and working group meetings. The NSO heads who participated in the advocacy high-level meeting in Barbados in July 2010 and the national experts who attended the technical workshop in the Bahamas in November 2010 expressed a strong commitment to participating in all the activities required to enable their countries to be represented in the 2011 round of the ICP. This commitment was evident from attendance at project activities, including meetings and workshops (table 3). However, some of the countries faced staffing constraints and had one focal person for both national accounts and price statistics. Table 3 ATTENDANCE AT PROJECT MEETINGS AND WORKSHOPS BY COUNTRY Meeting/Workshop High Level Meeting of Heads of NSOs Technical Meeting on National Accounts and Price Statistics Working Group Meeting on Price Statistics (3 groups) Expert Group: National Accounts Working Group Meeting on Price Statistics Working Group Meeting on National Accounts Expert Group: National Accounts and Price Statistics Expert Group Meeting on Price Statistics Expert Group Meeting on Price Statistics Venue Barbados Bahamas Trinidad Trinidad Trinidad Trinidad Saint Kitts Aruba Chile Date July 2010 Nov. 2010 Mar. 2011 Sept. 2011 Oct. 2011 Oct. 2011 Mar. 2012 Aug. 2012 Dec. 2012 Attendance 15 countries 18 countries 19 countries 19 countries 17 countries 19 countries 21 countries 18 countries 21 countries Source: Minutes of meetings and workshops. 47. At the high level meeting in Barbados in 2010, participants expressed concern about funding, but agreed to make every effort possible to move forward with implementation of the ICP. They requested more specific details on data compilation requirements in order to be able to include these activities in their budget planning and to assess additional country needs in terms of staffing, training and technical assistance.15 48. Project beneficiaries confirmed the relevance and usefulness of the project in their response to the electronic survey (figure 2). 15 High Level Meeting to Launch the 2011 Round of the ICP in the Caribbean, Barbados, 24 July 2010. 19 Figure 2 BENEFICIARY PERCEPTIONS OF PROJECT USEFULNESS Contributed to increased knowledge No response (39%) A great deal (30%) Scope and quality of project activities Excellent (17%) No response (43%) No at all (1%) Not much (8%) Som ewhat (22%) Very poor (0%) Poor (1%) Average (6%) Good (33%) 4.2. Effectiveness 49. This section provides an assessment of the project’s performance against the planned objective and its contribution to the overall project objective, which was “to bridge the capacity gap of national statistical offices in 20 Caribbean countries, to establish common data collection protocols related to purchasing power parity and to produce and analyse National Accounts in order to increase poverty information in the Caribbean” (annex 5). Finding 4. Through effective advocacy, the number of beneficiary countries was increased from an initial 7 to 20 countries 50. When it was initially conceptualized in early 2010, it was thought that only a limited number of countries would have the ability or be in a position to participate in the 2011 ICP round. However, after the first round of technical assessment missions, ECLAC realized that excluding some countries from participating was untenable, and leaving out some countries for lack of resources or preparedness, typically the poorest ones (Belize and Guyana), would bias regional results. 51. The project strengthened its advocacy work in the region, which included developing metadata questionnaires to identify differences in treatment and coverage of national accounts between countries and proposing ways to overcome these differences. ECLAC also asked to examine the national accounts of the member States of the Eastern Caribbean Currency Union (ECCU) in order to find ways in which they could be included in the ICP round. As a result of these efforts, the number of participating countries had increased to 20 by the end of the project in December 2012. Finding 5. The overall objective of compiling purchasing power parities for the Caribbean on the basis of the ICP methodology was successfully achieved 52. The World Bank had a mandate to lead the 2011 round of the ICP, with ECLAC as the focal point for the Latin America and Caribbean (LAC) region. The project (10/11AH) contributed to overall achievement of the objective to “compile the purchasing power parity for the Caribbean based on ICP methodology”. Although there were vast disparities between participating countries in terms of their statistical and technical capacities as well as the status of their statistical infrastructure, they harmonized 20 their CPIs through use of a common classification of consumer goods and services. All 20 participating countries compiled their GDP by type of expenditure as per the ICP requirements. 53. Going by the responses to the electronic surveys, there was general agreement among both implementing partners and project beneficiaries that the project achieved a high level of success in terms of its expected accomplishments. With regard to project beneficiaries, 62% agreed that participation in the workshops and seminars had helped their institution to adopt common data collection protocols, definitions and classifications for a set of statistics on prices and national accounts, and 22% partially agreed (table 4). Table 4 BENEFICIARY ASSESSMENT OF EFFECTIVENESS Please indicate below your level of agreement or disagreement with the following statements: Activity Totally Agree Agree Partially agree Disagre e Totally disagree NR Number of responses Project strengthened data collection for 5 3 5 2 CPI and national accounts 13 19 Project strengthened technical capacity 5 7 3 0 for CPI 03 18 Project strengthened technical capacity 4 8 4 0 for national accounts statistics 03 19 Project strengthened regional network 7 7 4 0 of experts 01 19 54. However, there appeared to be mixed views among project stakeholders regarding the project’s effectiveness in (i) contributing to the development of concrete policies to strengthen poverty measurement in participating countries, (ii) developing political will among participating countries to follow through with the project’s processes and results and (iii) addressing other capacity issues related to sustainability of the project’s processes and results, including financial and technical capacities (figure 3).16 Figure 3 MIXED VIEWS ON EFFECTIVENESS AMONG IMPLEMENTING PARTNERS Contribution to concrete policies No (20%) No response (0%) Political will to follow through No (20%) No response (0%) Financial/technical capacities to continue No response (20%) No (0%) Yes (80%) Yes (80%) Yes (80%) 16 With only five implementing partners responding, the differences of opinion are only indicative and may not be statistically significant. 21 55. Some of the participating countries’ focal points observed that there was insufficient awareness of the project’s benefits at the highest decision-making levels, which could potentially affect issues related to sustainability. Participating countries were not all at the same level of capacity and completion of the ICP processes. Some of the countries were more advanced than others, depending on their specific circumstances in terms of financial resources, institutional structures and staffing constraints. The mini case studies for the Bahamas and Grenada illustrate this point (box 2). Box 2.A THE BAHAMAS With a population of approximately 350,000, the Bahamas is made up of some 700 islands of which about 30 are inhabited. About 80% of the population live on two of the islands, Nassau (70%) and Grand Bahamas (10%). Two sections of the National Statistics Office were involved in the project. The National Accounts Section has a staff of 8, and the Consumer Prices Section has 9 staff members of whom 6 are based in Nassau and 3 in Grand Bahamas. The country undertook its last Household Budget Survey (HBS) in 2006 and its last census in 2010, and also undertakes biennial Labour Force Surveys in May and November each year. The Government was implementing a Social Safety Net Programme, and had therefore undertaken an HBS in 2012, but the results were yet to be published. From the above information, it was clear that the statistical infrastructure and system in the Bahamas was quite advanced by comparison with other participating countries. The Bahamas was already producing the CPI based on the classification of individual consumption according to purpose (COICOP), which is in line with the International Comparison Programme (ICP) methodology. The national consumer basket had about 125 products with close to 4,000 varieties. Although the regional consumer basket had more products, the country did not face any significant challenges in identifying the products and collecting the price data. The Bahamas independently carried out the price computations and was able to submit analysed price data to ECLAC. On the national accounts side, the Bahamas was already using the production and expenditure approach in the compilation of gross domestic product (GDP) and had already produced its supply use tables. The country experienced some slight challenges in that some of the headings required under the ICP methodology were different, but the data were largely available. It was noted that the ICP basic headings had been successfully adopted and would continue to be used in the future. Box 2.B GRENADA Grenada has a population of about 110,000 and consists of seven islands, of which Grenada is the largest, with a majority of the population. Grenada is a member of the Eastern Caribbean Currency Union (ECCU), whose common central bank is the Eastern Caribbean Central Bank (ECCB). The National Accounts Section of the Statistical Office has four officers, three of them permanent staff and one temporary. Section responsibilities include national accounts (GDP), balance of payments (BoP), the CPI and other economic indicators. The last HBS was undertaken from October 2007 to February 2008. The country experienced challenges in complying with the ICP processes. The national consumer basket had +/- 200 products with about 3,000 varieties. Data collection for the larger regional consumer basket was therefore a challenge. The ICP specifications were quite strict, and much more time and human resources were therefore required to identify the specific products. There was no budget to hire additional staff and they had to resort to working outside office hours, but policy did not provide for overtime pay for civil servants. On the national accounts, they only used production accounts for GDP and it was a challenge to comply with the ICP requirement for expenditure-side GDP data. They had to restructure government expenditure because the Government uses the cash accounting method, whereas ICP required accrual accounting. At the time of drafting, they were still working on the basic headings with the expectation of completion by mid-2014. The office did not have supply use tables and was now in the process of developing them. Some of the specifications for the basic headings were new to them, e.g., rent, which had to be measured at the price per square metre and take into account such issues as the age of the building, etc. The focal persons noted that the preparation time was too short, with collection of price data scheduled for the last quarter of 2011. In addition, there were only 3 workshops for national accounts, which was insufficient for the ICP processes to be adequately prepared for. 22 Finding 6. The core list of consumer products for the Caribbean region was developed and adopted by all participating countries 56. The project undertook nine technical assistance missions (TAMs) on prices and national accounts. Seven additional TAMs were undertaken with funding from the World Bank. As a result, the 20 participating countries defined and adopted a common list of consumer products and services (regional basket) for the computation of harmonized CPIs at the basic heading level. Ten of the countries initiated price data collection in the last quarter of 2011 using the methodological recommendations of the project, but the majority of the countries started data collection in the first quarter of 2012 using the tool kit developed by the project. 57. Follow-up workshops were undertaken in 2012 to validate price statistics and national accounts data. The first of these workshops was held in Saint Kitts and Nevis in March 2012 to revise, analyse and validate the price data from the 10 countries that had collected those data in the last quarter of 2011. On the basis of the lessons emerging from the workshop, additional workshops were planned and undertaken, including an additional workshop on national accounts to further assist the countries with the Model Report on Expenditure Statistics (MORES). This enables the countries to correct any errors in their data, as well as establishing the guidelines for strengthening the price statistics compilation process. Finding 7. Some participating countries faced challenges in completing their diagnosis of the structure of GDP by expenditure in accordance with the ICP methodology 58. A survey was undertaken in the last quarter of 2010 to determine the methodologies employed by Caribbean countries in compiling GDP by expenditure. However, citing staff and time constraints, national teams were unable to provide detailed responses to the questionnaire. The project then engaged a consultant to carry out the assessments in seven countries: the Bahamas, Barbados, Belize, Guyana, Jamaica, Saint Lucia and Trinidad and Tobago.17 In March 2011, when the outcomes of the assessment were presented, participating countries agreed on the methodological adjustments required and established a timetable to complete compilation of the final consumption expenditure component of GDP for the 2011 benchmark year. The project fielded another assessment for member countries of the ECCU in August 2011.18 59. By the end of 2011, 10 of the participating countries had adopted conceptual and methodological improvements in the compilation of national accounts in line with the 1993 SNA. The improvements were related to the disaggregation of GDP by expenditure based on the Model Report on Expenditure Statistics (MORES), and were further verified against the metadata information submitted to ECLAC through the Data Quality Assessment Framework and the Exhaustiveness Questionnaire. By the end of 2012, an additional nine countries had adopted the conceptual and methodological improvements. 60. However, additional support was still required to improve the area of national accounts in line with international standards (box 3). Indeed, project beneficiaries noted during interviews that they required further capacity building and technical support in that area. They also observed that only three workshops had been dedicated to national accounts during the project lifespan, which was insufficient to develop the capacities required to fully comply with the ICP methodology. The project management team 17 Gervais, G. (2011), “Mission report: Assessment of national accounts and price statistics of selected Caribbean Countries for the 2011 ICP round”. 18 Redeby, J. (2011), “Mission report: Assessment of the national accounts for the 2011 ICP round for selected members of the Eastern Caribbean Currency Union”. 23 agreed that less capacity development work was targeted on national accounts than on price statistics. They observed that, essentially, national accounts were outside the scope of the ICP programme and were only required as a facilitating tool to establish a weighting system for the basket of consumer products. In other words, while a PPP could not be produced without price statistics, it could still be produced with estimated weights in the absence of national accounts by expenditure type. Box 3 THE NATIONAL ACCOUNTS CHALLENGE The limited development of national accounts was one of the main constraints for the Caribbean countries. Even though about 14 countries reported using the 1993 SNA, the full adoption of international recommendations and standards was limited to the goods and services accounts. Project report, December 2012 Finding 8. All countries adopted a common classification of goods and services correlated with classification of household expenditure in GDP, but some countries still faced challenges in compiling CPIs in line with international standards 61. In order to finalize the regional list of products for Caribbean countries, two working group meetings on price statistics were held in Trinidad and Tobago in October 2011. The first workshop (10-12 October) was attended by 23 participants from 19 countries, and the second workshop (13-15 October) was attended by 21 representatives from 17 countries. The workshops provided practical coaching through field simulation on price data collection using the structured product description (SPD) format and the ICP tool kit. 62. By the end of 2011, 10 of the participating countries had sufficiently developed their technical capacities for collecting price data. However, most of the participating countries continued to face challenges in compiling CPIs due to various constraints, including difficulty with product classification, lack of up-to-date household surveys, inappropriate data collection methods and undeveloped statistical infrastructure in general. Finding 9. The project helped to develop informal networks among participating countries, but the planned web-based forum for networking national statistical experts was not established 63. While the online forum (http://teamrooms.cepal.org/LotusQuickr/icp2011/Main.nsf) was physically set up in 2010, its function —to enable experts from national statistical offices and international organizations to exchange information and coordinate activities for the development of common data collection protocols, definitions and classifications in accordance with the ICP guidelines— was never developed. The project had also planned that the platform would provide a repository of reference and technical materials, including workshop reports, instructions, common templates and forms, and a set of international recommendations for price and national accounts data collection, estimation and validation.19 64. Although this link is available on the web, the forum was never launched or operationalized. Project beneficiaries who were interviewed were not aware of its existence. The project managers explained that, given the workload faced by the project focal points of participating countries, it had been concluded that the forum was unlikely to be used effectively. Instead, a decision was made to have a 19 The evaluator was unable to review the online site because it required a login username and password. 24 general project website with links to the ECLAC and World Bank websites. The web page was only a repository of reference materials and not a forum, and the site did not contain a tracking tool to monitor the extent of its usage. 65. Nonetheless, the project had facilitated the development of informal networks among national statisticians in participating countries. Project beneficiaries agreed that the project constituted a platform for networking with regional peers. For example, the Government of the Bahamas planned to introduce value added tax (VAT), and the focal person for national accounts had used the networks established through project participation to solicit ideas from peers on how their countries had rolled out the VAT process. In response to the electronic survey question “Did your participation in the workshops and seminars help your institution to strengthen its regional network of experts and practitioners and increase interchange of experiences, best practices and methodologies?”, 33% of beneficiaries agreed and another 22% partially agreed. Although none disagreed, 45% did not respond. 4.3. Efficiency 66. This section focuses on the project’s efficiency, including an assessment of implementation effectiveness in terms of time and cost estimates, and whether or not the project resources were focused appropriately on critical activities. Finding 10. The project design did not set out a clear governance and management structure 67. The project document did not address the issue of project governance and management. The section on implementation arrangements20 sets out the roles and responsibilities of some of the key stakeholders but falls short of providing a governance and management structure (box 4). Box 4 EXTRACTS FROM PROJECT DOCUMENT ON IMPLEMENTATION ARRANGEMENTS The Statistics and Economic Projections Division of ECLAC will be responsible for the overall technical coordination and execution of the project’s main activities. A team based in the subregional office of ECLAC in Port of Spain will provide local support and coordinate the logistics related to technical assistance missions, organization of workshops and meetings, and follow-up of survey questionnaires. CARICOM will support coordination activities and provide focal points with technical expertise on national accounts and price statistics, and support the establishment and strengthening of a network of subregional and national experts in both topics. CARICOM will also actively participate in the regional workshops and meetings convened by ECLAC. 68. The role of CARICOM in the governance and management of the project was not clearly defined. By its very nature, as the regional organization representing Caribbean member countries, CARICOM was a key partner and stakeholder. It was a very influential partner during the advocacy phase in garnering support and political will for participating countries to commit themselves to the ICP processes in the context of the financial and human resources required for their participation. Project stakeholders noted that, beyond this, the role of CARICOM was not clearly defined. Indeed, the project progress reports allude to the point (box 5). 20 ProDoc. Section 7, p. 16. 25 Box 5 THE ROLE OF CARICOM (PROGRESS REPORT: JULY 2010 TO DECEMBER 2012, P. 9) The strategic partnership with regional agencies is deemed crucial for the success of the project. While the support of the CARICOM Secretariat proved crucial and indispensable in the organization of all project meetings, providing invaluable logistical support and cofinancing the participation of country delegates by enabling the organization of project meetings back to back with various CARICOM Secretariat events, there was still a need to clarify at the highest level possible its role and responsibilities in the framework of the 2011 round of the ICP in the Caribbean. During 2012, the role of CARICOM was not strong enough in terms of capacity building and financial support. 69. The evaluation noted that the project design process followed the guidelines established by the Capacity Development Office (CDO) of DESA in its capacity as the custodian of the Development Account. However, projects with multiple stakeholders usually require a project steering committee to be established to ensure that these are engaged and their voices are heard. In addition, it would be useful for a project document to include a section on project governance and management, clearly defining responsibilities for both strategic and operational decision-making, as well as oversight functions. 70. During interviews, project beneficiaries said that they normally contacted ECLAC project managers in Santiago by e-mail and telephone if they required guidance or reference materials. They also noted that the project management was very responsive to their queries, but observed that it would have been more efficient if the ECLAC Subregional Headquarters for the Caribbean in Port of Spain (Trinidad and Tobago) had the capacity to address their queries. Project beneficiaries observed that the ECLAC Subregional Headquarters in Trinidad and Tobago did not provide technical support to participating countries. Project managers agreed and confirmed that the Subregional Headquarters lacked technical capacity in statistics, but noted that the project strategy was to provide substantive technical assistance to member countries from ECLAC headquarters in Santiago, Chile. The project management also noted that the Subregional Headquarters in Port of Spain only had an administrative and coordinating role in the project. Finding 11. The project was responsive to the needs of beneficiaries, with the number and timing of activities being changed accordingly 71. While the project was initially planned to be implemented over a period of four years, from 2010 to 2013, activities were completed ahead of time, with the project’s last activity taking place in March 2012. Effective activity implementation therefore took place over just under two years (21 months). 72. The project was responsive to changing conditions and varied its activities as new information became available. According to the project document, six technical assistance missions (TAMs) were planned for the period from the third quarter of 2010 to the last quarter of 2011. However, as the number of participating countries increased to 20, and it became clear that most of them did not have enough statistical infrastructure and capacity to implement the ICP methodology, the number of TAMs was increased to nine. The project design also envisaged that four surveys would be undertaken in the second and third quarters of 2010 to determine the methodology employed by Caribbean countries in the production of consumer price indices and the compilation of national accounts statistics. The surveys were intended to determine the existing situation and capacity gaps in the countries in terms of source data, coverage and degree of compliance with ICP requirements. However, rates of response to the electronic questionnaires were very low and the project decided to supplement them with an assessment mission undertaken by an independent consultant. The evaluation noted that the assessment mission was undertaken in the first quarter of 2012 and was very unlikely to be useful as a capacity needs assessment, serving rather as validation of the effectiveness of capacity building work already undertaken. 26 73. Two regional workshops to harmonize CPIs were also planned for 2010 and 2011. The first workshop was intended to establish a timetable and list of activities to be carried out by each country in order to implement the harmonized CPI and the core list of the consumer basket for Caribbean countries, while the second workshop would establish a final and validated core list of consumer products in the regional basket. After recognizing the extent of the challenges faced by the countries, the project increased the number of these workshops to three. 74. As already noted, the sequence of the implemented activities departed from the plan, something that had the potential to affect attainment of the project objective (table 5, see also annex 4). For example, the survey of methodologies used by the countries should clearly have been undertaken in advance in order to determine the capacity gaps that had to be addressed. The evaluation also noted that the implemented activities were not sufficient by themselves to achieve the project objective. The project management agreed, and explained that project 10/11AH was being implemented within the framework of the overall ICP programme led by the World Bank. The immediate project objective was therefore to build capacity in participating countries to enable them to participate in the 2011 round of the ICP. Looked at from this perspective, the planned and implemented activities were quite appropriate. Table 5 ACTIVITIES PLANNED AND IMPLEMENTED Planned activities Technical assistance missions Survey of CPI methodologies Regional workshops on CPI harmonization Survey of GDP methodologies Workshop on national accounts High level meeting with heads of NSOs Design of web-based forum Working group meetings 2010 2011 2012 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 1 2a a a a 3a 2a 1 aa 1 a1 a 1 1 a a1 1a 1 1 a1 1a a b a b a b a b a b a a 3a a Planned activity timeframe b Number of activities implemented in the timeframe. Finding 12. The project established partnership and collaboration with the World Bank and CARTAC 75. A good partnership and collaboration were established between ECLAC, CARICOM and the World Bank. The Statistics and Economic Projections Division (SEPD) of ECLAC was responsible for the overall technical coordination and execution of the project, with coordination support from the subregional office in Trinidad and Tobago. 76. CARICOM provided focal points with technical expertise in national accounts and price statistics and supported the establishment and strengthening of a network of subregional and national experts, while its technical arm, CARTAC, provided technical support at the regional workshops and meetings convened by ECLAC. 77. The World Bank directly funded the project (table 6) to complement the ECLAC activities. When the United Nations Security Council made the decision to implement the 2011 round of the ICP, the World Bank was given a mandate to lead the programme and host the Global Office, and ECLAC was 27 asked to perform the role of focal point for the LAC region.21 The World Bank was therefore both an implementing agency and a funding partner, and the Development Account funding contributed to the global programme of the 2011 ICP round. Table 6 WORLD BANK FUNDING OF PROJECT ACTIVITIES (Dollars) Meetings Technical assistance missions Travel: staff and participants Sub-grants Contribution in kind Total 300 000 80 000 55 000 45 000 80 400 560 400 78. Of the key implementing partners, 25% rated the level of coordination between partners in the implementation of project activities as excellent and the other 75% rated it as good. Finding 13. A large portion of the project budget was allotted to substantive activities, and the level of delivery was very high 79. Key activities such as the TAMs and capacity development for price and national statistics methodologies and data collection received the bulk of the project budget (figure 4). Over 77% of the total budget was allotted to substantive project activities, including fellowships and grants (46%) and consultants and expert groups (31%). Figure 4 BUDGET ALLOCATION BY ACTIVITY AREA Fellowships, grants and contributions 183 555 (46%) Consultants and expert groups 150 527 (38%) Operating expenses 4 871 (1%) Staff travel 62 048 (15%) 80. The project also achieved a highly satisfactory level of budget delivery. At the time of drafting this report (October 2013), almost 99% of the total allocated budget had been delivered (table 6). 21 In other regions, it was regional development banks such as the African Development Bank and the Asian Development Bank that were focal points for their respective regions. 28 Budget allocation (US$) Total expenditure (US$) Delivery rate (%) Table 7 BUDGET DELIVERY (CUMULATIVE) December 2010 401 000 106 572 26.6% December 2011 274 478 68.4% December 2012 366 083 91.3% August 2013 396 477 98.9% 4.4. Sustainability 81. This section presents an assessment of the likelihood of project processes and key results continuing beyond the project cycle, including a discussion of potential risks to their sustainability. Finding 14. The project design did not specify a sustainability strategy and plan 82. While recognizing that the project was a part of the larger ICP programme led by the World Bank, and that the design was based on CDO-DESA guidelines for projects under the Development Account, the evaluation nonetheless observed that the project design did not contain a specific sustainability plan. If sustainability is required, the project should develop a sustainability plan and integrate it into the project design. The sustainability plan should clearly spell out what needs to be sustained, who will sustain it and how, and what this will require. The plan should address the critical factors that may affect the project’s sustainability, including (i) contextual factors, such as political and institutional aspects at the national level, and (ii) project-level factors, such as tool kits and reference manuals. In this context, the evaluation noted that the project-level factors were sufficiently addressed. 83. Sustainability may not be required for all aspects of a project, because while some activities or outputs may be maintained, others may not be necessary (diagram 3). Diagram 3 PROJECT SUSTAINABILITY Inputs Grants Project activities Project outputs Meetings - Price data collection Workshops - Expenditure-based national accounts - Validation of prices of regional basket CPIs aligned with ICP methodology Need to be sustained Project outcomes Updated PPP 29 Finding 15. Some countries may face challenges in sustaining project processes and outputs 84. The evaluation noted that there was no common consensus among participating countries as to whether they would be able to sustain essential project activities and outputs. While some of the participating countries had said that they would continue collecting data using the ICP methodology, not all of them had categorically made such a commitment. Among those that had not committed themselves to continuing with the ICP methodology, the concern was that the outputs would not be useful without ECLAC support to verify the price data which they had to collect for the national basket, and they did not have the resources to collect price data for the much larger regional basket. 85. On the other hand, the project management saw a need for participating countries to continue price data collection, and even to revise their national CPI baskets to bring them closer to the regional basket. In that connection, the World Bank was considering how to continue funding data collection so that the Caribbean countries could be better prepared for the next ICP round. V. LESSONS LEARNED 86. All participating countries expressed satisfaction with their participation in the 2011 ICP round, noting that they were either unaware of or not in agreement with previous PPP rankings because they did not know how the data had been obtained. The evaluation noted that the original project concept envisaged participation by only 7 countries, later increased to 20 Caribbean countries. One of the key lessons, therefore, concerned the need for broader consultation and advocacy before developing a regional programme. Some additional lessons also emerged from project implementation. Lesson 1: National accounts statistics are just as important as price statistics in generating a credible regional PPP comparison 87. A majority of participating countries observed that disproportionate attention was given to price statistics as compared to national accounts statistics. Some stakeholders explained that this was due to the scope of the project, which ultimately was about price (purchasing power) comparisons. However, since the items in the basket of consumer products have to be weighted on the basis of relative expenditures against other products, national accounts data were seen as being equally important. In addition, the beneficiaries observed that national accounts statistics were a more specialized field of study, and therefore countries tended to have fewer national accounts statisticians than price statisticians. The emerging lesson, therefore, was that equal attention needed to be paid to national accounts and price statistics. Lesson 2: PPP comparability requires harmonization of both data collection approaches and timeframes 88. Price data were weighted on the basis of consumer spending as captured through the national accounts. Many of the countries were only compiling their GDP data from the supply side and then estimating expenditure data as a residual. Many of the countries were introduced to the use of MORES tables for the first time through the project, and this enabled them to collect GDP expenditure data and eliminate statistical discrepancies between supply-side and expenditure data. However, a key challenge was that the base years used were different in each country, and therefore any resultant trend analysis did not 30 start from the same point. The emerging lesson, therefore, concerned the importance of harmonizing data collection protocols and methodologies as well as primary data collection timeframes such as the HBS. Lesson 3: Clearly defined roles and responsibilities for project partners reduce conflicts of expectations 89. Project progress reports indicated that there were conflicting expectations with regard to the roles and responsibilities of the key project stakeholders. The evaluation noted that this multi-partner project was implemented without an inclusive project steering committee to provide strategic guidance and oversight. While recognizing that the project (10/11AH) was a small component of the overall ICP global programme led by the World Bank, the evaluation was also aware that this was a specific project funded separately by the Development Account and led by ECLAC. This being so, the project had its own partnership strategy, which required specific and effective measures for managing that collaboration. Lesson 4: Sustainability depends on effectively identifying and managing critical risks and assumptions 90. The ICP round is undertaken every five years, and the next one was therefore expected in 2016. Key project partners noted that it would be counterproductive for Caribbean countries to stop the ICP process and then start all over again when the next round was launched. Clearly, therefore, there was an expectation that some of the key project processes and outputs would be continued independently by participating countries beyond the project cycle. Some countries had already integrated some of the key project processes into their national systems, continuing to use MORES tables, for example, to collect expenditure-based GDP data. However, some of the participating countries had no such intention, much less a clear strategy for sustaining key project activities. The emerging lesson here was that sustainability involved risks and assumptions which should be identified in the project design and managed during implementation. VI. CONCLUSIONS 91. For the majority of participating countries, the project delivered valuable experience by introducing new approaches for their national statistical systems. In particular, most of the countries did not have effective methods for balancing their GDP by production approach with an expenditure approach. Many of the countries used the MORES tables and associated basic headings for the first time, and were able to balance their supply-side GDP with demand-side GDP. 92. As they were participating in the ICP for the first time, many of the countries found the ICP methodology very strict and demanding, owing partly to human resource constraints and also to unavailability of reliable up-to-date data on prices and expenditure trends. The project therefore provided an opportunity for beneficiary NSOs to strengthen their statistical systems. 93. Though the project was very much appreciated at the technical and working level of NSOs, there was probably insufficient advocacy at the policy level to ensure commitment at higher decision-making levels that would ensure continuity. Most notably, some participating countries saw PPP from a technical perspective, in terms of its ability to provide comparative data between countries. However, its more strategic application as a poverty measurement tool and therefore as a policy support tool was not sufficiently integrated into the design and implementation process. For example, many of the countries were not particularly aware of how gender and human rights issues could be integrated into their statistical systems. 31 VII. RECOMMENDATIONS 94. On the basis of the foregoing analyses, findings and lessons learned, the evaluation made four recommendations addressed to ECLAC and two recommendations addressed to participating Caribbean countries. Recommendation 1: For each ECLAC project, ECLAC implementing partners should establish a project steering committee to provide strategic governance and oversight for the project. 95. Projects with multiple stakeholders and implementing partners require a specific coordination mechanism to manage partner expectations. To ensure effective coordination, key partners should secure appropriate ownership and control of activities by participating in the project governance and management structures. In this connection, CDO-DESA might also review its guidelines on Development Account projects to make the establishment of project steering committees a mandatory requirement. Recommendation 2: ECLAC should develop technical capacity at its subregional office in Trinidad and Tobago for more effective support to Caribbean countries. 96. Most of the Caribbean countries found it more logistically convenient to access Trinidad and Tobago than Chile. In addition, most of them, including Trinidad and Tobago, have English as their official language. Recommendation 3: ECLAC should develop a clear sustainability plan in the project design. 97. Project sustainability can be defined as the proportion of project-initiated outputs and other local initiatives resulting from the project processes that will be continued and maintained after termination of project implementation. Sustainability therefore requires that a rigorous sustainability analysis should be undertaken at the time a project or programme is formulated. This analysis should be followed up by development of a sustainability strategy and plan. CDO-DESA may wish to review its guidelines for Development Account projects to include a sustainability plan as a prerequisite for project approval. Recommendation 4: ECLAC should conduct specific capacity evaluation after completing capacity development activities. 98. Capacity development involves learning by agents of change that brings about changes in policyrelated and organizational factors and thereby enhances the effectiveness and efficiency of efforts to achieve a specific development goal. ECLAC should develop and undertake targeted evaluations for its capacity development work to determine to what extent capacities have improved and whether such capacities are being applied effectively to the intended development goals. Recommendation 5: ECLAC should explore ways to enhance application of the project’s technical activities as a poverty measurement and policy support tool at the national and regional levels. 99. Although PPP was developed, it was clear that participating countries were not all at the same level of capacity, with some countries facing challenges due to financial, institutional, and staffing constraints. 32 100. In addition, there were indications of a possible lack of awareness at the highest decision-making levels of the project’s benefits and application as a policy and decision support tool. It is therefore recommended that ECLAC should explore ways to enhance application of the project’s technical activities as a poverty measurement and policy support tool at the national and regional levels. 33 Annex 1 DOCUMENTS REVIEWED 1. Evaluation Terms of Reference: Improving poverty measurement: Building national statistical capacity in Caribbean countries to estimate Purchasing Power Parities 2. ECLAC (2013), Draft Evaluation Policy and Strategy 3. Annual Development Account Progress Report: July 2010-December 2012 4. Annual Development Account Progress Report: July 2010-December 2011 5. Annual Development Account Progress Report: July 2010-December 2010 6. ECLAC (2010), project document: Improving poverty measurement: Building national statistical capacity in Caribbean countries to estimate Purchasing Power Parities 7. Regional Status Report for Latin America and the Caribbean: Seventh Regional Coordinators’ Meeting, 19-21 September 2012 8. The ICP in the Caribbean: Challenges and opportunities, 24 July 2010 9. Evaluation of Workgroup Meeting on Price Statistics, Port of Spain, 28-30 March 2011 10. List of Participants and Meetings Agenda (2010-2012) 11. Gervais, G. (2011), Mission report: Assessment of national accounts and price statistics of selected Caribbean countries for the 2011 ICP round 12. Redeby, J. (2011), Mission report: Assessment of the national account for the 2011 round of the International Comparison Programme, selected member States of the Eastern Caribbean Currency Union 13. Summary of conclusions: High Level Meeting to Launch the 2011 Round of the International Comparison Programme in the Caribbean, Barbados, 24 July 2010 14. Minutes of the Expert Group Meeting on National Accounts in the Caribbean: ICP Round 2011, Port of Spain, 26-28 September 2011 15. Minutes of the Expert Group Meeting on National Accounts: ICP Round 2011, Saint Kitts and Nevis, 29-30 March 2012 16. Minutes of the Expert Group Meeting on Price Statistics: ICP Round 2011, Saint Kitts and Nevis, 26-28 March 2012 34 Annex 2 INDIVIDUALS INTERVIEWED Name Title Agency/Country Stakeholders (implementing partners/project managers) 1. Bodemann-Ostow, N. Programme Officer, DPPO ECLAC 2. Reyes, A. Programme Assistant, DPPO ECLAC 3. Rodriguez, N. Team Assistant, DPPO ECLAC 4. Rosengren, M. Programme Officer, CDO DESA 5. Tincati, C. Associate Evaluation Expert, CDO DESA 6. Zivy, R. Economic Affairs Officer ECLAC 7. Alexander, D. Computer Info Systems Assistant ECLAC (Port of Spain) 8. Moonie, S. Statistics Assistant ECLAC (Port of Spain) 9. Garstenfeld, P. Chief, Statistics Division ECLAC 10. Marconi, S. Statistician ECLAC 11. Savio, G. Statistician ECLAC 12. De Camino, C. Statistics Assistant ECLAC 13. Ovalle, A. Statistics Assistant ECLAC 14. Blokland, M. CARTAC 15. Corbin, H. ECCB 16. Mouyelo-Katoula, M. ICP Global Manager World Bank Project beneficiaries 17. Carty, C. Statistical Officer Anguilla 18. Aska, J. Research Officer Antigua and Barbuda 19. Turnquest, C. Assistant Director Bahamas 20. Bomberg, S. Senior Analyst Curaçao 21. Forbes, S. Statistician Turks and Caicos Islands 22. Tyson, B. Statistician Dominica 23. Frame, M. Statistician Grenada 24. Newland, Y. Director, Economics Jamaica 25. Trejo, D.C. Manager Belize 26. Jackson, M.P. Assistant Statistician Bermuda 27. George, T. Economist British Virgin Islands 28. Christian, S. Senior Statistician Cayman Islands 29. Phipps, C. Director, Statistics Saint Kitts and Nevis 30. Beroo-Joseph, S. National Accounts Statistician Saint Lucia 31. George, E. Statistician Saint Vincent and the Grenadines 32. McCalman, A. Statistician Trinidad and Tobago 35 Annex 3.A ANALYSIS OF STAKEHOLDER RESPONSES TO THE ELECTRONIC SURVEY Number of e-mails administered ……………….14 Number of responses……………………………5 Response rate……………………………………36% Relevance of project objective to national priorities Project alignment to mandate of partners Did project address relevant capacity gaps? Project had synergy with other regional initiatives Integration of gender analysis into project design Integration of human rights into project design Stakeholder participation in project design Effective project governance and management Project resources were used appropriately Were there any efficiency gaps identified? Were there any unintended results? Any useful tools delivered by the project? Were intended results achieved overall? Overall implementation efficiency Political will to sustain project processes/results Project contribution to poverty measurement policies Project contribution to regional partnership and cooperation Coordination between implementing partners Achievement of expected accomplishments Yes No NR Comments 41 41 41 2 3 CPI-IPC integration 23 23 41 41 41 1 2 2 Lack of follow-up activities 2 3 All the countries participated 2 3 ICP tool kit and MORES tables 4 1 Improvement in GDP breakdown 41 212 311 14 Exc Gd Av P VP 13 One non-response 31 One non-response Note: NR = No response; Exc = Excellent; Gd = Good; Av = Average; P = Poor; VP = Very poor. 36 Annex 3.B ANALYSIS OF BENEFICIARY RESPONSES TO THE ELECTRONIC SURVEY Number of e-mails administered ……………….78 Number of responses……………………………25 Response rate……………………………………32% Activity key: A – High Level Meeting of Heads of NSOs, Barbados, July 2010 B – First Technical Meeting on National Accounts and Price Statistics, Bahamas, December 2010 C – Workgroup Meeting on Price Statistics, Trinidad, 21-23 March 2011 D - Workgroup Meeting on Price Statistics, Trinidad, 24-26 March 2011 E - Workgroup Meeting on Price Statistics, Trinidad, 28-30 March 2011 F – Expert Group Meeting on National Accounts, Trinidad, 26-28 September 2011 G - Workgroup Meeting on Price Statistics, Trinidad, 13-15 October 2011 H – Expert Group Meeting on National Accounts and Price Statistics, Saint Kitts, 23-30 March 2012 How relevant were the contents of the seminars and workshops for your institution’s priorities related to improving poverty measurement and data collection protocols? Activity Highly relevant (5) 4 3 2 Not relevant (1) No response Number of responses A 0100089 B 1 3 4 3 0 4 15 C 1 2 3 0 0 6 12 D 1 2 1 0 0 7 11 E 1 4 1 0 0 6 12 F 0 6 3 0 0 6 15 G 3 3 1 0 0 7 14 H 3 5 5 2 1 2 18 How would you assess the quality of the materials, facilitation and overall methodology used during the seminars and workshops in which you participated? Activity A B C D E F G H Excellent 1 1 1 1 2 2 4 5 Good 0 8 4 2 2 6 3 8 Average 0 1 1 1 1 1 0 1 Poor 0 0 0 0 0 0 0 1 Very poor 0 0 0 0 0 0 0 0 No response 8 4 6 6 5 6 7 2 Number of responses 9 14 12 10 10 15 14 17 37 How would you assess the scope and quality of the activities and discussions during the seminars and workshops in which you participated? Activity A B C D E F G H Excellent 0 1 1 1 0 1 1 4 Good 1 6 3 2 4 6 5 10 Average 0 3 0 1 0 1 0 1 Poor 0 0 0 0 0 0 0 0 Very poor 0 0 0 0 0 0 0 1 No response 7 4 6 6 6 6 6 2 Number of responses 8 14 10 10 10 14 12 18 To what extent did each of these workshops/seminars contribute to increasing your knowledge in your specific area of work? Activity A great deal A0 B2 C4 D2 E3 F3 G6 H7 Somewhat 0 5 1 1 1 4 1 7 Not much 0 3 0 1 1 1 0 1 Not at all 0 0 0 0 0 0 0 1 No response 8 4 6 6 5 5 6 1 Number of responses 8 14 11 10 10 13 13 17 To what extent have you applied the knowledge gained from the workshops or seminars in your daily work? Activity A great deal A0 B1 C2 D1 E2 F2 G4 H7 Somewhat 1 6 2 3 3 6 2 6 Not much 0 4 2 0 0 1 1 3 Not at all 0 0 0 0 0 0 0 0 No response 8 4 6 7 6 5 5 1 Number of responses 9 15 12 11 11 14 12 17 38 How would you assess the contribution of the workshops and seminars in which you participated to increasing your understanding of the gender perspective as it relates to poverty measurement and data collection? Activity A B C D E F G H Excellent 0 1 1 1 0 0 0 0 Good 0 0 1 0 1 2 3 4 Average 0 2 2 1 1 1 2 2 Poor 0 2 0 0 1 1 1 2 Very poor 0 2 0 0 0 2 0 4 No response 9 8 8 9 8 9 8 6 Number of responses 9 15 12 11 11 15 14 18 How would you assess the contribution of the workshops and seminars in which you participated to increasing your level of understanding of the human rights approach as it relates to poverty measurement and data collection? Activity A B C D E F G H Excellent 0 1 1 1 0 1 0 1 Good 0 1 0 0 1 2 2 3 Average 0 1 2 1 1 1 2 2 Poor 0 2 0 0 1 1 1 2 Very poor 0 2 0 0 0 2 0 4 No response 9 8 9 9 8 8 9 6 Number of responses 9 15 12 11 11 15 14 18 In relation to the seminars or workshops in which you participated, please indicate below your level of agreement or disagreement with the following statements: Activity Project strengthened data collection for CPI and national accounts Project strengthened technical capacity for CPI Project strengthened technical capacity for national accounts statistics Project strengthened regional network of experts Totally agree 5 5 4 7 Agree 3 Partially agree Disagree Totally disagree 521 7300 8400 7400 NR Number of responses 3 18 3 18 3 19 1 19 39 How satisfied are you with the technical assistance that you received? Activity Technical assistance national accounts (Saint Vincent) Technical assistance national accounts (Grenada) Technical assistance national accounts (Saint Kitts) Technical assistance for ICP (Antigua) Very satisfied 0 0 0 0 Satisfied 2 1 2 0 Not very satisfied 0 0 0 0 Not at all 0 0 0 0 NR Number of responses 9 11 10 11 8 10 10 10 How relevant were the contents of the technical assistance sessions for your institution’s mandate and priorities related to improving poverty measurement and data collection protocols? Activity Technical assistance national accounts (Saint Vincent) Technical assistance national accounts (Grenada) Technical assistance national accounts (Saint Kitts) Technical assistance for ICP (Antigua) Very satisfied 0 1 0 0 Satisfied 1 0 1 0 Not very satisfied 0 0 2 0 Not at all 0 0 0 0 NR Number of responses 10 11 10 11 7 10 10 10 How would you assess the overall quality of the technical assistance sessions in which you participated? Activity Technical assistance national accounts (Saint Vincent) Technical assistance national accounts (Grenada) Technical assistance national accounts (Saint Kitts) Technical assistance for ICP (Antigua) Excellent 0 0 0 0 Good 2 1 2 0 Average 0 0 0 0 Poor 0 0 0 0 Very poor 0 0 0 0 NR Number of responses 9 11 10 11 8 10 10 10 40 To what extent have you applied the knowledge gained from the technical assistance sessions in your daily work? Activity Technical assistance national accounts (Saint Vincent) Technical assistance national accounts (Grenada) Technical assistance national accounts (Saint Kitts) Technical assistance for ICP (Antigua) A great deal 1 1 2 0 Somewhat Not Much 01 00 00 00 Not at all 0 0 0 0 NR Number of responses 9 11 10 11 8 10 10 10 In relation to the technical assistance sessions in which you participated, please indicate below your level of agreement or disagreement with the following statements: Activity Project strengthened data collection for CPI and national accounts Project strengthened technical capacity for CPI Project strengthened technical capacity for national accounts statistics Project strengthened regional network of experts Totally agree 0 1 1 1 Agree Partially agree Disagree Totally disagree NR Number of responses 4 1 0 05 10 3 1 0 05 10 1 3 0 04 9 2 2 0 04 9 Survey question Has there been follow-up or upscaling of workshop topics in your country? Do you have the financial, technical and institutional capacity to apply the knowledge gained through the project? Has there been follow-up or upscaling in your country of the issues covered during technical assistance missions? Yes No. Percentage No. 4 21.1% 12 4 28.6% 6 5 35.7% 4 No Percentage 63.2% 42.9% 28.6% No response No. Percentage 3 15.8% 4 28.6% 5 35.7% 41 Annex 4 SCHEDULE OF PLANNED AND IMPLEMENTED ACTIVITIES Planned activities 2010 2011 2012 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Technical assistance missions 1H *2I * * * *3J *2K 1L Survey of CPI methodologies ** 1M Regional workshops on CPI harmonization * 1B * 1E 1F Survey of GDP methodologies * *G Workshop on national accounts *1B 1D 1F High level meeting with heads of NSOs * 1A Design of web-based forum *N *N *N *N *N *N Working group meetings * * *3C NOTES 1B 1A 3C 1D 1E 1F 1H 2I 3J 2K 1L 1M N G First Technical Meeting on National Accounts and Price Statistics High Level Meeting of Heads of National Statistical Offices 3 Workgroup Meeting on Price Statistics Expert Group Meeting on National Accounts Workgroup Meeting on Price Statistics Expert Group Meeting on Price Statistics and National Accounts June 2010, covered Barbados, Trinidad and Guyana (Gylliane Gervais) September 2010, covered Jamaica, Bahamas and Belize, October 2010 Saint Lucia (Gylliane Gervais) Saint Vincent and the Grenadines, Grenada, Saint Kitts and Nevis (Jan Redeby) Antigua and Barbuda, Trinidad and Tobago (Jan Redeby) Saint Kitts and Nevis (Jan Redeby) Saint Kitts and Nevis (Francisco Morales) Design regional list website (http://depe.cepal.org/icp/caribbean/admin/acceso.php) Bahamas, 30 November-1 December 2010 Planned Undertaken Q4 6 2 2 2 1 1 6 3 23 9 1 3 1 3 1 6 3 27 42 Annex 5 RESULTS-BASED PERFORMANCE MATRIX Expected accomplishments/ Main activities EA 1 Adoption by Caribbean countries of common data collection protocols, definitions and classifications for a set of statistics on prices and National Accounts that constitute a basis for estimating purchasing power parities according to the guidelines of the International Comparison Programme (ICP). EA 2 EA 3 EA 4 Strengthened technical capacity to produce and analyse representative and internationally comparable consumer price indices (CPIs) according to international classifications, standards and recommendations. Strengthened technical capacity to produce and analyse National Accounts by components of expenditure in line with the System of National Accounts 1993 (SNA 1993) and related international classifications, standards and recommendations. Strengthened regional network of experts and practitioners and increased interchange of experiences, best practices and methodologies on the production and use of price statistics and National Accounts. Indicators IA1.1 Number of countries having defined a core list (or basket) of final consumption products and services for the computation of harmonized CPIs at the basic heading level and shared its results with all participating countries by considering the project’s recommendations. IA1.2 Number of countries having developed a diagnosis of the structure of GDP by type of expenditure according to ICP requirements to develop weights for the calculation of PPPs. IA1.3 Number of countries using a common classification of goods and services correlated with the classification of household final consumption expenditure in GDP. Number of countries having adopted an implementation strategy to achieve the same result. IA2.1 Number of countries having adopted conceptual and methodological improvements in the compilation of their CPIs that are in line with international standards and the project’s recommendations. IA3.1 Number of countries having adopted conceptual and methodological improvements in the compilation of national accounts that are in line with the SNA 1993. IA4.1 Number of countries taking part in an interactive network of technicians responsible for CPI and National Accounts of the Caribbean countries. IA4.2 Number of participants reporting benefits from the network’s activities and exchange of experiences to improve their production and use of price statistics and national accounts. Status of indicators All 20 participating Caribbean countries had defined and adopted a common regional consumer basket. Some countries faced challenges with meeting the ICP specifications, but they were able to provide reasonably acceptable alternative items for the products. Some of the participating countries still faced challenges with their national accounts data due to unavailability of some of the services under the basic headings, e.g., rail services. Also, some countries noted that Government used cash accounting as opposed to the accrual method as required by the ICP methodology. All 20 participating countries had adopted compilation of CPIs in line with project recommendations, albeit to varying extents. A few of the countries were not fully using COICOP. Some countries still faced challenges with application of the MORES tables, but most had started the process of moving towards adoption of all the basic headings. The planned web-based forum was not established. However, participating countries were benefiting from the exchange of information with their peers in other countries via telephone and electronic mail. 43 Annex 6 EVALUATION TERMS OF REFERENCE Evaluation of the Development Account project “Improving poverty measurement: Building national statistical capacity in Caribbean countries to estimate Purchasing Power Parities”. I. EVALUATION BACKGROUND AND TOPIC The Development Account The Development Account is a programme of the United Nations Secretariat aimed at enhancing capacities of developing countries in the priority areas of the United Nations Development Agenda.22 It is financed through the Secretariat’s regular budget and has a biennial budget of US$ 19 million, funding approximately 28 projects implemented by 10 entities of the Executive Committee of Economic and Social Affairs (DESA, the five United Nations Regional Commissions, UNCTAD, UNEP, UN‐HABITAT and UNODC). Since its establishment in 1997, 256 projects have been funded from the Account, with a total envelope of US$ 156.9 million. The present project being evaluated, entitled “Improving poverty measurement: Building national statistical capacity in Caribbean countries to estimate Purchasing Power Parities” was approved under the 7th tranche of the Development Account for the 2010‐2011 biennium, and was implemented by the Economic Commission for Latin America and the Caribbean’s (ECLAC) Statistics Division and Subregional Headquarters for the Caribbean. Project overview These terms of reference (ToR) describe the final evaluation to be conducted of this Development Account project. The key objective of the project was to bridge the capacity gap of national statistical offices in Caribbean countries,23 to establish common data collection protocols related to purchasing power parity and to produce and analyse national accounts in order to improve poverty information in the Caribbean. Specifically, it sought to: i) ensure that the Caribbean region is represented in the 2011 round of the International Comparison Programme (ICP), ii) significantly improve the measurement of poverty in the region in accordance with internationally comparable methodologies and iii) enable the estimation of PPPs by all Caribbean countries on a sustainable basis.24 22 Development Account projects are implemented in the following thematic areas: advancement of women; population/countries in special needs; drug and crime prevention; environment and natural resources; governance and institution building; macroeconomic analysis, finance and external debt; science and technology for development; social development and social integration; statistics; sustainable development and human settlement; and trade. See also the United Nations Development Account website [online] http://www.un.org/ esa/devaccount/projects/active/theme.html. 23 See footnote 25 for a list of countries. 24 Project information in this document is derived from the ECLAC project document for the additional funds in the seventh tranche of the Development Account, “Improving poverty measurement: Building national statistical capacity in Caribbean countries to estimate Purchasing Power Parities”, 21 April 2010. 44 The project was implemented in the Caribbean region, covering 20 countries.25 The overall budget of the project, totalling US$ 401,000, was financed by the United Nations Development Account, and was supplemented by World Bank resources (through various World Bank trust funds). The project was implemented in close collaboration with the World Bank and CARICOM, both of which are entities involved in the ICP. Overall management and coordination of the project was undertaken by ECLAC. While the project was initially planned to be implemented over a period of four years, from 2010 to 2013, activities were completed ahead of time, in December 2012. The overall duration of the project was therefore two and a half years, with progress reports prepared on a yearly basis. Expected accomplishments, indicators of achievement and planned activities As part of its results framework, the project contained a set of expected accomplishments, their corresponding indicators of achievement, as well as specific activities aimed at achieving these accomplishments. These are as follows: Expected accomplishments: 1. Adoption by Caribbean countries of common data collection protocols, definitions and classifications for a set of statistics on prices and national accounts that constitute a basis for estimating purchasing power parities according to guidelines of the ICP. 2. Strengthened technical capacity to produce and analyse representative and internationally comparable consumer price indices (CPI’s) according to international classifications, standards and recommendations. 3. Strengthened technical capacity to produce and analyse national accounts by components of expenditure in line with the System of National Accounts 1993 (SNA 1993) and related international classifications, standards and recommendations. 4. Strengthened regional network of experts and practitioners and increased interchange of experiences, best practices and methodologies on the production and use of price statistics and national accounts. Indicators of achievement: IA1.1 Number of countries having defined a core list (or basket) of final consumption products and services for the computation of harmonized CPIs at the basic heading level and shared its results with all participating countries by considering the project’s recommendations. IA1.2 Number of countries having developed a diagnosis of the structure of GDP by type of expenditure according to ICP requirements to develop weights for the calculation of PPPs. 25 The number of Caribbean countries participating in this project was increased from the 7 in the original concept note to 20, following a recommendation by CARICOM and other strategic partners. The 20 countries are: Antigua and Barbuda, the Bahamas, Barbados, Belize, Dominica, Grenada, Guyana, Jamaica, Montserrat, Saint Lucia, Saint Kitts and Nevis, Saint Vincent and the Grenadines, Suriname, Trinidad and Tobago, Anguilla, Bermuda, British Virgin Islands, Cayman Islands, Turks and Caicos Islands and Aruba. 45 IA1.3 Number of countries using a common classification of goods and services correlated with the classification of household final consumption expenditure in GDP. Number of countries having adopted an implementation strategy to achieve the same result. IA2.1 Number of countries having adopted conceptual and methodological improvements in the compilation of their CPIs that are in line with international standards and the project’s recommendations. IA3.1 Number of countries having adopted conceptual and methodological improvements in the compilation of national accounts that are in line with the SNA 1993. IA4.1 Number of countries taking part in an interactive network of technicians responsible for CPI and national accounts of the Caribbean countries. IA4.2 Number of participants reporting benefits from the network’s activities and exchange of experiences to improve their production and use of price statistics and national accounts. Planned activities: A1.1 Deliver technical assistance and advisory missions to the various countries participating in the project. These missions are essential in order to ensure in each country the synchronization of the different stages in the areas of prices and national accounts. A2.1 Carry out a survey of the methodology employed by Caribbean countries in the production of consumer price indices (CPI) out through a questionnaire and direct communication with the agencies responsible for price statistics in each country and other relevant national and international agencies. This survey will seek to determine the current situation and capabilities of the countries in terms of source data, coverage, and degree of compliance with ICP requirements. A2.2 Organize two subregional workshops in the Caribbean on CPI harmonization. The main purpose of the first regional workshop on CPI will be to establish a timetable and list of activities to be carried out by each country to implement the harmonized CPI and the core list of the consumer basket for Caribbean countries. The second regional workshop on CPI will establish a final and validated core list of the consumer basket for Caribbean countries. A3.1 Carry out a survey of the methodology employed by Caribbean countries in the compilation of gross domestic product (GDP) by expenditure through a questionnaire and direct communication with agencies responsible for national accounts statistics in each country and other relevant national and international agencies. This survey will seek to determine the current situation and capabilities of the countries in terms of source data, coverage and degree of compliance with ICP requirements. A3.2 Organize one subregional workshop on national accounts where the participating countries will agree on the methodological adjustments required and establish a timetable for the compilation the final consumption expenditure component of the GDP for the 2011 benchmark year. A4.1 Organize an advocacy high level meeting with heads of national statistical offices from Caribbean countries and other major stakeholders to ensure that all subsequent activities in the project are aligned with national, regional and global strategies for the development of statistics. 46 A4.2 Design and implement a web‐based forum to promote exchanges of experts between statistical offices to encourage the adoption of common methodologies and facilitate sharing of information on statistical standards and best practices. A4.3 Organize working group meetings to draw on the strengths and achievements of the countries and to create opportunities for staff, from less statistically developed countries to learn from their peers about best practices in the Caribbean subregion (South‐South cooperation). II. PURPOSE AND OBJECTIVES OF THE EVALUATION Context This evaluation is in accordance with the General Assembly resolutions 54/236 of December 1999 and 54/474 of April 2000, which endorsed the Regulations and Rules Governing Programme Planning, the Programme Aspects of the Budget, the Monitoring of Implementation and the Methods of Evaluation (PPBME).26 In this context, the General Assembly requested that programmes be evaluated on a regular, periodic basis, covering all areas of work under their purview. As part of the general strengthening of the evaluation function to support and inform the decision‐making cycle in the United Nations Secretariat in general and ECLAC in particular and within the normative recommendations made by different oversight bodies27 endorsed by the General Assembly,28 ECLAC’s the Executive Secretary is implementing an evaluation strategy that includes periodic evaluations of different areas of ECLAC work. This is therefore a discretionary internal evaluation managed by the Programme Planning and Evaluation Unit (PPEU) of ECLAC’s Programme Planning and Operations Division (PPOD). Objectives As the final evaluation of the project, this exercise is summative in nature. Specifically, it seeks to: 1. Analyse the design of the project as well as the relevance of its stated goals to the thematic area and region within which it operated, with particular emphasis on the needs of its beneficiaries. 2. Assess the project’s level of efficiency in implementing its activities, including its governance and management structures and use of resources. Moreover, the evaluation will examine the level of coordination among implementing partners and assess the project’s strategic partnerships. 3. Take stock of the results obtained by the project and evaluate the extent to which it achieved its objectives. To the extent possible, assess initial impact attributable to the project. 26 ST/SGB/2000/8, articles II, IV and VII. 27 OIOS report entitled “Assessment of evaluation capacities and needs in the United Nations Secretariat” (IED-2006-006, 24 August 2007) and the Joint Inspection Unit report entitled “Oversight lacunae in the United Nations system” (JIU/REP/2006/2). 28 Including General Assembly resolutions 54/236 and 54/474 endorsing the PPBME rules and regulations (ST/SGB/2000/8). 47 4. Evaluate the sustainability of the project, considering its financial, political, institutional and technical dimensions. Identify prospects for scale‐up and replication of its activities and outputs to other countries. 5. Identify best practices and lessons learned in the project implementation process that can serve to inform future development projects on the topic, undertaken either by ECLAC or other United Nations entities. III. SCOPE OF THE EVALUATION The evaluation’s unit of analysis is the project itself including both the design and implementation of planned activities and the results and impacts achieved. The timeframe to be studied corresponds to the period beginning with the project’s initial design and ending with the completion of its final activities, amounting to two and half years in total. In this project, ECLAC worked with a range of stakeholders at national, regional and global levels, aiming to strengthen their statistical capacity to compile economic and social indicators and to address their specific capacity needs in relation to achieving internationally agreed development goals. These stakeholders included: project implementing agencies and their partners at national level; national agencies responsible for compiling price and national accounts statistics; entities constituting the global ICP governance framework; and regional and international agencies promoting statistical capacity building. The evaluation will seek input from all of these different stakeholders, taking into account diverse perspectives on the project’s strengths and weaknesses. The target audience and principal users of the evaluation include all project implementing partners and beneficiaries, as well as other regional commissions and agencies of the United Nations system and government counterparts active in the Caribbean region and on the topic of statistical capacity for poverty measurement. IV. GUIDING PRINCIPLES AND ETHICS ECLAC guiding principles The evaluator will apply ECLAC’sguiding principles to the evaluation process.29 In particular, special consideration will be taken to assess the extent to which ECLAC’s activities and products respected and promoted human rights, equity and justice. This includes a consideration of whether ECLAC interventions treated beneficiaries as equals, safeguarded and promoted the rights of minorities, and helped to empower civil society. Moreover, the evaluation process itself, including the design, data collection, and dissemination of the evaluation report, will be carried out in alignment with these principles. 29 See ECLAC, “Preparing and conducting evaluations: ECLAC guidelines” (2009) for a full description of its guiding principles. 48 The evaluation will also examine the extent to which gender concerns were incorporated into the project —whether the design and implementation of the project incorporated the needs and priorities of women, whether women were treated as equal players, and whether it served to promote women’s empowerment. When analysing data, the evaluator will, wherever possible, disaggregate by gender. Development Account criteria Finally, the evaluation will place particular emphasis on measuring the project’s adherence to the following key Development Account criteria:30 • Result in durable, self‐sustaining initiatives to develop national capacities, with measurable impact at field level, ideally having multiplier effects; • Be innovative and take advantage of information and communication technology, knowledge management and networking of expertise at the subregional, regional and global levels; • Utilize the technical, human and other resources available in developing countries and effectively draw on the existing knowledge/skills/capacity within the United Nations Secretariat; • Create synergies with other development interventions and benefit from partnerships with non‐United Nations stakeholders. Norms, standards and ethics The evaluation will be conducted in line with the norms and standards laid out in the “Norms for evaluation in the United Nations system” and “Standards for evaluation in the United Nations system”. Moreover, the evaluation will apply United Nations Evaluation Group (UNEG) ethical principles as per its “Ethical Guidelines for evaluation”. Coordination Any previous reviews or assessments undertaken by units or divisions participating in the project will be taken into account in carrying out the evaluation. To this end, coordination with project partners will be critical to access relevant information. V. EVALUATION CRITERIA AND QUESTIONS This evaluation encompasses the different stages of the given project, including its design, process, results and impact, and is structured around four main criteria: relevance, efficiency, effectiveness and sustainability. Within each of these criteria, a set of evaluation questions will be applied to guide the analysis.31 The responses to these questions are intended to explain “the extent to which”, “why” and “how” specific outcomes were attained. 30 United Nations General Assembly, “Guidelines for the preparation of concept notes for the seventh tranche of the Development Account (2010-2011)”. 31 The questions included here will serve as a basis for the final set of evaluation questions, to be presented by the evaluator in the inception report. 49 1. Relevance: (a) Were the programme’s objectives relevant to the implementing countries’ development needs and priorities? (b) Were the project’s objectives aligned with the mandate of ECLAC and that of the Statistics subprogramme? (c) Were there any synergies or complementarities between planned outputs and other poverty measurement tools being developed by relevant actors in the region? (d) Was project design carried out through active involvement of all implementing partners? 2. Efficiency: (a) Did the governance and management structures of the project contribute to effective implementation of its operations? (b) Were services provided in a reliable and timely manner? (c) Were resources used efficiently and cost‐effectively? (d) Did the project apply protocols and practices to ensure that workflows were carried out effectively and coherently? (e) To what extent did implementing partners successfully coordinate the implementation of project activities? 3. Effectiveness: (a) What were the intended and unintended, direct and indirect results of the project? (b) To what extent did the project achieve the goals and objectives outlined in the project document? (c) How satisfied were the project’s main beneficiaries with the quality and timeliness of the services they received? (d) What are the preliminary impacts of the project? Has it contributed to increasing access to information, technical skills, and resources of the beneficiaries? 4. Sustainability: (a) Are project results expected to have a lasting impact on beneficiaries’ access to knowledge and technical capacity in the medium to long term? (b) Have national counterparts demonstrated the political will and commitment to carry project activities forward? (c) Do beneficiaries have the financial, technical and institutional capacities to take over the implementation of project activities? (d) Has the project contributed to the development of concrete policies aimed at strengthening poverty measurement in its target countries? (e) Does the project demonstrate potential for replication and scale‐up of successful practices? 50 VI. METHODOLOGY 1. Desk review and stakeholder mapping All relevant project information will be reviewed as part of the data collection process, including Development Account project criteria, the project document, annual progress reports, the final project report, workshop and meeting reports and surveys, country reports and the project webpage. Furthermore, stakeholder mapping will be carried out to chart project managers, implementing partners within the United Nations system and among national and civil society institutions at country level, as well as programme beneficiaries. 2. Electronic surveys Self‐administered electronic surveys will be developed and disseminated to three different types of stakeholders: (a) project managers within the Commission, (b) project partners within the United Nations system and the 20 countries taking part in the project and (c) project beneficiaries. 3. Country visits, stakeholder interviews and focus groups The evaluator will undertake an initial visit to ECLAC Headquarters in Santiago to meet with evaluation managers in the Programme Planning and Operations Division (PPOD) and focal points in the Statistics Division responsible for implementing the project. In addition, the evaluator will visit the ECLAC Subregional Headquarters for the Caribbean in Port of Spain, Trinidad and Tobago, and possibly one participating country within the region, as needed. The main data collection methods to be used during country visits are semi‐structured interviews and focus groups, which will be conducted with a range of project stakeholders including implementing agencies, partner institutions and project beneficiaries, to discuss the activities, results and impacts of the project. Additional interviews may be carried out via tele‐ or videoconference in countries that the evaluator is not able to visit. Information from these interviews will be validated and triangulated against the desk review and electronic survey. VII. THE EVALUATION PROCESS 1. Inception First, the ECLAC Evaluation Unit will establish an evaluation reference group (ERG), composed of representatives of the Department for Economic and Social Affairs (DESA) Capacity Development Office (as the managing unit of the Development Account) and of the project’s main implementing partners. The evaluator will begin the evaluation process by undertaking a desk review and data analysis of all relevant project documentation as well as a stakeholder mapping of key actors. Based on the desk review, the evaluator will prepare an inception report describing the background and context of the evaluation, its scope, methodology, and key questions. The draft of this report will be reviewed by the evaluation task manager and discussed with the evaluator before being revised and finalized. 51 2. Data collection Once the inception report has been completed, the evaluator, with the assistance of the ECLAC Evaluation Unit, will conduct electronic surveys among key project stakeholders. Moreover, the evaluator will undertake country visits to ECLAC Headquarters in Santiago as well as the ECLAC Subregional Headquarters for the Caribbean in Port of Spain and one participating country within the region, as needed. During country visits, the evaluator will conduct interviews and focus groups with implementing partners, national partners and beneficiaries to discuss the activities, results and impacts of the project. Additional interviews will be conducted via tele‐ or videoconference. Upon completion of the country visits and the overall data collection process, the evaluator will present the evaluation’s preliminary findings in a stakeholder debriefing via videoconference. During this meeting, project stakeholders, including the ERG, will be invited to provide comments and feedback to the evaluator. 3. Analysis and report drafting After completion of the data collection process, the evaluator will conduct an analysis of the various sources of data collected, including project documents, survey results, and interview and focus group findings. This analysis serves as the basis for the evaluation findings, conclusions, lessons learned and recommendations. The analysis will be followed by the preparation of the draft evaluation report, which will be reviewed by both the evaluation task manager and the ERG for comments. These comments will be addressed by the evaluator in the revision process, and will be responded to formally by the evaluator in a revision matrix, indicating what adjustments were made according to each comment and why. Once the revision is complete, the evaluator will submit the final evaluation report. Upon finalization of the evaluation report, the evaluator will formally present the evaluation findings, conclusions and recommendations to programme stakeholders in a closing meeting. The report is then sent for editing and translation and is disseminated via the ECLAC intranet system as well as ECLAC’s public webpage. The evaluation report is also shared with DESA’s Capacity Development Office, the management unit of the Development Account so as to contribute to the accountability and learning process of the evaluation. 4. Follow‐up ECLAC’s Evaluation Unit will undertake a follow‐up to the evaluation for an initial period of two years so as to ensure that the Commission benefits institutionally from the lessons learned and recommendations developed through the exercise. Once the evaluation is finalized and disseminated, the Evaluation Unit will convene a meeting with all implementing partners to formulate a management response to each evaluation recommendation and determine key actions to carry the recommendations forward. Follow‐up actions will be reported on by assigned division focal points on a six‐monthly basis over the two‐year period, with meetings held after the first year as well as at the end of the two‐year period. Throughout the follow‐up process, the PPEU will issue annual reports to all ECLAC divisions summarizing the recommendations and actions of all evaluations. Additionally, evaluation recommendations and follow‐up actions will be incorporated into programme implementation guidelines 52 on an annual basis and will be included in guidance provided in the programme “kick-off” meetings at the outset of the implementation of planned activities. VIII. KEY PRODUCTS 1. Inception report The evaluation inception report provides background and overview of the project and defines the purpose, scope and key questions of the evaluation. It also describes the methodology, key stakeholders and data collection instruments to be used in the analysis. Interview guides and survey questionnaires are also included in this first report. 2. Stakeholder debriefing Once the data collection process and country visits have been completed, a summary of preliminary findings will be presented in a debriefing with stakeholders (via teleconference), in which the consultant will present the initial evaluation findings to ECLAC implementing partners. 3. Revision matrix The revision matrix contains the evaluator’s response to all comments made by both the ERG and the evaluation task manager. It indicates whether a comment was addressed in the revised report,how it was addressed, and a justification for the evaluator’s decision. 4. Evaluation report The final report presents the outcomes of the overall evaluation process. It describes the main activities and results of the project, the findings of the data collection process, and the lessons, conclusions and recommendations derived from it, including the project’s prospects for sustainability. The evaluation recommendations are key to guiding improvement efforts in management and implementation of future Development Account projects. 5. Closing meeting The main findings, conclusions and recommendations of the final evaluation report are presented to project stakeholders in a closing meeting. 6. Management response and follow‐up action plan ECLAC’s management response will outline the Commission’s position on each of the recommendations, as well as how it plans to address them. The action plan that is developed as part of the follow‐up to the evaluation indicates specific actions to be taken as a means of improving work processes and organizational performance, along with the respective responsible units and timeframe for completion. 53 IX. ROLES AND RESPONSIBILITIES IN THE EVALUATION 1. Commissioner and task manager of the evaluation: The Programme Planning and Evaluation Unit (PPEU): • Commissions the evaluation. • Develops the evaluation ToR. • Selects and recruits the evaluator. • Provides overall management of the evaluation and its budget and strategic guidance on the evaluation process. • Provides coordination support for the data collection process, including the dissemination and processing of the electronic survey, organization ofcountry visits. • Coordinates communication and information flow between the evaluator, project stakeholders and the ERG. • Manages the quality assurance process of the evaluation. Along with the ERG, reviews and provides feedback on evaluation deliverables. • Takes responsibility for the editing, translation and dissemination of the evaluation report. • Manages the evaluation follow‐up process, including the management response and action plan. 2. Evaluator The external evaluation consultant: • Undertakes a desk review, designs the evaluation methodology and prepares the inception report. • Conducts the data collection process, including design of the electronic survey, interviews and focus groups, and country visits. • Undertakes data analysis. • Prepares the evaluation report and carries out revisions, including a revision matrix. • Conducts two stakeholder debriefings to present: (a)preliminary evaluation findings and (b) the final conclusions and recommendations of the evaluation. 3. Evaluation Reference Group Representatives of programme-managing divisions and DESA as the Development Account management unit: • Participates in the presentation of preliminary evaluation findings and provides feedback. • Reviews and provides detailed comments on the draft evaluation report. • Participates in the presentation of the final evaluation report.