Sección o Parte de un Documento 1995 es 382.3 B584L(58739) Corden, W. Max Coediciones NAFTA AMERICA LATINA LIBERALIZACION DEL INTERCAMBIO NEGOCIACIONES COMERCIALES TRATADOS ZONAS DE LIBRE COMERCIO FREE TRADE AREAS LATIN AMERICA TRADE LIBERALIZATION TRADE NEGOTIATIONS TREATIES NAFTA Una zona de libre comercio en el Hemisferio Occidental: posibles implicancias para América Latina Incluye Bibliografía En: La liberalización del comercio en el Hemisferio Occidental - Washington, DC : BID/CEPAL, 1995 - p. 13-40 Proyecto Apoyo al Proceso de Liberalización Comercial en el Hemisferio Occidental http://hdl.handle.net/11362/1510 2014-01-02T14:51:16Z 2014-01-02T14:51:16Z Made available in DSpace on 2014-01-02T14:51:16Z (GMT). No. of bitstreams: 0 Previous issue date: 1995 POLÍTICA COMERCIAL Y ACUERDOS COMERCIALES TRADE NEGOTIATIONS COMERCIO INTERNACIONAL E INTEGRACIÓN INTERNATIONAL TRADE AND INTEGRATION Texto May 2019 ASSESSMENT OF DEVELOPMENT ACCOUNT PROJECT 14/15 BD Strengthening the capacity of Central American and Caribbean countries in the preparation of sustainable energy policies and strategies FINAL ASSESSMENT REPORT FINAL ASSESSMENT REPORT ASSESSMENT OF DEVELOPMENT ACCOUNT PROJECT 14/15 BD Strengthening the capacity of Central American and Caribbean countries in the preparation of sustainable energy policies and strategies May 2019 This report was prepared by Jon García, an external consultant, who led the evaluation. Mr. García worked under the overall guidance of Raúl García-Buchaca, Deputy Executive Secretary for Management and Programme Analysis of the Economic Commission for Latin America and the Caribbean (ECLAC), and Sandra Manuelito, Chief of the Programme Planning and Evaluation Unit of ECLAC; and under the direct supervision of Anne-Sophie Samjee, Programme Management Officer of the same Unit, who provided strategic and technical guidance, coordination, and methodological and logistical support. The evaluation team is grateful for the support provided by its project partners at ECLAC, all of whom were represented in the Evaluation Reference Group. Warm thanks go to the programme managers and technical advisers of ECLAC for their cooperation throughout the evaluation process and their assistance in the review of the report. All comments on the evaluation report by the Evaluation Reference Group and the evaluation team of the Programme Planning and Evaluation Unit were considered by the evaluator and duly addressed, where appropriate, in the final text of the report. The views expressed in this report are those of the author and do not necessarily reflect the views of the Commission. Copyright © United Nations, August 2019. All rights reserved Printed at United Nations, Santiago S.19-00473 iii TABLE OF CONTENTS Page ACRONYMS ...................................................................................................................................................... iv EXECUTIVE SUMMARY ..................................................................................................................................... vii 1. INTRODUCTION ........................................................................................................................................... 11 2. BACKGROUND ............................................................................................................................................. 12 2.1 GENERAL INFORMATION ON THE PROJECT .......................................................................................................... 12 2.2 STAKEHOLDER ANALYSIS ........................................................................................................................................... 13 2.3 THEORY OF CHANGE ................................................................................................................................................. 13 3. METHODOLOGY ........................................................................................................................................... 15 3.1 REVIEW METHODS ....................................................................................................................................................... 15 3.2 CHALLENGES AND POTENTIAL LIMITATIONS ......................................................................................................... 15 4. FINDINGS ..................................................................................................................................................... 16 4.1 RELEVANCE .................................................................................................................................................................... 16 4.2 EFFECTIVENESS ............................................................................................................................................................. 21 4.3 EFFICIENCY .................................................................................................................................................................... 25 4.4 SUSTAINABILITY ............................................................................................................................................................ 38 4.5 CROSS-CUTTING ISSUES ............................................................................................................................................ 40 5. CONCLUSIONS ............................................................................................................................................. 42 6. LESSONS LEARNED ...................................................................................................................................... 46 7. RECOMMENDATIONS .................................................................................................................................. 48 ANNEXES .......................................................................................................................................................... 51 ANNEX 1 EVALUATION MATRIX .................................................................................................................................... 52 ANNEX 2 LIST OF CONSULTED DOCUMENTS ............................................................................................................. 58 ANNEX 3 LIST OF PERSONS INTERVIEWED ................................................................................................................. 59 ANNEX 4 SEMI STRUCTURED INTERVIEW PROTOCOLS ............................................................................................ 60 ANNEX 5 ONLINE SURVEY .............................................................................................................................................. 63 ANNEX 6 EVALUATOR’S REVISION MATRIX ................................................................................................................ 66 iv ACRONYMS ACRONYM DEFINITION ADEME French Environment and Energy Management Agency AF Adaptation Fund ALIDES Alliance for the Sustainable Development of Central America AMEXCID Mexican Agency for International Development Cooperation BIEE Energy Efficiency Database CAN Andean Community of Nations (initials in Spanish) CARICOM Caribbean Community CIF Climate Investment Funds COP Conference of the Parties CSME Sectoral Council of Energy Ministers (initials in Spanish) DA Development Account DNRI Division of Natural Resources and Infrastructure EA Expected Accomplishment ECLAC Economic Commission for Latin America and the Caribbean ECLAC Mexico Subregional Headquarters of ECLAC in Mexico ECPA Energy and Climate Partnership of the Americas ENRU Energy and Natural Resources Unit FAO Food and Agriculture Organization of the United Nations GCF Green Climate Fund GEF Global Environment Facility GIS Geographic Information System GIZ German Agency for International Cooperation (initials in German) IA Indicator of Achievement IDB Inter-American Development Bank IADGs Internationally Agreed Development Goals IRENA International Renewable Energy Agency MERCOSUR Southern Common Market ME Monitoring and Evaluation v ACRONYM DEFINITION NAMA Nationally Appropriate Mitigation Action NDC Nationally Determined Contribution NGO Non-governmental Organization OLADE Latin American Energy Organization (initials in Spanish) PPOD Programme Planning and Operations Division PPT Presidency Pro Tempore PoW Programme of work REDD+ Reducing Emissions from Deforestation and Forest Degradation SAI Andean Integration System (initials in Spanish) SEI Stockholm Environment Institute SDG Sustainable Development Goal SE4ALL Sustainable Energy for All SICA Central American Integration System (initials in Spanish) SICREEE Renewable Energy and Energy Efficiency Centre TOR Terms of Reference UNAM National Autonomous University of Mexico (initials in Spanish) UNDP United Nations Development Programme UNEG United Nations Evaluation Group UN Environment United Nations Environment Programme UNFCCC United Nations Framework Convention on Climate Change UNIDO United Nations Industrial Development Organization WHO World Health Organization vii EXECUTIVE SUMMARY I. OVERVIEW OF THE EVALUATION SUBJECT 1. Belize, Costa Rica, the Dominican Republic, El Salvador, Guatemala, Honduras, Nicaragua and Panama make up the Central American Integration System (SICA). The SICA countries face challenges regarding universal access to modern energy sources, penetration of renewable energy sources and energy efficiency. Under its ninth tranche (2015–2017), the Development Account (DA) funded a project to address this challenge: “Strengthening the capacity of Central American and Caribbean countries in the preparation of sustainable energy policies and strategies”, also known by the reference 1415BD. Implemented between June 2015 and December 2018 by the Subregional Headquarters of the Economic Commission for Latin America and the Caribbean (ECLAC) in Mexico, in particular by the Energy and Natural Resources Unit (ENRU), the project aimed to strengthen the capacities of national governments to design and execute energy policies that ensure universal energy access and sustainable development of the energy sectors in the SICA subregion. II. EVALUATION OBJECTIVES, SCOPE AND METHODOLOGY 2. The purpose of this evaluation is to conduct the end-of-cycle review of the above-mentioned project. This assessment covers all the activities implemented by the project. The project has been screened through 17 broad evaluation questions assessing the following aspects: (i) relevance; (ii) effectiveness; (iii) efficiency; (iv) sustainability; and (v) consideration of human rights and gender issues. 3. The findings of this evaluation are based on a desk review of relevant documents, interviews of a wide range of stakeholders at the programme, national and regional levels, and an e-survey. Based on the information collected, the evaluator has cross-analysed and triangulated the data in order to inform the selected indicators and answer the evaluation questions presented in Annex 1. The evaluation was conducted in accordance with the norms and standards of the United Nations Evaluation Group (UNEG) as well as the ECLAC guiding principles for evaluations. In addition, UNEG ethical guidelines have been strictly observed. This evaluation was conducted by Jon Garcia between January and May 2019. III. MAIN FINDINGS Relevance 4. The project was relevant to the needs and problems of the participating countries and the region not only when it was formulated, but also when it was completed. Project design and implementation was participative. The project is aligned with the 2030 Agenda for Sustainable Development. The project is mostly aligned with Sustainable Development Goal (SDG) 7, but also contributes significantly to SDGs 1, 13 and 15, and less significantly to another four SDGs. The project objectives and activities were highly correlated with the ECLAC programmes of work (PoW) for 2014–2015, 2016–2017 and 2018–2019 with regards to the subprogramme on subregional activities in Central America, the Dominican Republic, Haiti and Mexico. The project built on previous work by ECLAC. Effectiveness 5. The level of achievement of the end-of-project targets in the project results framework is high. Four out of five had been met by project completion. One end-of-project target was exceeded, and another was not achieved. This reflects the project promoting significant changes. The technical capacity of government officials and other stakeholders increased as a result of project activities. A number of laws, policies, strategies and regulations advancing sustainable energy were adopted in viii participating countries during project implementation. The support provided by the project at the regional level was considerable in terms of strategies (such as development of a draft SICA 2030 Sustainable Energy Strategy) and institutional mechanisms (for example constitution of the SICA Sectoral Council of Energy Ministers (CSME) as a legal body). By supporting SICA countries in achieving SDG 7, the project contributed to progress on ECLAC priorities. The project generated some unplanned positive outcomes. Stakeholders were generally highly satisfied with the benefits received from the project. Efficiency 6. This complex project was extended one year (from two to three years) mostly owing to administrative and geopolitical factors. Some activities were also adjusted during implementation. At May 2019, some obligations were still outstanding, but additional funds were needed to complete activities. To increase efficiency, the project sought complementarities with the governments of participating countries, development partners and internally at ECLAC, but human resources for project management were limited. 7. The project document provides only a very generic ME plan, with limited resources. Monitoring roles were further developed during implementation. The results framework had significant gaps in terms of structure, indicators, baselines, targets and means of verification. The reporting template is comprehensive and useful but has some shortcomings regarding completion of activities and finance. Overall, reports were produced in a timely manner, are complete and are of good quality. 8. Reporting on challenges encountered and actions taken to solve them is generally good. Two of the three main challenges should have been anticipated during project design, establishing mitigation actions. The other main challenge was identified, and mitigation actions were defined to the extent possible. That said, the actions taken by the project to address the challenges encountered are generally adequate. The project sought to exchange best practices, success stories, experiences or lessons from non-SICA countries but did not have a strategy to systematically document and share lessons from SICA countries and the project. As noted, the project collaborated with governments of some participating countries and development partners, within and outside the United Nations system. Sustainability 9. Participating countries do not seem to have put in place institutional mechanisms to maintain and increase the technical capacity gained as a result of the project once it ended. The project has contributed to participating countries adopting laws, policies, strategies and regulations on sustainable energy. However, experience shows that legal frameworks are not always implemented, even when there is national ownership and monitoring frameworks exist. Conflicts with other legal frameworks and complex institutional arrangements can compromise implementation. Institutional progress at the national level is mixed, with regional progress contributing to national sustainability. Domestic funding seems to be limited, but a number of development partners are likely to support SICA countries on this matter. 10. A strategy to maintain and increase gained technical capacity does not seem to be in place at the regional level either. The progress at policy level is likely to continue, as once the SICA 2030 Energy Strategy and the Action Matrix are approved, SICA member states will have to implement them, including the 13 pillars and more than 200 actions. Considerable progress has been made at the institutional level with the constitution of the CSME as a legal body of SICA. The status of CSME as a legal body of SICA will continue. Two follow-up meetings have been organized in the first half of 2019, with support from other development partners, after the completion of the project. However, limited human and financial resources may compromise the ability of SICA to organize meetings, provide technical follow-up to the decisions made and support resource mobilization for implementation in the medium-term. ix Cross-cutting issues 11. The project did not factor in gender and human rights in the energy sector and has not produced a detailed assessment of these areas. The coverage of this during project implementation was limited; the project worked on biomass, which has positive gender equity impacts and the regional Action Matrix proposes some work on this. Other development partners plan to fill the gap. Participation of women in project activities was limited (women represented 17% of total project participants). Recruitment at ENRU prioritized women. IV. LESSONS LEARNED 12. Regional integration systems, such as SICA, have a key role to play in supporting sustainable development planning at the national level, particularly on topics that are highly technical and where interdependencies between countries are important, such as energy. 13. It is important to update national and regional policies, strategies and plans, including those related to energy, when there are major changes in international development commitments, such as the 2030 Agenda for Sustainable Development and its corresponding SDGs. 14. It is crucial to consider previous work when selecting an executing agency. Project delivery benefited from ECLAC experience in energy and regional integration in Central America. 15. Administrative changes should be planned in advance, establishing actions to reduce likely delays to the extent possible. 16. Regional integration is subject to instability and conflict between and within countries and within the regional integration mechanism. There is little room for action regarding some of these factors. This should be factored in when defining timeframes for regional projects. 17. Sustainable development planning at country and regional level may require significant funds. While complementarities with other projects should always be encouraged, each project should have enough funds to implement its main activities regardless of other projects. 18. Project management requires sufficient human resources. Availability of staff has to be analysed and dealt with in advance, particularly when recruitment processes are drawn-out. 19. ME has a key role to play in promoting effective and efficient project delivery. ME plans need to be well developed, with clear roles and responsibilities and schedules and sufficient budget. Results frameworks need to be clear and use specific, measurable, achievable, relevant and time- bound (SMART) indicators, baselines, targets and means of verification. 20. Projects related to regional integration should systematically promote the exchange of lessons and best practices not only from countries outside the region, but also within the region and from the project itself. 21. The implementation of policies, strategies or plans improves when there are monitoring frameworks and teams. Implementation can however be compromised by conflicting legal frameworks, complex institutional arrangements and limited technical and financial capacities. Projects should address these potential barriers, to the extent possible. Development partners have a role to play in supporting developing countries in mobilizing international resources, particularly for pilot, demonstration projects that can catalyse investment. 22. For regional integration to be sustained (and regional strategies implemented and followed) sufficient human and financial resources are needed. Development partners have a key role to play in this. x V. RECOMMENDATIONS 23. Recommendation 1. ECLAC should draw lessons from project implementation and systematically disseminate them. The dissemination strategy should distinguish between types of lessons and audiences. 24. Recommendation 2. ECLAC should consider the lessons learned from this project in the design and implementation of future projects on sustainable energy and on other topics. 25. Recommendation 3. ECLAC should explore the possibility of promoting the institutionalization of regional bodies on other topics at SICA and in other regional integration systems on energy and other topics. 26. Recommendation 4. ECLAC should support SICA countries and SICA in developing project proposals, particularly, but not only, for the GCF, to mobilize resources to advance in the implementation of the sustainable energy policies, strategies and plans supported by the project. FINAL ASSESSMENT REPORT 11 1. INTRODUCTION 1. This evaluation was conducted by Jon Garcia, between January and May 2019, responding to a commission by the Economic Commission for Latin America and the Caribbean (ECLAC). The objective of the assignment was to conduct, as per the Terms of Reference (TOR), an end-of-cycle review of the Development Account (DA) ninth tranche project “Strengthening the capacity of Central American and Caribbean countries in the preparation of sustainable energy policies and strategies”, also known as 1415BD. This assessment covers all the activities implemented by the project from project start, in September 2015, to finish, in December 2018. The evaluation was conducted following the norms and standards of the United Nations Evaluation Group (UNEG) as well as the ECLAC guiding principles for evaluations. In addition, UNEG ethical guidelines have been strictly observed. 2. The evaluation assesses the level of achievement of project results, documents good practices and draws lessons to improve the sustainability of the benefits generated by the project and foster potential for replication. Following UNEG guidelines, the evaluator has focused on assessing the extent to which the project addressed human rights and gender aspects, making every effort to produce and analyse gender-disaggregated data. 3. Based on the reconstructed theory of change of the project (section 3.3) and the indicators of achievement provided in the project document, the evaluator built an evaluation matrix. A series of evaluation questions were developed, building on the questions suggested in the TOR and grouped under the evaluation criteria described in the TOR: (1) Relevance, (2) Effectiveness, (3) Efficiency, and (4) Sustainability. A section on cross-cutting issues focusing on human rights and gender issues was also added. For the sake of clarity, 17 broad evaluation questions were proposed in the evaluation framework, encompassing the questions from the TOR and covering all relevant aspects of the project design, process, results and impacts. The Development Account criteria were incorporated as cross-cutting aspects in the matrix (see annex 1 for the evaluation matrix detailing all the evaluation features and see below for more methodological details). FINAL ASSESSMENT REPORT 12 2. BACKGROUND 2.1 GENERAL INFORMATION ON THE PROJECT 4. Belize, Costa Rica, the Dominican Republic, El Salvador, Guatemala, Honduras, Nicaragua and Panama make up the Central American Integration System (SICA). These countries have historically been characterized by high levels of poverty and inequality, and a high dependence on oil derivatives, specifically on imported fossil fuels, and traditional energy sources (firewood) for energy consumption.1 The volatility of international oil prices and climate change have hurt the economies of these countries. It was therefore crucial that policymakers of the subregion develop energy policies and strategies to make energy accessible and affordable for all and to promote economic growth and sustainable development. However, the capacities of the governments to prepare such policies and strategies were limited.2 5. Under its ninth tranche (2015–2017), the DA, a capacity development programme of the United Nations Secretariat, funded a project to address this challenge: “Strengthening the capacity of Central American and Caribbean countries in the preparation of sustainable energy policies and strategies”, also known by its reference 1415BD. Implemented by the Subregional Headquarters of ECLAC in Mexico (ECLAC Mexico), and more precisely its Energy and Natural Resources Unit, between June 2015 and December 2018, the project aimed to strengthen the capacities of national governments to design and execute energy policies that ensure universal energy access and sustainable development of the energy sectors in the SICA subregion. 6. The project had two expected accomplishments (EAs): (i) Increased knowledge and understanding of officers in the energy ministries and the SICA energy bodies regarding sustainable energy management and planning. (ii) Increased capacity of target countries to formulate sustainable energy strategies and policies and to comply with international commitments on sustainable development. 7. With a total budget of US$ 500,000, the project planned to undertake the following activities to achieve these EAs: • (A1.1) Organization of two regional workshops to discuss, coordinate and monitor the regional activities in the main areas of the energy strategy • (A1.2) South-South cooperation to solve specific national problems related to sustainable energy development • (A1.3) Study tours to learn about successful experiences in sustainable, equitable and inclusive energy development • (A1.4) Technical assistance for the development of prospective studies of national and regional energy development scenarios 1 25 million people in the region depend on firewood (82% concentrated in three countries), with huge negative impacts on health, while oil is overall the main source of commercial energy. Project document, p. 13. El Salvador, Dominican Republic and part of Guatemala have in addition a high population density. 2 The theory of change in section 1.3 illustrates the challenges that the project sought to address, based on the problem analysis presented in the project document (p. 9–13), including the problem tree (p. 13). FINAL ASSESSMENT REPORT 13 • (A2.1) Organization of one regional workshop for directors of energy and hydrocarbons to set objectives for the long-term regional sustainable energy development strategy • (A2.2) Organization of three national workshops for discussing national energy strategies or specific issues of energy development in some countries • (A2.3) Proposal of actions, projects and initiatives, at national and regional levels, to advance policies and strategies for sustainable energy, including recommendations for reducing the impact of high fuel prices that comply with international commitments on sustainable development • (A2.4) Organization of a workshop for the directors of energy and hydrocarbons for discussion and approval of the roadmaps for energy development • (A2.5) Organization of one regional seminar of energy ministers for presentation, final review and possible approval of the long-term regional energy strategy 2.2 STAKEHOLDER ANALYSIS 8. The project aimed to primarily benefit the government authorities and officials of the eight participating countries and more particularly the: • Energy ministries and technical directorates (energy and hydrocarbons) • Energy national technical offices (energy regulators, rural energization offices including public electricity utilities and forestry institutions) • Subregional and regional entities such as the SICA regional energy entities including the SICA Energy Coordination Unit and Regional Workgroups 9. The project was implemented by ECLAC and collaborated with various partners, including the International Renewable Energy Agency (IRENA), the Latin American Energy Organization (OLADE), the United Nations Industrial Development Organization (UNIDO), the Food and Agriculture Organization of the United Nations (FAO), the German Agency for International Cooperation (GIZ) and Sustainable Energy for All (SE4ALL), which is supported by the Inter-American Development Bank (IDB) and the United Nations Development Programme (UNDP).3 2.3 THEORY OF CHANGE 10. The project document includes a problem analysis (pp. 9–13), comprising a problem tree (p. 13), an analysis of the objectives (pp. 17–18) with an objective tree (p. 18) and a results framework (pp. 28–33), but does not explain how these are linked. The evaluator has therefore built on them to propose a reconstructed theory of change of the project. 11. A preliminary analysis shows that the project acts at national and regional levels. There are three expected impacts: (i) Capacity building has increased at national and regional levels (ii) Policy and strategies on sustainable energy have been developed at the national levels (iii) Regional cooperation in the area of universal energy access and sustainable energy policies has been enhanced 3 Launched in September 2011, the Sustainable Energy for All initiative (SE4ALL) is a multi-stakeholder partnership between governments, the private sector, and civil society. It has three interlinked objectives to be achieved by 2030: (i) ensure universal access to modern energy services; (ii) double the global rate of improvement in energy efficiency, and (iii) double the share of renewable energy in the global energy mix. FINAL ASSESSMENT REPORT 14 12. The figure below illustrates the reconstructed underlying rationale of the project. Figure 1 Reconstructed Theory of Change Source: Evaluator, based on the project document (pp. 13, 18, 20 and 28–33) Objective: Strengthen the technical capacity of staff to enable them to design energy policies for sustainable development of the energy sector and for achieving the goal of universal energy access EAsActivities Impacts EA1:Increased knowledge and understanding of officers in the Energy Ministries of SICA member countries and the SICA Energy bodies regarding sustainable energy management and planning EA2:Increased capacity of target countries to formulate sustainable energy strategies and policies and to comply with international commitments on sustainable development Barriers Training to increase technical knowledge on sustainable energy (A1.3) Development of a regional cooperation strategy in the energy sector (A1.1,A2.1, A2.3, A2.4, A2.5) Limited technical knowledge and understanding regarding sustainable energy planning and management at national and regional levels Insufficient national policies and strategies on sustainable energy Policies and strategies on sustainable energy have been developed at the national level, contributing to SDG 7 on universal energy access and other international commitments Enhanced regional cooperation on sustainable energy Increased technical knowledge on sustainable energy planning and management High dependence on oil derivatives for energy consumption, especially for commercial purposes Countries are net importers of oil Communities without access to electricity/dependence on firewood Volatility and steep rise in international oil prices Vulnerability to climate change Weak coordination on sustainable energy at the regional level Support for the development policies and strategies on sustainable energy at the national level (A1.2, A1.4, A2.2, A2.3, A2.4) Problems FINAL ASSESSMENT REPORT 15 3. METHODOLOGY 3.1 REVIEW METHODS 13. The findings of the end-of-cycle evaluation are based on a desk review of relevant documents and interviews and surveys aimed at the key national and regional stakeholders. 3.1.1 Desk review 14. The evaluator has systematically reviewed all the project documentation available. Relevant background documentation, monitoring and evaluation (ME) documents and policy documents were screened to ensure coverage of all five evaluation criteria. The in-depth documentation review included the programmes of work of ECLAC (2014–2015 and 2016–2017), the project document, the annual progress reports, the workshop and meeting reports, the reports from technical assistance missions, the evaluation survey and national and regional energy policies and strategies, among other documents. A list of the documentation reviewed during the evaluation is presented in annex 2. 3.1.2 Interviews and surveys with key stakeholders and beneficiaries 15. In February and March 2019, the evaluator conducted ten semi-structured interviews with strategic reporting entities at the programme, national and regional levels.4 A list of interviewees is included in Annex 3. The semi-structured interview protocols are presented in annex 4. 16. In addition, a Spanish-language e-survey was conducted, with the assistance of ECLAC, of additional key beneficiaries and cooperating stakeholders including officials of energy and hydrocarbon directorates; other national relevant stakeholders, such as universities, non-governmental organizations (NGOs) and private sector entities; members of the SICA energy coordination unit and energy integration bodies; and collaborating agencies and organizations such as UNDP, IDB, IRENA, OLADE, UNIDO and GIZ. The survey was open from 21 February to 15 March 2019. In total, 74 complete responses were received to 365 successfully sent invitations, giving an overall response rate of 20%. Annex 5 lists the questions asked and the response options offered. 17. The data collected were compiled and analysed using the evaluation matrix. Triangulation of the information was applied to all the data collected through documentation review, interviews and e-survey responses. 3.2 CHALLENGES AND POTENTIAL LIMITATIONS 18. The review was comprehensive in terms of scope, collecting data from different sources through separate instruments and using a robust data analysis method. The only limitations were timing and resources. While interviews provided detailed insights from some stakeholders and the e-survey provided some insights from a wide range of stakeholders, this evaluation is not informed by in-depth interviews with a large number of stakeholders. In addition, the assessment is not informed by direct observation, as no country visits were conducted. A broader number of in-depth, in-person, in-country interviews would likely have provided more insights. In any case, these limitations are typical of regional-scale evaluations. More importantly, the data collection instruments used ensure sufficient evidence has been collected and the data analysis methods used ensure the findings, lessons and recommendations are evidence-based. 4 17 people were reached out. FINAL ASSESSMENT REPORT 16 4. FINDINGS 4.1 RELEVANCE 4.1.1 How in line were the objective, activities and outputs delivered with the priorities of the targeted countries? Level of alignment between the project (objective, EAs and activities) and national needs and problems when it was developed and during the implementation 19. The project is very well aligned with the needs and problems of participating countries. The development and energy policies, strategies and plans of the eight targeted countries highlight the root problems mentioned in the project document, namely dependence on imported fossil fuels with high and volatile oil prices and limited development of renewable energy sources, access to modern energy services, and efficiency. In some countries, such as Guatemala, Honduras and Nicaragua, high reliance on biomass, particularly firewood, is also a problem. The development of clean, sustainable and more efficient national energy is therefore often high on the policy agenda in the eight project countries. 20. In particular, • In Belize, the National Energy Policy Framework drafted in 2012 aims to put the country on a path to energy efficiency, sustainability and resilience over the next 30 years. This will be implemented by the Energy Unit of the Ministry of Energy, Science, Technology and Public Utilities whose mission statement is to plan, promote and effectively manage the production, delivery and use of energy through Energy Efficiency, Renewable Energy, and Cleaner Production interventions for the sustainable development of Belize. • Costa Rica’s 2015–2018 National Development Plan has the following objective for the Environment, Energy, Seas and Territorial Planning sector: “meet the country’s energy demand through an energy matrix that ensures optimal and continuous supply of electricity and fuel, promoting the efficient use of energy so as to maintain and improve the country’s competitiveness”. Likewise, the 2015–2030 National Energy Plan considers energy efficiency an opportunity for sustainable development and includes a strategic objective to improve the participation conditions in the regional energy market. • The Salvadorian 2010–2024 National Energy Policy identifies the key challenge of reducing oil in the total national primary energy supply and in demand. The strategic lines of the policy are also very much concordant with the project.5 • In Guatemala, the 2013–2027 national energy policy considers scenarios for the international electricity and hydrocarbon market to develop a long-term strategic plan for efficient national energy production, trade and distribution. • The national vision of Honduras for 2010–2038 and its action plan for 2010–2020 seek to reduce its vulnerability to fluctuations in oil prices and stresses the need to increase the use of renewable energy to 80%. Thus, the country aims by 2022 to foster public-private investments in projects using renewable sources of energy, in energy efficiency projects and in projects to reduce emissions from deforestation and forest degradation (REDD). 5 These lines are diversification of the energy matrix and promotion of renewable energy sources; strengthening the institutional framework of the energy sector and user protection; promotion of a culture of efficiency and energy saving; extension of coverage and preferred social rates; innovation and technological development; and regional Energy Integration. FINAL ASSESSMENT REPORT 17 • In Nicaragua, the high dependency on oil derivatives is highlighted as the main sector issue in the Sustainable Energy for All Rapid Assessment and Gap Analysis report for 2012–2013. The energy policy implemented sought to expand energy generation using renewable resources and to change the generation matrix, as well promoting rural electrification. • Panama’s 2015–2050 National Energy Plan also advocates for more regional integration and energy security. The four pillars of its operational plan are well aligned with the project, since they address the same priorities: (i) universal access and reducing energy poverty; (ii) decarbonization of the energy matrix; (iii) energy efficiency and energy sobriety; and (iv) energy security. • In the Dominican Republic, the strategic lines of the 2010–2025 National Energy Plan are also aligned with the project.6 This is also in keeping with the Law on Incentives for Development of Renewable Sources of Energy, which was passed in 2007. 21. In addition, ECLAC had bilateral cooperation agreements with El Salvador and Costa Rica on the energy sector. Importantly, the energy policies, strategies and plans in place in the eight participating countries when the project was designed had been formulated and/or updated taking into account the commitments adopted in the Central American Sustainable Energy Strategy 2020. However, as stressed below, this strategy needed to be updated, which in turn entailed a need to update energy policies, strategies and plans in participating countries. Nevertheless, the targeted countries had limited technical capacity and knowledge to design and evaluate policies and measures for sustainable economic development in the energy sector, which is a highly technical topic. 22. Against this backdrop, 55% of the e-survey respondents confirmed that the project was fully aligned with the development priorities of the participating countries. That being said, some topics, especially biomass, did not generate the same level of interest in all countries.7 Level of national stakeholder consultation in the design process of the project 23. Interviews with national and regional stakeholders show that regional and national stakeholders were consulted during project design. Indeed, the of energy and hydrocarbon directorates were consulted and approved the project design, which was also presented to the SICA Presidents meeting and the SICA Council of Energy Ministers (CSME). In addition, the vast majority of e-survey respondents confirmed satisfaction with their involvement in the project design and implementation. Specifically, 42% of respondents said they were highly satisfied, and 53% were satisfied. Furthermore, 62% of respondents indicated that the project design and implementation was quite participative. 4.1.2 How in line were the objective, activities and outputs delivered with the priorities of SICA? Level of alignment between the project (objective, EAs and activities) and regional needs and problems when it was developed and during implementation 24. The project is well aligned with regional needs and problems, in two ways. First, the project contributes to strengthening regional integration in the energy sector (a highly technical topic where interdependencies between countries are important) when there was a need to do so because the 6 The main challenges identified are: (i) high oil prices in the international market; (ii) high participation of oil in the energy balance; (iii) low domestic production and dependence on energy imports; (iv) high energy costs; (v) inefficient use of energy; (vi) responsibility with the environment and environmental commitments in the production, transmission, distribution and use of energy. 7 The importance of biomass differs between countries. It is a crucial issue in Guatemala, Honduras and Nicaragua, a relevant but not pressing issue in the Dominican Republic and El Salvador, and less important in Belize, Costa Rica and Panama. FINAL ASSESSMENT REPORT 18 main regional road map was becoming obsolete. Central America has a long history of integration. As early as 1994, the presidents of seven participating countries —Belize, Costa Rica, El Salvador, Guatemala, Honduras, Nicaragua and Panama— signed an alliance (Alliance for the Sustainable Development of Central America (ALIDES)) to harmonize policies in order to achieve development goals, including energy sector policy. More recently, in 2007, the SICA countries signed the Central American Sustainable Energy Strategy 2020, which established clear guidelines for sustainable development of this sector. ECLAC supported development of the strategy. As noted, there was a need to update this strategy to cover the period up to 2030, in line with international agreements, particularly the 2030 Agenda for Sustainable Development that was adopted by all United Nations Member States, including SICA Member States, in 2015. 25. Second, and related to the first point, the project contributed to addressing problems and needs that required an integrated solution. ALIDES already focused on promoting renewable energy sources, energy efficiency and electricity interconnection.8 The formulation of the 2020 strategy was based on detailed studies of the region.9 On that basis, the strategy established goals to: (i) reduce dependence on hydrocarbons; (ii) increase the participation of renewable sources; (iii) reduce greenhouse gas emissions; (iv) increase electricity coverage; and (v) increase the efficiency of energy supply and demand. The DA project built on this strategy, thereby ensuring its alignment with regional needs and priorities. 26. Available evidence suggests that the project was relevant not only to the needs and problems of the region when it was formulated, but also to current regional needs and problems. In the 2018 Declaration of Belize, the governments of the SICA member states advocated an update of the Central American Sustainable Energy Strategy 2020 and highlighted the importance of promoting energy efficiency, renewable energies and access to energy. Interviews indicate that while there are discrepancies on certain topics (for example consolidation of electric vehicles), the project has contributed to participatory elaboration of the 2030 strategy. 27. In this context, 53% of the respondents to the e-survey stated that the project was fully aligned with the regional and SICA priorities and 45% responded that the project was mostly aligned. 4.1.3 How in line were the objective, activities and outputs delivered with international commitments on sustainable development? Level of alignment of the project activities with the Internationally Agreed Development Goals (IADGs) and SDGs (more precisely SDG7) 28. The project is line with Internationally Agreed Development Goals (IADGs), in particular with the 2030 Agenda for Sustainable Development and the resulting Sustainable Development Goals (SDG), which were approved in September 2016, after the project was approved. In this regard, the project team made a great effort to ensure that the project was aligned with the 2030 Agenda. The project is mostly tied to SDG 7, which involves three targets: (SDG 7.1) by 2030, ensure universal access to affordable, reliable and modern energy services; (SDG 7.2) by 2030, increase substantially the share of renewable energy in the global energy mix; (SDG 7.3) by 2030, double the global rate of improvement energy efficiency. Many project activities sought to contribute to this. Activity 1.1 consists of a regional workshop in coordination with SE4ALL in the Americas. Other activities focused on energy efficiency, development of renewable sources and universal access to energy (regional workshop, study tours, prospective studies of EA1), clearly contributing to SDG7. Moreover, the project specifically elaborated a regional roadmap 8 “In view of the serious situation facing the countries of Central America, it is imperative to formulate a policy and master plan for energy generation, marketing and consumption that encourages the use of renewable and alternative energy sources, energy-efficient programmes, and a common electric power network in Central America”. Alliance for the Sustainable Development of Central America, 1994. 9 They considered the socioeconomic context of the region, the existing energy system, greenhouse gas emissions and the institutional setting of the region. FINAL ASSESSMENT REPORT 19 for achievement of SDG7 in December 2018.10 In this respect, the mainstreaming of SDG 7 and the commitment to achieve the goal of universal access to modern energy services are the main results of activity A2.5, regarding the regional strategy. 29. In addition to SDG7, the project contributes to other SDGs. The project rationale clearly recognized that universal access to energy is a development driver and a priority to eradicate poverty. In this regard, the project was also aligned with SDG1 “No poverty”. Furthermore, the project contributed to SDG13 “Climate action”. The project carried out the first studies assessing the impact of climate change on the energy sectors in Latin America and presented the results in a seminar-workshop, thus providing insight to identify adaptation actions. The project also contributed to climate change mitigation, through studies on sustainable use of biomass, looking at cleaner energy alternatives (renewable energy sources and energy efficiency) and evaluations of greenhouse gas mitigations scenarios. Reducing deforestation also contributed to SDG15 “Life on land”. Furthermore, links can also be identified to SDGs 8, on decent work and economic growth; 9, on industry, innovation and infrastructure; 12, on responsible production and consumption; and 17, on partnerships for the goals. 30. The e-survey follows the same line, since 62% of the respondents considered that the objective of the project was fully aligned with the international commitments of the participating countries, and 33% considered it mostly aligned. 4.1.4 How aligned was the proposed project with the activities and programmes of work of ECLAC, specifically those of the subprogrammes in charge of implementation of the project? Level of alignment of the project activities with the ECLAC 2014–2015 and 2016–2017 programmes of work (PoWs), and more specifically with the subprogramme 12 strategies, activities and budget? 31. The project objectives were highly correlated with the EAs of the ECLAC PoWs for 2014–2015, 2016– 2017 and 2018–2019 with regards to the subprogramme on subregional activities in Central America, Cuba, the Dominican Republic, Haiti and Mexico. The project objectives are particularly consistent with EA2, which reads: “Increased technical capacity of the countries in the subregion to design, implement and evaluate policies and measures for economic development and structural transformation, trade and integration and sustainable development, including energy, agriculture and climate change”, as the project fostered the development of sustainable energy strategies and policies. The project activities have also been structured in alignment with the ECLAC PoWs for Central America, the Dominican Republic, Haiti and Mexico, for the bienniums 2014–2015, 2016–2017 and 2018–2019. They significantly cover the list of activities proposed by the PoWs for 2014–2015, 2016–2017 and 2018–2019, namely: Table 1 Main activities versus planned activities (according to the project design and extension) PoW 2014–2015 PoW 2016–2017 PoW 2018–2019 1. Expert groups, rapporteurs, depository services Ad-hoc expert group meetings (ii) A meeting of experts to consider the progress achieved in the Central American Sustainable Energy Strategy 1. Substantive servicing of meetings Ad hoc expert group meetings (i) A meeting of experts to discuss issues of energy policy and energy strategy: energy integration, energy transition and/or medium- and long- term sustainable energy development goals Servicing of intergovernmental and expert bodies and reports thereto (i) Meeting of experts to follow up on issues related to subregional energy integration, the
diversification of energy sources and its role in supporting the universalization of modern energy
services 10 ECLAC, Hoja de ruta regional para alcanzar la meta de acceso del objetivo de desarrollo sostenible, 7 December 2018. FINAL ASSESSMENT REPORT 20 PoW 2014–2015 PoW 2016–2017 PoW 2018–2019 2. Non-recurrent publications (ii) A study on relevant energy issues in the framework of the Central American Sustainable Energy Strategy (renewable sources, efficiency and access to energy, diversification of the energy matrix and/or policies for energy and climate change) 3. Other substantive activities Technical material (i) Maintenance, updating and expansion of the database on the hydrocarbon sector of the countries in the subregion (ii) Maintenance, updating and expansion of the database on the electric-power sector of the countries in the subregion 4. Advisory services (i) Provision of technical cooperation services to countries in the subregion and other relevant stakeholders that request it in areas relating to economic, environmental and social sustainability in the energy sector (ii) Provision of technical cooperation services, upon request, to regional cooperation institutions or mechanisms, including the Central American Integration System (CAIS) energy forums, the Meso-American Project and OLADE. (ii) A meeting of experts to discuss specific topics of the subregional energy agenda: sustainable energy (access to modern energy, renewable energy, and efficiency and rational use of energy) and/or issues associated with external fossil fuel dependence 2. Non-recurrent publications (i) A study to evaluate the status of the energy transition, energy integration and synergies with national energy agendas in selected countries of the subregion (ii) A study on the status and progress of sustainable energy goals (access to modern energy, renewable energy, and efficiency and rational use of energy) in selected countries of the subregion 3. Other substantive activities Technical materials (i) Maintenance and updating of the database on the hydrocarbon sector of the countries of the subregion (ii) Maintenance and updating of the database on the electric power sector of the countries in the subregion 4. Advisory services (i) Provision, upon request, of technical cooperation services to countries in the subregion and other relevant stakeholders in areas relating to economic, environmental and social sustainability in the energy sector (ii) Provision of technical cooperation services, upon request, to regional cooperation institutions or mechanisms, including the Central American Integration System (SICA) energy forums, the Meso-American Project and OLADE. (ii) Meeting of experts to follow up on issues related to Sustainable Development Goal 7 (energy) and its
links to other Goals of the 2030 Agenda for Sustainable Development, especially Goals 6 (water),
1 (poverty), 5 (gender) and 13 (climate change) Advisory services (i) 
 Provision, upon request, of technical cooperation services to regional cooperation institutions or
 mechanisms, including the Central American Integration System energy forums, the Mesoamerican
 Integration and Development Project, Sustainable Energy for All and the Latin American Energy Organization (ii) Provision, upon request, of technical cooperation services to countries in the subregion and other relevant stakeholders in areas relating to the energy sector Source: Desk review 32. The project took advantage of the momentum of other initiatives carried out by ECLAC. The Commission supported the development of the SICA Sustainable Energy Strategy, 2020. In addition, ECLAC had been working on energy efficiency in the region since 2012: first in Costa Rica and Panama, then in Salvador and Nicaragua, and then in Guatemala and Honduras. The project resumed this activity. One of the first tasks was to develop the methodology for energy efficiency plans. FINAL ASSESSMENT REPORT 21 4.2 EFFECTIVENESS 4.2.1 To what extent were the expected accomplishments met? 33. As noted in section 5.3.2, the results framework has significant shortcomings in terms of structure, indicators, baselines, targets and means of verification that make it difficult to measure the effectiveness of the programme. Level of achievement of the impact indicators from the results framework 34. Four of the five end-of-project targets of the impact indicators had been met by project completion.11 In one case,12 the target had been exceeded. The end-of-project target of one impact indicator had not been achieved.13 The text below provides details. • 70% of participants in the workshops confirm an increased understanding of sustainable energy management planning 35. Nine seminar workshops were held during the project. These workshops focused on geospatial tools for evaluating biomass energy potential and estimating firewood consumption scenarios, and the challenges of entry of large blocks of non-conventional or intermittent renewable energies (solar and wind). The average participation of the first type of seminars was 20 people, with more than 60 people attending the second type. About three quarters of the respondents to the exit evaluation considered that they had increased their understanding of sustainable energy management planning. The target therefore appeared to have been met. Interviews and the e-survey suggest that these workshops have had a significant impact on technical capacity and knowledge. Government officials also learned from project activities not directly oriented at training, through learning by doing, for example on energy balances. • Four of the eight participating countries have identified national problems relating to sustainable energy development and have approved actions to overcome them. 36. As planned, at March 2019, four countries had identified national problems relating to sustainable energy development and had approved actions to overcome them. Costa Rica, Guatemala and Honduras identified a need to update their Energy Efficiency Database (BIEE) to provide a baseline to measure progress with energy efficiency. This activity was undertaken in 2017–2018. In addition, Costa Rica, with support from the project, reviewed and updated its electricity tariff system to enable, among other things, an increase in the share of renewable energy that allows the country to reduce the use of hydrocarbons in the transport sector and boost electromobility. El Salvador identified the potential of biogas in rural communities and its role in the sustainability of agroindustry small business and developed a policy to address this. Moreover, in Guatemala, in 2018 the state electric utility approved a bidding process for the supply, construction, installation and start-up of photovoltaic solar systems up to a maximum of 110 megawatts, beginning in 2019. The decision to 11 Percentage of participants in the workshops that confirm an increased understanding of sustainable energy management planning; number of participating countries that have identified national problems of sustainable energy development and have approved actions to overcome them; number of countries that have included in their national energy policies and strategies sustainable development goals including universal access to modern energy service, renewable sources and energy efficiency that comply with international commitments to sustainable development and are consistent with the Regional Energy Strategy; and development and discussion of a draft proposal of the regional energy strategy for eventual adoption by the SICA Energy Ministers. 12 In particular, number of countries that have included in their national energy policies and strategies sustainable development goals including universal access to modern energy service, renewable sources and energy efficiency that comply with international commitments to sustainable development and are consistent with the Regional Energy Strategy. 13 In particular, number of beneficiary countries that have approved programmes and/or projects to promote sustainable regional energy development. FINAL ASSESSMENT REPORT 22 contract electricity supply through solar generation has the objective of diversifying the generator park and reducing the climatic risk to water resources. Activities carried out within the project with regard to non-conventional variable renewable energy and vulnerability to climate change form part of the background of this initiative. The final report also states that the project had contributed to all eight countries taking measures to promote and encourage the participation of renewable energies through the addition of renewable plants with a capacity of 1,910.8 MW.14 The target was therefore met. • At least four countries have included in their national energy policies and strategies sustainable development goals including universal access to modern energy services, renewable sources and energy efficiency that comply with international commitments to sustainable development and are consistent with the Regional Energy Strategy. 37. At January 2019, the project had achieved the following: • Belize updated its energy policy in 2017 and formulated its Growth and Sustainable Development Strategy • Costa Rica, which already had approved its 2015–2030 National Energy Plan in 2015, also announced a national decarbonization plan in 2019 with an objective of carbon neutrality by 2050. Also, in 2017 the country approved Law 9518, establishing a comprehensive legal framework to promote electric mobility, as a mechanism to spearhead decarbonization of the economy. In addition, as noted above, the country updated its electricity tariff system to enable, among other things, an increase in the share of renewable energy • El Salvador approved a norm of distributed generation with renewable energies in 2017 (and launched a tender in 2018) and approved technical guidelines on sustainable energy in 2019 • In September 2018 Guatemala approved the 2019–2050 National Energy Policy which proposes the development of energy efficiency plans • Nicaragua published a document that proposed the development of hydroelectric, wind, solar and geothermal projects: “Development policies and projects to enhance investment 2019–2021” • In April 2016 Panama approved its 2015–2050 National Energy Plan including a massive scaling-up of solar photovoltaic and wind energy combined with hydropower and other renewable energies and presented its “Renewables Readiness Assessment” identifying key actions to expand renewable energy development • Honduras formulated an energy policy proposal in 2017 that has not yet been approved 38. In addition, six of the SICA countries included sustainable energy topics in their Nationally Determined Contributions (NDCs) to the United Nations Framework Convention on Climate Change (UNFCCC) in 2016, including specific renewable energy and energy efficiency goals. The target of at least four countries having included sustainable development goals in their national energy policies has been exceeded, since more than four countries have reached that level. However, the direct contribution of the project to these achievements is complex, and these achievements cannot be fully and exclusively attributed to the project. • Five beneficiary countries have approved programmes and/or projects to promote sustainable regional energy development 14 According to the DA progress report 2019, the average share of renewable energies in the subregion grew from 52.1% in 2015 (when the project began) to 60.2% in 2017. DA progress report, January 2019. FINAL ASSESSMENT REPORT 23 39. The final report refers to four aspects. However, it is not certain that all of them can be considered programmes or projects. Two of the mentioned elements can be considered projects. The project contributed to the approval of a project to promote sustainable regional energy development in El Salvador. In this country, the project supported the National Energy Council in identifying the potential of biogas in rural communities. The construction of the first biodigester started in the second half of 2018, and it was yet to be inaugurated in early 2019. In addition, in July 2018 the Green Climate Fund (GCF) approved US$ 20 million for the Energy Efficiency in Public Buildings in El Salvador project, which was approved in 2017 by the NAMA Facility, and will be executed through the Development Bank of El Salvador. Activities carried out by the project in energy efficiency supported the development of this NAMA. Besides, the final report mentions the creation of the SICREEE, which was approved by the eight SICA countries, and has also been supported by UNIDO. According to the final report, this centre is a project. However, a centre and a project are usually different, primarily because projects are timebound and centres are not supposed to be linked to the timeframe of a particular project. The final report also mentions a bidding process for the supply, construction, installation and start-up of photovoltaic solar systems up to a maximum of 110 megawatts, but a bidding process is not a project. Although not mentioned in section 2 on the review of indicators of achievement, the final report claims that SICA initiated discussions with the United Nations Environment Programme (UN Environment) and a Central American multilateral bank and the SICA for a proposal to replace inefficient equipment that will be presented to the GCF. Since only two programmes or projects can be clearly confirmed, the target was not met. • Draft proposal of the regional energy strategy developed and discussed with national directors of energy and hydrocarbons for eventual adoption by the SICA Energy Ministers 40. By December 2018, a draft of the 2030 Central American Sustainable Energy Strategy had been developed and discussed with the national directors of energy and hydrocarbons and the Sectoral Council of Energy Ministers of SICA countries. The target was therefore met. The process included identification of regional priorities by the countries and discussion of progress and preliminary results with the energy and hydrocarbons directors of the countries. In fact, the interviews revealed that the project had a great impact on making coordination possible and improving its technical aspects. Importantly, an action plan was designed, identifying what needs to be done by the countries. 4.2.2 To what extent was the overall goal of the project achieved? Level of progress on technical capacity on sustainable energy planning and management of key stakeholders at the national and regional levels 41. From the results of the interviews, the level of progress was assessed as being moderate to high. Most stakeholders considered that the knowledge transfer was needed, efficient and relevant. The exchange of experiences was also considered valuable. The content and quality of the presentation of the trainings was also reportedly quite relevant and good. As per the survey, this was not only useful for government officials, but also for academia and the private sector (industry). Government officials learned from training and other project activities. However, the practical exercises received a more modest evaluation since only 37% of respondents scored them as excellent, while 53% rated them as quite good and 11% as improvable. Some interviewees also highlight that it may be difficult to put into practice some of the knowledge gained due to resistance at the political level. Number and quality of national policies and strategies incorporating contributions from the project 42. As noted in section 5.2.1 above, a number of laws, policies, strategies and regulations advancing sustainable energy were approved in participating countries during project implementation. As mentioned, these achievements cannot be fully and exclusively attributed to the project. In some countries, including Guatemala, Panama and Costa Rica, although the project contributed to identifying problems, there is no evidence that it contributed to national policies and strategies, namely because the existing legal framework already integrated the topics discussed. In some of these countries, such as Panama, the project did not contribute greatly to energy policy development, FINAL ASSESSMENT REPORT 24 but increased awareness of its importance. In most countries, the project contributed to establishing a baseline, identifying problems and the potential of renewable energy. In Belize, the country updated its Energy Policy in 2017 and formulated its Growth and Sustainable Development Strategy, to which the ECLAC project contributed. Number and quality of regional policies and strategies incorporating contributions from the project 43. According to interviewees, the support provided by the project at the regional level was significant. From 2015 onward, the project supported SICA in updating the regional energy strategy for 2020–2030 through assessments, baseline information and technical assistance that informed the decision-making and negotiation process. This was important, given the new international commitments, particularly the 2030 Agenda for Sustainable Development, the fact that the 2020 strategy had not been followed up with actions and monitoring, and the limited availability in the region of the specific technical capacities and knowledge that are needed to advance regional sustainable energy planning. The project enabled formulation of a strategy with a shared vision and common regional objectives that respected national diversity in the different roadmaps to achieve these objectives. An action plan was then designed, identifying what needed to be done by the countries. 44. In addition, with support from the project, the countries approved the first four regional standards on energy efficiency (Central American technical energy efficiency regulations for commercial refrigerators and freezers, and inverter-type and split-type air conditioners), during the seventh meeting of the SICA Council of Energy Ministers in June 2018. A specialized working group of SICA has continued to discuss other regional energy efficiency standards (including for engines, pumps and lighting). These actions will have an impact on SDG 7.3 on energy efficiency. 45 Moreover, the project contributed to SICA countries signing a proposal for a “regional alliance for the universalization of modern energy and decarbonization services of the SICA countries” in December 2018. The Alliance will take the form of a collaboration framework to coordinate regional efforts to achieve universal access to modern and sustainable energy services, increase the participation of renewable energy sources and improve energy efficiency. Number and type of regional coordination mechanisms that have been improved 46. Available information reveals that the project significantly improved regional communication and coordination. For instance, the project strengthened the work of the ECLAC technical groups, provided support for organization of in-person and virtual meetings of the Council of Energy Ministers and the meetings of the directorates. This support included ECLAC actively participating in presentations, facilitating meetings, contributing to travel and accommodation costs of participants and inviting participants on behalf of SICA. 47. According to the stakeholders interviewed, the assistance provided not only improved knowledge and capacities, but also the decision-making process at the regional level. In this regard, the key was the institutionalization of the SICA Sectoral Council of Energy Ministers (CSME).15 Established as a legal body of the SICA, the decisions adopted by the Council shall be binding for the eight countries of SICA. The aforementioned regulations were discussed and approved in regional project meetings. As noted above, not only was a draft 2030 regional energy strategy developed, four technical regulations have also been approved in the Council of Energy Ministers. In addition, the eight SICA countries approved (in December 2018) the creation of the Renewable Energy and Energy Efficiency Centre (SICREEE). 15 This was achieved through a positive recommendation on the “Rules of Organization and Operation” of the Council by the Executive Committee of SICA (22 May 2018 in Santo Domingo, Dominican Republic) and the respective final adoption by SICA Energy Ministers (22 August 2018, Santo Domingo, Dominican Republic). With adoption of this instrument, CSME has complied with the legal provisions of the Central American Integration Protocol and has been constituted —according to international law— as a legal body of the Central American Integration System (SICA). FINAL ASSESSMENT REPORT 25 4.2.3 How has the project contributed to enhancing the ECLAC programme of work/priorities and activities? Evidence of changes in the ECLAC programme of work, priorities and activities that can be attributed to the project 48. By supporting SICA countries in achieving SDG 7, the project contributed to progress on ECLAC priorities. In particular, the project managed to obtain the approval of the eight countries on universal access to modern energy services, and their commitment to achieving this goal, in addition to the existing goals on renewable energy and energy efficiency, for which progress is significant. 4.2.4 Did the project generate results not reflected in the results framework? Number and type of unplanned consequences from project activities or outputs to date 49. The observed unplanned consequences are all positive. • The project contributed to institutional strengthening of SICA, specifying the regulations of the Council of Energy Ministers • ECLAC also managed to give new impetus to the biomass working group, which was not initially foreseen by the project, and links SDG 7 to other SDGs, such as SDG 1 and SDG 15 • Moreover, the project achieved the first formal step towards electrical integration of Mexico with Central America, through the technical seminar and study tour conducted • Six of the eight countries presented their actions related to energy efficiency in the UNFCCC Conference of the Parties (COP) 21 in Paris in 2015 • The workshops facilitated networking, putting targeted countries in contact not only with each other, but also with Mexico and other international organizations (e.g. IRENA, CLASS, NIST) and experts 4.2.5 How satisfied are project beneficiaries with the services received? Level of satisfaction with the benefits received from the project 50. In general, as indicated in the interviews, the level of satisfaction with the benefits received from the project is very high. Generally, the respondents to the e-survey indicated that the activities contributed to increasing the technical capacities of national and regional stakeholders, increasing the capacities of non-governmental entities, identifying problems and developing actions, and integrating universal access to energy, renewable energy and energy efficiency into national policies and strategies. They also indicated that the project contributed to preparation and adoption of national roadmaps for sustainable energy and to identification of long-term objectives for the regional strategy. 4.3 EFFICIENCY 4.3.1 To what extent were the services and support delivered in a timely and cost-effective manner, according to the priorities established by the project document? Timing and sequence of outputs against work plan/Nature and impact of implementation bottlenecks 51. The project was extended one year, meaning that the planned two-year project became a three- year project. The extension was justified not only by the complexity of the project —working at the regional level and at the national level in eight countries, each with its own dynamic— but also by FINAL ASSESSMENT REPORT 26 administrative and geopolitical factors. In terms of administration, ECLAC adopted a new United Nations management system (Umoja) in the second half of 2015. The introduction of this new system led to delays, particularly at the beginning of the project (up to the second quarter of 2016), while administrative staff learned how to use it. The first disbursement was delayed (from September to December 2015) and the first regional activity in November 2015 could not be funded by the project as planned and was instead funded by other agencies (SE4ALL). Geopolitical tension between countries also delayed project execution at the regional level, postponing some regional integration meetings or leading to more meetings to ensure coordination, as decisions at SICA are made by consensus. Geopolitical tensions included fraught relations between some countries and their neighbours, or temporary withdrawal from the integration system: migratory crises (of Cubans and their passage through Costa Rica, in mid-2016, or the exodus to the United States from countries of the Northern Triangle of Central America, in the last quarter of 2018), tensions between Nicaragua and its neighbours (for various reasons) and with cooperation agencies, including the United Nations Agencies, as well as the position of Guatemala regarding the electrical interconnection with Mexico. 52. The project also had to deal with a changing institutional environment at the regional and national levels. SICA is led by a Presidency Pro Tempore (PPT), which is rotated every six months. During project execution, the PPT changed eight times.16 Some of these changes took place when tensions between and within countries were particularly high.17 The General Secretariat of SICA also underwent a change (in June of 2017). At the national level, during implementation of the project (September 2015–December 2018), six of the eight countries had presidential elections. Two countries changed their president and the other four opted for re-election.18 The rotating PPT contributed to delays. The change in the General Secretariat of the SICA did not affect the counterparts of the project, as the Secretary General of SICA remained in his position. The contribution of elections in participating countries to the delay was limited, as energy sector managers changed in just two of the eight countries. However, elections entailed handover periods, which led to some delays. 53. In addition, there were some delays related to specific issues at the country level. For instance, Panama experienced some difficulties in aligning national standards regarding energy efficiency of cooling systems, lighting and electric motors. Furthermore, the firm that was going to support the preparation of the SICA Energy Strategy had to withdraw at the last minute, due to an unfortunate and unforeseeable accident involving the team leader (the Chief Senior Energy Economist of the think tank Fundacion Bariloche). The guidance defining Development Account project timeframes should consider these causes of delays and the difficulty to anticipate and overcome some of them, allocating more than two years for project completion. Divergences between planned and actual activities and nature of changes 54. The final report claims that there were no major changes. However, available evidence suggests there were some changes to the original design of the project, apart from its duration, as noted above. There was an increase in the calls for cooperation regarding geospatial systems of biomass, initially estimated only for three countries and which finally took place in seven countries, all participating countries except Belize. Expanding support with the existing budget was positive. Moreover, there was a change in the nature of the activity A1.1. According to the project document, workshops organized under A1.1 would be technical in nature, considered as support for Activities 16 When the project started (September 2015) the PPT was held by El Salvador. In the subsequent six-month periods, the PPT rotated as follows: Honduras and Nicaragua (first and second half of 2016, respectively); Costa Rica and Panama (2017); Dominican Republic and Belize (2018), and Guatemala (first half of 2019). 17 Nicaragua held the PPT of SICA in 2016 when the country had tense relations with its Central American neighbours and with international cooperation donors. 18 Guatemala (January 2016); Belize (February 2016, re-election); Dominican Republic (August 2016, re-election); Nicaragua (January 2018, re-election); Honduras (January 2018, re-election), and Costa Rica (May 2018). El Salvador (elections were in February 2019) and Panama (elections will be in the second quarter of 2019), did not have changes in administration. FINAL ASSESSMENT REPORT 27 A2.1 and A2.4. However, the description of the activities in the final report shows that workshops conducted under A1.1 were political in nature and were not clearly different to workshops conducted under A2.1 and A2.4. Activities A2.1 and A2.4 were not sufficient to promote regional integration and activity A1.1 filled the gap. In total four meetings were needed, one at inception (in 2016), two mid-way (2017) and a final one (2018), but A2.1 and A2.4 only covered two in total (one each). This was not very positive, as the technical meetings planned under A1.1 were not conducted. Level of alignment between planned and incurred project costs and nature of divergences 55. At May 2019, some obligations were still outstanding, including the one related to this consultancy, which makes it difficult to determine final numbers. Some funds were redistributed with the introduction of Umoja. At May 2019, the project had spent more than planned on workshop logistics (contractual services and general operating expenses) and on travel of staff, country officials and experts. Increased costs were due to a substantial increase in project activities, including workshop seminars, training courses and regional meetings, an increase in the cost of air tickets and terminal expenses, and Umoja restrictions (which do not allow discounting the portion of the Daily Subsistence Allowance for meals and other services if these were provided by third parties). However, the project spent significantly less than planned on supplies of materials through the use of open-access software, other software and energy planning systems and Geographic Information System (GIS) platforms of specialized agencies and universities, such as IRENA, OLADE, the National Autonomous University of Mexico (UNAM) and the Stockholm Environment Institute (SEI). Overall, the increase in costs was greater than the savings and the project had to request technical cooperation funds from ECLAC to cover the final project meeting, at a cost of US$ 14,000. This has a complex reading. On one hand, the fact that the project required additional funding despite partners covering many aspects (see section 5.3.4) suggests the funds estimated during project design were insufficient for project completion. On the other hand, positively, the project was able to expand activities and cover the additional costs by leveraging funds from partners, including ECLAC. Level of alignment between planned and incurred project management costs and nature of divergences 56. The project document and the 2019 final report do not explicitly refer to project management costs. They use object class/code and descriptions. However, based on the detailed description of the object codes, the following codes can be considered project management costs: general temporary assistance, which refers to administrative assistance for implementation of project activities; and travel of staff, which refers to travel of United Nations Staff from the implementing entity for the purpose of project coordination, including seminars, workshops, study tours and internships, in support of project activities.19 Based on the data provided in the final report, considering these two budget codes, project management costs were 5% higher than planned. This is explained by an 8% increase in travel costs, as the allotment for general temporary assistance did not change. In any case, travel of staff was not only for the purpose of coordination, but also to deliver technical assistance. Evidence of use of sound financial and management practices for project execution and management 57. The project implemented measures to promote sound financial and administrative management. It sought complementarities with other programmes, projects or initiatives, the governments of participating countries and other development partners. As noted, the first regional meeting was funded by another institution (SE4ALL) and the second regional meeting was jointly financed by several institutions. There were also significant savings on supplies and materials through the use of open-access software, other software and energy planning systems and GIS platforms of specialized agencies and universities such as IRENA, OLADE, UNAM and SEI (Stockholm Environment Institute). Moreover, in order to limit the expenses related to rental equipment, the project attempted 19 According to the description in the project document, operating expenses could also be linked to project management costs, as they are related to communication in support of activities. However, according to the description in the final report, operating expenses are related to workshops (such as rental of equipment) (similarly to contractual services —catering) and are therefore not related to project management costs. FINAL ASSESSMENT REPORT 28 to organize most of the training activities in halls and auditoriums of energy ministries. Section 5.3.4 provides more details of complementarities. At ECLAC, there was key collaboration between the Energy and Natural Resources Unit (ENRU), administrative staff, the Programme Planning and Operations Division (PPOD), and the General Secretariat, which requested the extension from United Nations Headquarters. Evidence of the project using the technical, human and other resources available in participating countries to increase efficiency 58. The project made frugal use of human resources. Project management and execution mostly relied on the project manager, who was supported by one P-2 for 9 months and by an assistant. The project was also supported by other ECLAC staff at the Subregional Headquarters in Mexico and the Regional Headquarters in Chile. ENRU staff had quarterly meetings with the Research Coordinator and the Programme Officer of ECLAC Mexico and when required with the Director of ECLAC Mexico and PPOD at Headquarters. With additional tasks related to the regular programme of the office and emerging tasks,20 project staff worked very hard, often completing more than 10 hours a day and sacrificing weekends and holidays. However, while project staff were very knowledgeable and dedicated, the project would have benefited from higher levels of staffing, which did not happen owing to slow administrative response, difficulty in recruiting appropriately qualified personnel and unforeseeable circumstances.21 4.3.2 To what extent was the ME plan well-conceived and sufficient to monitor results and track progress toward achieving objectives? To what extent was the ME plan effectively and efficiently implemented? Existence of a clear and appropriate ME plan including scheduling, assignment of roles and responsibilities, and provision of adequate resources 59. The project document provides only a very generic monitoring and evaluation (ME) plan. The definition of roles and responsibilities is broad. According to the project document (p. 26), the Subregional Headquarters of ECLAC in Mexico, and particularly ENRU, would be responsible for monitoring and PPOD at Headquarters in Santiago, Chile, would be responsible for the supervision of a terminal external evaluation. Regarding the schedule, the project document only indicates, imprecisely, that the external evaluation will take place after completion of the project or during its last month.22 In this regard, the project document does not clarify what monitoring comprises: what type of monitoring reports have to be prepared (quarterly and annual, or only annual), with what content, when they have to be prepared, with whom, or how they have to be shared. Similarly, the project document does not clarify what supervision entails and does not provide any indications of the content of the terminal evaluation. The ME plan does not include a mid-term review, although this is reasonable because it is a short (two-year) project. 60. During implementation of the project, monitoring roles were further developed. As noted above, ENRU staff had quarterly meetings with the Research Coordinator and the Programming Officer of ECLAC Mexico to monitor progress and identify potential problems and solutions. In addition, PPOD 20 In addition to the activities of the project, ENRU officials carried out tasks contained in the Biannual Programme of ECLAC Mexico and emerging tasks related to requests for cooperation and assistance from countries and requests from ECLAC Headquarters. 21 Note that this statement does not assess the virtue of project staff but focuses instead on project staffing. ENRU staff acted in accordance with the Staff Rules and with the mandates of the General Assembly. 22 The project document states that the external evaluator will “discuss relevant issues with the beneficiaries involved. Ideally, he/she will travel during some of the last workshops related to this project to take advantage of the presence of the countries and, hence, be able to dialogue with them as well”. Note that the word “ideally” introduced uncertainty on when the terminal evaluation would take place. FINAL ASSESSMENT REPORT 29 conducted monitoring videoconference meetings at least twice a year with the project implementation team to review financial and substantive execution. 61. The project document allocates resources to the ME plan. No resources are directly allocated to monitoring. Resources allocated for the terminal evaluation are rather low compared to the resources typically allocated for terminal evaluations: the project document allocates a total of US$ 18,000 (US$ 12,000 in fees and US$ 6,000 in travel expenses) for this regional-level terminal evaluation. While country-level terminal evaluations are usually allocated between US$ 25,000 and US$ 30,000, regional-level terminal evaluations are on occasion budgeted US$ 100,000. While the percentage of total project resources allocated to the terminal evaluation (3.4%) is quite reasonable —the total budget is also small— the limited resources budgeted for evaluation can compromise the depth and breadth of data collection and therefore the soundness of the evaluation. According to Umoja’s philosophy, additional resources should not be allocated for ME. This compromises the independent nature of the evaluation and could represent a monitoring challenge for staff who already have too much non-project related work. Existence of appropriate performance indicators, and adequate baseline information 62. The project has one objective, two expected accomplishments (EA) and nine main activities. The structure of the objective and expected accomplishments is a little confusing, as the scope of the project is somewhat unclear as regards scale (national (specific countries) or regional (SICA)) and content (types of capacity). The objective (to strengthen the capacity of the eight SICA countries for sustainable energy solutions) relates to the national level, with no reference to the regional level. EA1 refers to the national and regional levels, while EA2 explicitly refers to the national level, although its activities also cover the regional level. In terms of scale, the objective and the EAs are therefore not fully aligned. Regarding the definition of capacity, EA1 refers to knowledge and understanding on sustainable energy management and planning and EA2 refers to capacity to formulate sustainable energy strategies and policies and comply with international commitments on sustainable development. EA1 seems to be included in EA2, as increased capacity to formulate strategies and policies implies increased knowledge and understanding. With their emphasis on capacity, the objective and EA2 also look quite similar. Moreover, alignment with international commitments applies to both EAs and not only to EA2. In this respect, it would have made more sense to distinguish between scales (national and regional) and/or types of capacity (knowledge or technical capacity and strategies and policies or planning instruments). In particular, it would have made more sense to have one EA at the national level and one EA at the regional level, distinguishing between types of knowledge, or having one EA on knowledge and another EA on planning tools, distinguishing between the national and regional levels. 63. This is even clearer when analysing the logical framework in more detail. This includes indicators, means of verification and risks/assumptions for the EAs. The logical framework does not include indicators, baselines, targets or means of verification for the objective. It does not identify outputs and therefore does not provide indicators, baselines, targets or means of verification at that level. The logical framework includes five sets of indicators and means of verification at EA level: two for EA1 and three for EA2. As a result, the allocation of indicators of achievement (IAs) to EAs is not coherent. In terms of scale, IA1.1 refers to both the national and the regional level; IA1.2 and IA2.1, to the national level; IA2.3, to the regional level; and IA2.2 has a more hybrid scope referring to impacts at the national level that also have an impact at the regional level. IAs focusing on the same scale are allocated to different EAs. The structure is clearer in terms of types of capacity, although the lack of clear references to the scale makes it more difficult to understand the links. For instance, impacts on identification of national problems and approval of actions to overcome them are considered an IA (IA1.2) of EA1, while impacts on incorporating sustainable development goals into national energy policies and strategies are considered an IA (IA2.1) of EA2, when not only do they refer to the same scale (the national level), but the two types of capacity development are clearly linked (IA1.2 focusing on knowledge and IA2.1 focusing on planning tools). FINAL ASSESSMENT REPORT 30 64. All the indicators are targets rather than indicators. An indicator would be formulated as “number of or percentage of”, while a target would be formulated as “X number of or Y percentage of”, in both cases with a time reference. The logical framework does not provide baselines. The project document does not provide mid-term targets. This is appropriate, because it is a short project (two years). 65. The sets of indicators and means of verification are inadequate. In IA1.1, the means of verification does not specify exactly when the satisfaction survey of workshop participants will take place. A survey immediately after workshops would not provide sound information on relevant increases on understanding, as this should be measured at least some time after the workshop to determine whether the increased understanding is still available. In IA1.2, the source of verification (contact with project beneficiaries) is not sound, as proper verification would require a comprehensive desk review to confirm whether actions have been approved to overcome the identified problems. The IA1.2 indicator is also vague, as it is unclear what the term “action” entails. In IA2.1 the means of verification is also inappropriate, as verifying whether sustainable development goals have been included in national energy policies and strategies would require reviewing those policies and strategies. Stakeholders are not a reliable source of information in this case and programmes of work of energy ministries are different to policies or strategies. The IA2.2 indicator is vague, as it is unclear what programmes or projects are. In this case, the means of verification seems appropriate. The IA2.3 indicator is somewhat vague, as preparing a draft is different to discussing it. The means of verification is not appropriate, as verifying this indicator would require desk review of the draft regional strategy and minutes of SICA meetings. Moreover, the target for IA1.2 seems low. Targets for IA2.1 and 2.2 could probably also be higher. Proportion of executed monitoring budget against planned monitoring budget 66. As noted above, the project document did not allocate resources for monitoring. Types, number and quality of reporting materials submitted a) correctly and b) on time 67. Annual progress reporting follows the outline of DA projects. This includes a brief project table (project title, DA project code, executing entity, reporting period, implementation rate and important issues); a summary of achievements to date; a section for reviewing performance indicators (EAs) and activities; a section on the challenges/problems encountered and the actions taken to solve the issues; a supplementary funding table; a section on revisions; a work plan for remaining activities; additional information; and financial information, per object class. 68. Overall, the template is comprehensive and useful. However, there is room for improvement in certain aspects. It would be valuable to add a section showing not only the work plan for the upcoming year, but also the extent to which the activities planned for the previous year were completed on time. The table for reviewing activities has a column (in the 2016 and 2017 reports) indicating status (cancelled, delayed, not yet started, in progress, completed). This is useful, but it is important to note the length of delays, if any, as activities may be both delayed and in progress. The financial information provides cumulative numbers. It would be good to also have information for each year, EA and activity. It is difficult to analyse expenditure in a manner that links spending with project results. 69. Progress reports were produced in a timely manner. Annual progress reports were prepared for 2015, 2016 and 2017. The 2015 Annual Progress Report has some shortcomings, in terms of the problems encountered and reporting only on the indicators and activities where work has been done. The 2016 and 2017 Annual Progress Reports were fully completed and provide a good level of detail, reporting on all indicators and all activities. In the 2017 Report, it is not clear which comment corresponds to each indicator. In 2019, the project did not produce a 2018 Annual Progress Report, but instead produced a final report, following the template mentioned above, with the same gaps. The final report is comprehensive and provides a good summary of the execution of the project. FINAL ASSESSMENT REPORT 31 Number, type and quality of project management responses to issues raised in ME reports Table 2 Challenges encountered and actions taken to solve them Challenge encountered Annual Progress Report Action(s) taken to solve it Implementation and further stabilization of the new management system Umoja 2016 Learning the new system, communication and sharing lessons learned with colleagues. More attributions or delegations of authority within Umoja have been requested for ECLAC Mexico officials, in order to expedite Umoja processes. 2017 Learning the new system, communication and sharing lessons learned with colleagues. This issue has been overcome, but it caused delays at the beginning of the project (especially during the second quarter of 2016). Selection and recruitment of personnel for ECLAC Mexico ENRU 2015 Decisions and actions are in the hands of United Nations central review bodies and ECLAC headquarters authorities 2016 Due to the mobility of staff in ECLAC Mexico the ENRU had only one Professional-category staff member for a period of over one year, which limited the speed of implementation of project activities. Accelerated recruitment processes and relocation of Professional-category staff contributed to inclusion of two new officials in the unit: one P-2 in September 2016 and one P-3 in October 2016. 2017 Selection and recruitment remain a problem, due to the regrettable and sad death of a P-3 colleague (July 2017). The position is vacant and in the recruitment phase. In April 2017, the Programming Assistant (G-5) of the Energy and Natural Resources Unit retired. The recruitment process has been finalized. The new official was to start in February 2018 Weakness and specific problems of the counterparts in SICA countries 2016 The project focused on constant communication with the countries in order to address this challenge; however, the results were not successful in all cases. The Presidency Pro Tempore of SICA was held by Honduras during the first half of 2016. Despite a slow response from the authorities of this country, the first meeting was held with the directors of energy and hydrocarbons (in Roatán, Honduras, in May 2016). In the second half of 2016, the Presidency Pro Tempore of SICA was held by Nicaragua. The centralization of cooperation activities (and in some cases a tightening relationship with some donors, including some United Nations agencies) affected the project. The tense relations of Nicaragua with its northern neighbours (from the area known as the Northern Triangle, especially Honduras and Guatemala) and its neighbours to the south (Costa Rica) also complicated the activities of the project. In 2017, the Presidency Pro Tempore of SICA moved to Costa Rica, a country with whom work was expected to flow without difficulty. 2017 The environmental and geopolitical tensions in the subregion, mainly from: (a) Nicaragua with its Central American neighbours and with international cooperation and donors; (b) the United States initiative with the countries of the area known as the Northern Triangle —Guatemala, Honduras and El Salvador, Plan of the Alliance for the Prosperity in the Northern Triangle— and; (c) the position of Guatemala regarding the electrical interconnection with Mexico. These three issues have generated tensions and have caused bureaucratization of the regional processes of subregional integration. All the above has complicated and delayed activities of the project. Happily, and at the request of SICA and with the support of ECLAC Executive Secretary and ECLAC Programme Planning and Operations Division, the project execution period was extended until December 2018. Knowledge of the countries of the subregion and good communication with national counterparts, as well as an ongoing search for synergies with donors and the criteria for rational and efficient use of resources were the main strengths of the subregional office of the ECLAC in Mexico. FINAL ASSESSMENT REPORT 32 Challenge encountered Annual Progress Report Action(s) taken to solve it 2016 project work programme must be projected to the countries 2015 Mission to present the project work programme to new SICA Pro Tempore Authorities (the Honduras Ministry of Natural Resources and Environment) and to organize a regional meeting in March or April 2016, possibly in Tegucigalpa. Preparatory actions for 2016 activities 2015 Discuss cooperation frameworks with other agencies and cooperating institutions. Identify consultants and invite them to enter their personal history profile in the United Nations system of consultants and contractors (Inspira). Source: Annual Progress Reports 2015, 2016 and 2017. 70. As noted, the annual progress reports include a section on challenges encountered and actions taken to solve them. Reporting on these matters improved with time. Some of the aspects identified as challenges in the 2015 Annual Progress Report (for example, presenting the project to the countries and preparing 2016 activities) are in fact tasks to be completed. Apart from this, reporting on these matters is good, identifying challenges as they emerge23 and monitoring actions taken and their impact. The final report summarizes the challenges and actions taken to solve the issues. 71. Overall, the project faced three main challenges. The first challenge was implementation and further stabilization of the new management system Umoja, which was introduced in the second half of 2015. As noted in section 5.3.1, this caused delays at the beginning of the project. In 2016 the project team learned about the new system and communicated and shared lessons learned with colleagues. More attributions or delegations of authority within Umoja were requested for ECLAC Mexico officials, in order to expedite the Umoja processes. By 2017, the issue had been overcome. Moreover, ECLAC and its officials were able to minimize the impacts of delays due to the introduction of Umoja in the execution of project activities, helping mobilize external resources to cover what the project could not cover in time. 72. The second challenge was limited personnel for ECLAC Mexico ENRU. Due to the mobility of staff in the ECLAC Mexico Office, ENRU had just one professional-category staff member for a period of over one year (in 2016), which, as noted above, limited the speed of implementation of project activities. Accelerated recruitment processes and relocation of professional-category staff helped incorporate two new officials into the unit (one P-2 and one P-3) in September and October 2016. This challenge arose again in mid-2017. In April, the Programming Assistant (G-5) of ENRU retired. In July 2017, the P-3 passed away. A new G-5 started work in February 2018.24 The P-3 position was not filled. 73. The third challenge was weakness and specific problems of the counterparts in the SICA countries. As noted above, this included: a slow response from Honduras when it held the PPT of SICA in the first half of 2016; the centralization of cooperation activities (and in some cases a tightening relationship with donors, including certain United Nations agencies); the tense relations of Nicaragua with its northern neighbours (from the area known as the Northern Triangle, especially Honduras and Guatemala) and its neighbours to the south (Costa Rica) when it held the presidency in the second half of 2016; the United States initiative with the countries of the Northern Triangle —Guatemala, Honduras and El Salvador, Plan of the Alliance for the Prosperity in the Northern Triangle— and the position of Guatemala regarding the electrical interconnection with Mexico. These issues generated tensions and caused a bureaucratization of the regional processes of subregional integration. All the above complicated and delayed the activities of the project. The actions taken were to promote constant communication with national counterparts and to continuously search for 23 There is an exception. One of challenges was that the firm that was going to support the preparation of the SICA Energy Strategy withdrew at the last minute. This is not mentioned in monitoring reports. 24 The new official was hired in record time according to ECLAC records. FINAL ASSESSMENT REPORT 33 synergies with donors, building on the knowledge of ECLAC on the countries of the subregion. In addition, at the request of SICA and with the support of the Executive Secretary of ECLAC and ECLAC Programme Planning and Operations Division, the project execution period was extended until December 2018. 74. Some of the challenges could have been anticipated during project design. The launch of Umoja was probably planned well before its introduction in the second half of 2015, so it could have been anticipated and mitigation actions established in the project design. Surprisingly, however, the introduction of Umoja is not included in the risks foreseen in the project document and mitigation actions were not planned in advance. This caused problems that could have been avoided or mitigated.25 Similarly, while the death of an official could not be foreseen, the retirement of another official could have been dealt with earlier, ensuring an overlap of at least one month between the outgoing and incoming officers,26 and the general recruitment needs could have been addressed earlier. Again, there is no reference to human resource needs in the risk section of the project document. In contrast, the project document did foresee risks regarding instability in participating countries (such as elections) and the SICA presidency, and planned mitigation actions, which were then implemented, complementing them with some new measures.27 Tensions between countries and between countries and the United Nations system could have been foreseen in general, but the specific tensions faced could not have been predicted, given their nature, and specific mitigation actions therefore could not have been planned. The project document does not include this type of tensions in its risk analysis. 75. That said, the actions taken by the project to address the challenges encountered, including requesting a one-year extension, are generally adequate, as there is little room for action to address some of the challenges encountered, particularly tensions between and within countries, which are very politically sensitive.28 4.3.3 The flexibility and responsiveness of ECLAC to meet the requirements of the Project and the needs of the countries involved, reducing or minimizing the negative effects of externalities. Number of monitoring missions of ECLAC and meetings held 76. The ECLAC Mexico Subregional Office conducted monitoring, as per the project document. However, as indicated above, the project document gives very little information about project management structure or monitoring and reporting aspects. The annual reports and the final report provide limited information on monitoring. The chief of the ENRU in ECLAC Mexico personally participated in no less than 13 workshops, seminars, training courses and special meetings. However, as noted above, there were human resources challenges regarding monitoring and the provision of technical assistance. At ENRU, the project manager and chief of the department was mostly working alone. He got some support from assistants and from one P-2 for several months, but he was responsible for the whole administration and execution of the project with limited support. During these missions, the Chief of ENRU also addressed non-project related topics —issues related to the Biannual Work Agenda of ECLAC Mexico. 25 Training on the new system could have started well before its actual introduction. While some learning would probably have been needed then, as there is always a transition, this could have been shorter with previous training. Moreover, the date of the first regional workshop could have been planned for later and/or administrative processes started earlier taking into account delays, to avoid not having funds to cover it and relying on external sources to do so. 26 The recruitment of the person to substitute the retired person started four months before the latter retired. However, there was no overlap to ensure proper hand over. 27 The project document considered missions at the beginning of each administration with the purpose of introducing the new authorities to the work of ECLAC and supporting the work of the regional energy integration bodies, facilitating good, rapid and timely communication with partners and authorities, and documenting the commitments of stakeholders, usually by consensus in meetings. During implementation, the project also had alternative venues for meetings to deal with these challenges. 28 However, the project should have found ways to refund ECLAC staff for project-related costs. FINAL ASSESSMENT REPORT 34 Evidence of the ECLAC management response/changes in project strategy/approach as a direct result of information in progress reports, missions or meetings 77. See section 5.3.2 above —ECLAC being an implementing entity of the Development Account, the project manager was an ECLAC official— the project did not have a separate project team. For that reason, the responses of the project were entirely ECLAC responses. Evidence of collection of lessons learned and good practices on project activities and dissemination to relevant stakeholders 78. Two project activities included the exchange of best practices, success stories, experiences or lessons of non-SICA countries. Activity A1.2 focused on the organization of South-South cooperation to solve specific national problems related to sustainable energy development. The project document identified three leaders: Mexico in energy efficiency, Brazil in universal access to modern energy services and Uruguay in renewable energy, particularly wind power. The project actually organized two workshops to exchange experiences. One focused on the incorporation of large amounts of non- conventional or intermittent renewable energy, presenting the experiences of Uruguay and Mexico (solar) to 60 people from the public and private sectors of SICA countries. The project also organized a workshop in the Dominican Republic for officials of the country on the use of biomass for cogeneration of heat and electricity, presenting experiences from South America —the countries are not specified.29 In addition, the project planned to promote and did in fact promote the exchange of best practices or lessons through study tours (activity A1.3). The project document mentioned a number of countries and topics.30 The project actually organized two study tours, one in 2017 on electrical interconnection (15 people from 6 countries) and one in 2018 on energy efficiency standards (15 people from 8 countries), both to Mexico. 79. However, the project did not have a strategy to systematically document and share lessons from SICA countries. The project team argues that experiences were shared informally in most regional meetings. While positive, this is not a sound exercise of documenting lessons, which entails critical analysis of the experiences, nor a systematic way of sharing them. Furthermore, the project did not have a strategy for systematically documenting and sharing lessons from the project. The project team argues that success stories were shared with the Director of ECLAC Mexico. However, lessons learned comprise more than success stories. It is critical to document what works well, what works less well and what does not work. Moreover, lessons have to be documented and shared with a wide range of stakeholders, not only internally at ECLAC Mexico. In this regard, the section on challenges and actions taken to solve them in monitoring reports provides some insight into lessons learned from the project, but the annual reports and the final report do not consolidate lessons. The final report states that lessons from the project on assessing the impact of climate change on the energy sector could be transferred to South America (Andean countries and Southern Cone subregion), but specific lessons are not collected and no dissemination strategy is provided. Similarly, the 2017 Annual Progress Report refers to a learning process regarding Umoja but does not indicate what the lessons were and how the project would share them. In this respect, all activities are reported through mission reports and meeting minutes, but these do not include a chapter specifically taking stock of the lessons learned and good practices. Lessons seem to have been collected only regarding regional workshops, for which evaluations were conducted, giving the opportunity for the participants to formulate recommendations for further workshops, but these have not been compiled or shared. 29 The project did not finally contribute to the exchange of Brazil’s experience on universal access to modern energy services. The exchange on biomass was not planned in the project document. 30 Possible topics and countries to be visited for study tours include: energy efficiency (Mexico, the United States and Uruguay); universal access to energy (Brazil and Uruguay): strategies for developing renewable sources (Denmark, Germany, Spain and Uruguay), and biofuels (Brazil and Colombia). FINAL ASSESSMENT REPORT 35 80. The project manager wrote a text on energy integration in Central America that was included in a 2018 commemorative publication for the seventieth anniversary of ECLAC.31 The chapter covers energy at ECLAC, several historical periods (1950–1979, 1980–1996 and 1997–2018) and pending tasks. While the chapter provides a very useful historical overview and some good insight into the way forward, it does not document the lessons learned through the integration process32 nor, more specifically, the lessons learned through implementation of the project on which this evaluation focuses. The project team has stated that ENRU will document and share lessons after project completion. While this is positive (see the recommendations section), this should have been done as part of implementation of the project. 4.3.4 Were there any complementarities and synergies with other work being carried out by beneficiary countries, ECLAC or other development partners? Evidence of efforts to optimize synergies and avoid duplications with the other activities implemented by beneficiary countries 81. Available evidence shows that the governments of Costa Rica, the Dominican Republic and Panama provided in-kind co-financing for the project. The Government of Costa Rica covered the expenses of the logistics of a meeting of energy ministers. The Government of Panama covered the expenses of the logistics of another meeting of energy ministers and provided co-financing for a workshop training seminar on the incorporation of non-conventional or intermittent renewable sources (solar and wind).33 The Government of the Dominican Republic covered the expenses of the logistics of a seminar-workshop and technical course on evaluation of biomass energy potential. There is no additional evidence of complementarities of project activities with other activities implemented by beneficiary countries. Evidence of efforts to optimize synergies and avoid duplications with the other activities implemented by ECLAC 82. ECLAC provided non-project related funds to the project. In particular, ECLAC (Regular Programme of Technical Cooperation) provided US$ 14,000 for the final activity, that is, the regional seminar of energy ministers. In addition, care was taken to align most of the activities of ENRU and other units and divisions of ECLAC with the project. In almost all the missions carried out within the project, ENRU officers carried out coordination tasks related to the regular biannual programme of ECLAC Mexico, such as updating databases and information systems, coordination of technical assistance activities, meetings of experts, studies and publications, and synergies with regional initiatives: the Mesoamerica Project, the Energy and Climate Partnership of the Americas (ECPA) and the sectoral councils of SICA, especially those related to the environment, water and health. Moreover, the first regional study on energy and climate change included the results of the vulnerability assessment of the electric power transmission systems carried out by the project. Furthermore, ENRU followed up in Guatemala, Honduras and Costa Rica the work conducted by the Division of Natural Resources and Infrastructure (DNRI) at ECLAC in South America. 31 Ventura, Victor Hugo (2018): Energy in Central American Integration, in Martinez Piva, Jorge Mario (Ed.): Achievements and Challenges in Central American Integration. Insights from ECLAC, ECLAC, Santiago, Chile, pp. 297–332. 32 The chapters on mobility and climate change have sections on lessons learned, but the chapter on energy does not have a specific section on lessons learned. 33 The amount is not clear. The final report indicates US$ 70,000 but does not distinguish between the contribution from the Government of Panama and those from other stakeholders (IRENA and GIZ) that also contributed to this workshop. FINAL ASSESSMENT REPORT 36 Evidence of efforts to optimize synergies and avoid duplications with different initiatives developed by other development partners (such as the SE4ALL Initiative) 83. In accordance with the project document, the project promoted collaboration with other agencies, initiatives and regional institutions active in the energy sector. In total, the project mobilized US$ 172,500 in in-kind contributions from development partners to support project activities. Synergies with OLADE were drawn on in three workshops, one on regional transport systems’ vulnerability to climate change and two on energy planning and policies (including evaluation of energy scenarios), with a total in-kind contribution of US$ 25,000 (US$ 16,000 for the first workshop). OLADE and ECLAC also collaborated on preparation of the update study on the situation of energy efficiency in the region. The project also collaborated with SE4ALL on expert meetings on energy efficiency plans and regional energy efficiency standards, based on an agreement between ECLAC, IDB and UNDP.34 In addition, the project collaborated with IDB on the above-mentioned meetings and four other meetings (energy ministries, directors of energy and hydrocarbons of the SICA countries). Furthermore, the project raised funds from GIZ for the above-mentioned four meetings and workshop training seminar on the incorporation of non-conventional or intermittent renewable sources. In addition, the project was backed by IRENA, which supported a seminar on non-conventional and intermittent renewable energies. As well as these institutions, the project also raised funds from the French Environment and Energy Management Agency (ADEME), under the regional programme “Indicators Database for Energy Efficiency”, also supported by GIZ. This included workshops, meetings and publications.35 Additionally, the Mexican Agency for International Development Cooperation (AMEXCID) and institutions from Mexico’s energy sector supported a study tour to the country. In most of the cases in-kind collaboration included logistic costs of organization of meetings (meeting rooms and catering), and in some cases the travel costs of participants. 84. The project also cooperated with several United Nations agencies during implementation. For instance, the creation of a Renewable Energy and Energy Efficiency Centre (SICREEE) was supported by UNIDO. The project also contributed to launching discussions between UN Environment and SICA on a proposal to replace inefficient equipment and between ECLAC and the World Health Organization (WHO) to address issues related to mainstreaming of the 2030 Agenda with regards to health and energy. There was also some collaboration with FAO. 85. The e-survey shows that 50% of respondents considered the complementarity of the activities to be excellent and 42% of them considered it as quite good. In addition, 42% of respondents considered that the coordination of interventions to avoid duplications and optimize synergies was excellent. Of the remainder, 45% considered the coordination to be quite good and 13% thought it was improvable. 34 During the meeting in Washington DC on 23 February 2015, ECLAC, IDB and UNDP met to commit to implement strategic objectives on behalf of SE4All Americas. These include creation of knowledge products, help with planning for universal access to energy, coordination with national and international partners, monitoring of the status of SE4All in Latin American and Caribbean countries, policy analysis, and improved project preparation and access to finance for projects that support the goals of SE4All. 35 Planning meeting for the Energy Efficiency Indicator Database (BIEE), Bogota, May 2016; third Regional Meeting on the Energy Efficiency Indicator Database Programme, Panama, May 2016; Honduras National Energy Efficiency Monitoring Report, 2018; Guatemala National Energy Efficiency Monitoring Report, 2018; Costa Rica National Energy Efficiency Monitoring Report, 2018. FINAL ASSESSMENT REPORT 37 Table 3 Supplementary funding Institution Purpose Year Amount raised (In-kind) (US$) OLADE (i) Training Workshop on Regional Electric Transport Systems’ Vulnerability to Climate Change (San José, Costa Rica, 16 November 2015). OLADE covered: (a) the costs of the consultants who presented the results of the study of the electricity network’s vulnerability to climate change, and (b) the costs of the seminar logistics (meeting rooms and catering). (ii) two workshops on energy planning and policy 2015 25,000 SE4ALL and IDB Expert meeting on energy efficiency plans (San Salvador, El Salvador, 11–13 November 2015). SE4ALL and IDB covered the costs of the seminar logistics (meeting rooms and catering) and the participants’ travel costs. Owing to problems related to the first black-out of the Umoja management system ECLAC could not cover any of the expenses of this event (even the ECLAC official who participated in this event had to self-finance his mission). 2015 36,000 GIZ, BID and others Three meetings of directors of energy and hydrocarbons of SICA countries (San Salvador, El Salvador, November 2016; Roatán, Islas de la Bahía, Honduras, 11–13 May 2016, and Panama City, Panama, 29–30 November 2017). GIZ, IDB and other institutions covered the logistics costs of these three meetings (meeting rooms and catering) and the travel of some participants. 2016 and 2017 40,000 IRENA, GIZ and institutions of the Government of Panama Workshop training seminar on the incorporation of non- conventional or intermittent renewable sources (solar and wind). GIZ covered the logistics costs of these three meetings (meeting rooms and catering). IRENA covered the cost of participation of experts and lecturers from Europe, Australia and Africa and some participants from Central and South America. - 70,000 ECLAC (Regular Programme of Technical Cooperation) Final regional seminar of energy ministries 2018 14,000 Costa Rica and Panama Two Meetings of energy ministers (San José, Costa Rica, 12 May 2017 and Panama City, Panama, 1 December 2017). The Governments of Costa Rica and Panama covered the logistics expenses of these events (meeting rooms and catering). 2017 4,000 Dominican Republic Seminar-workshop and technical course on evaluation of biomass energy potential. The Government of the Dominican Republic covered the expenses of the logistics of these events (meeting rooms and catering) 2017 1,000 190,000 Source: Final report, 2019. FINAL ASSESSMENT REPORT 38 4.4 SUSTAINABILITY 4.4.1 What is the likelihood of project results being incorporated into future strategies and policies in participating countries? Existence of capacity-building follow-up strategy in the participating countries 86. As noted in section 5.2.1, the project provided training on biomass and renewable energies. As mentioned in that same section and in section 5.2.2, this training increased technical capacity, at least in the short-term. Government officials also learned about additional topics, such as energy balances, from other project activities. However, there is no evidence that the technical capacities that were built will remain and will increase in the future. The project document did not refer to any strategy to maintain and increase technical capacities once the project ends. The approval of a regional Action Matrix includes commitments to strengthen national capacities. However, this does not ensure that technical capacities will actually continue to be strengthened at the national level. Interviews suggest that countries do not have strategies in this regard. In some countries, trainees seem to be attempting to document processes and disseminate knowledge to peers, but knowledge documentation and transfer seems to take place on an individual rather than institutional basis. This is important not only because trainees tend to forget what they do not use regularly, but also because there is typically significant staff turnover in government institutions. Of the e-survey respondents, 68% considered that the project results with regards to capacity building were likely to be sustained, but the conditions for this to happen do not seem to be in place. Existence of legal mechanisms ensuring implementation of the sustainability policy and strategies in the participating countries/Perceived level of ownership of the policy documents and strategies developed in the participating countries 87. As noted in sections 5.2.1 and 5.2.2, the project has contributed to participating countries adopting laws, policies, strategies and regulations on sustainable energy. Progress on this front has been made in at least six participating countries (Belize, Costa Rica, El Salvador, Guatemala, Nicaragua and Panama). In another country, Honduras, an energy policy proposal was formulated but has yet to be approved. In addition, six of the SICA countries included sustainable energy topics in their NDCs. Interviews suggest that there is national ownership of these legal frameworks, which by nature have to be implemented. As noted in section 5.4.2 below, the SICA 2030 Energy Strategy, at the time of writing still a draft, will also set commitments that will have to be observed by member countries. This is important and aligned with the project document. The section on sustainability focuses on mechanisms for monitoring progress and compliance with the objectives of the energy policy, including establishing goals and indicators. Moreover, four regulations on energy efficiency were approved. 88. However, experience shows that legal frameworks are not always implemented, even when there is national ownership and monitoring frameworks exist. In some cases, this may be due to conflicts with other legal frameworks. Panama has a market approach to energy with a focus on security, whereby the government does not control what type of energy is used. For instance, the recent construction of large natural gas plants can affect the renewable energy potential that is already developed. While the progress on policy has been good, it is not clear whether it would actually be implementable. Implementing energy policies is also challenging in terms of the complexity of institutional arrangements. For instance, it requires coordination with customs. Implementation of the legal framework can also be compromised by limited availability of technical, institutional and financial means (see above and below, respectively). FINAL ASSESSMENT REPORT 39 Existence of institutional frameworks ensuring sustainability of the project results, including a knowledge-sharing platform, in the participating countries 89. As noted above, in 2018 the SICA CSME was constituted as a legal body of SICA. Its decisions must be observed by member states. This represents great progress at the institutional level, in terms of both regional integration and at the national level. However, at the institutional level, and at the national level, differences between countries are significant. While Costa Rica, Panama and El Salvador have solid and stable institutions, and there has been some progress in the Dominican Republic (a biomass network was created), the countries of Guatemala, Honduras and Nicaragua face challenges that could lead to delays in complying with regional and international commitments. In this regard, some informants argue that recent (2018–2019) changes of staff in managerial and executive positions in four of the countries compromise the sustainability of the project results. Level of dependence on future funding for the sustainability of national progress and likely availability of such resources 90. Interviews suggest that domestic funding is limited, not only for full roll-out of policies, but also for running pilot, demonstration projects that can catalyse investment. However, a number of development partners seem to be working in the sustainable energy space in the region and are likely to support SICA countries on this matter. As noted earlier, among other efforts, SICA has initiated discussions with UN Environment and a Central American multilateral bank for a proposal to replace inefficient equipment that will be presented to the GCF. The “regional alliance for the universalization of modern energy and decarbonization services of the SICA countries” could also help mobilize resources. Beyond these, efforts appear to have been limited to support SICA countries in developing project proposals —particularly, but not only, for the GCF— to mobilize resources to advance in implementation of the sustainable energy policies, strategies and plans supported by the project. It is not possible for the evaluator to assess the extent to which countries will be able to mobilize the resources needed to implement national sustainable energy policies. 4.4.2 What is the likelihood of regional cooperation efforts being sustained? Existence of capacity-building follow-up strategy at the regional level 91. In the e-survey, 71% of respondents considered that the project results with regards to capacity building at the regional level were likely to be sustained. However, as at the national level, a regional strategy to maintain and increase technical capacity on sustainable energy does not seem to be in place. Existence of legal mechanisms ensuring implementation of the regional strategy/Perceived level of ownership of national and regional bodies of the regional strategy developed 92. At the time of writing, a draft of the SICA 2030 Energy Strategy had been presented to CSME, including establishment of goals and indicators, and a regional framework of actions that includes 13 pillars and more than 200 actions. Moreover, four regulations on energy efficiency were approved. SICA member states will need to observe the 2030 Energy Strategy (once approved) and the energy efficiency regulations. The strategy is set to be approved, the regulations have already been approved by the member states, meaning that there is and will be ownership. Existence of an institutional framework ensuring sustainability of the project results, including a knowledge-sharing platform, at the regional level/Level of dependence on future funding for the sustainability of national progress and likely availability of such resources 93. As noted above, in 2018 the CSME was constituted as a legal body of SICA. Its decisions must be observed by member states. This constitutes great progress at the institutional level in terms of regional integration. However, interviews suggest that SICA may encounter challenges when coordinating implementation of the 2030 Energy Strategy, due to limited human resources and financial constraints. Regarding human resources, interviews suggest that the Secretariat of SICA has FINAL ASSESSMENT REPORT 40 limited capacity to organize meetings, provide technical follow-up to decisions made and support resource mobilization. On the financial side, while certain member states co-finance some of the costs of regional meetings, the limited availability of resources may compromise countries’ ability to attend in the medium term. The contributions of member states are likely to remain limited. As for the national level, a number of development partners seem to be working on the sustainable energy space in the region and are likely to support SICA on this matter. Indeed, two follow-up meetings have been organized in the first half of 2019 (after completion of the project), with support from development partners, in particular the Dutch development aid organization Hivos.36 However, efforts seem to have been limited to support SICA in developing project proposals, particularly, but not only, for the GCF, to mobilize resources to advance in the implementation of the sustainable energy policies, strategies and plans supported by the project. Interviewees highlighted the possibility of SICA generating resources —OLADE is mobilizing resources by providing consultancy services. It is not possible for the evaluator to assess the extent to which SICA will be able to mobilize the resources needed to implement regional sustainable energy policies and support national sustainable energy policies. 4.5 CROSS-CUTTING ISSUES 4.5.1 To what extent did project design, implementation, and monitoring take into consideration human rights and gender issues? Evidence of assessment of possible gender inequality and specific human rights issues in energy 94. The project document does not provide a detailed assessment of possible gender inequality and specific human rights issues in energy access and production in Central America. It elaborates briefly on the links between universal access to modern energy services and gender equality but does not provide specific data on how this general link specifically manifests in the targeted countries. Nor do the project activities include a detailed assessment of gender inequality and human rights in the energy sector, although the work on biomass provides some insight. The scenarios evaluation in the SICA 2030 Sustainable Energy Strategy briefly considers the role of women in the energy sector in the sense that reducing fuelwood consumption would have a social multiplier effect through higher integration of women in the labour market and more schooling in children, two groups that would no longer be subject to the opportunity cost associated with collecting firewood. Gender gaps would be significantly reduced.37 In the regional action framework, gender equality is identified as one of the 41 possible areas for action to be covered by the baseline studies for the 2030 Energy Strategy. 38 However, the project did not factor in gender and human rights in the energy sector and has not produced a detailed assessment of these areas.39 One of the ENRU staff supporting the project was in charge of gender, among other tasks, but was only able to collaborate for 9 months, and no one took over responsibility for this topic. Existence of a strategy to advance gender equality and human rights, including the existence of a HR and GE strategy in the regional and national policies and strategies 95. The project document did not include a strategy, as such, to advance gender equality and human rights. However, it included an action to advance gender equality. According to the project 36 At the end of March 2019, the Energy Directors of SICA held their first meeting of 2019. In early May 2019 they had a second meeting to follow up on the issue of universal energy access. ECLAC has been invited to participate in both meetings, but they were supported by Hivos. 37 CEPAL, Estrategia energética sustentable SICA 2030. Evaluación de escenarios, November 2017. 38 CEPAL, Estrategia Energética 2030 de los países del SICA Parte IV, October 2018. 39 Note that (i) as explained, the human rights element of energy access was not assessed in detail; and (ii) while access to energy is certainly one critical aspect of energy and human rights, it is not the only one. For instance, there are also issues regarding indigenous communities and production of energy. FINAL ASSESSMENT REPORT 41 document, activity A1.1 on regional workshops would potentially cover gender and energy,40 with the support of OLADE, as gender and climate change is one of the pillars of cooperation between OLADE and ECLAC. In this regard, workshops under A1.1 should have been technical in nature, issuing proposals to be submitted for approval by meetings of directors (activities A2.1 and A2.4). However, actual workshops under A1.1 were very high-profile (meetings of directors of energy and hydrocarbons of SICA countries) and focused on broader aspects,41 with no specific emphasis on gender. The evaluator could not find any explicit references to a human rights or gender equality strategy in the regional and national energy policies and strategies, apart from the baseline studies mentioned above. As noted, the work on biomass has a positive impact in terms of gender equity. Interviews indicate that other development partners (such as GIZ) are working with SICA to establish a network of women experts on energy that can review the 2030 Energy Strategy and ensure it is gender-sensitive. Gender balance in participation in project workshops, seminars, meetings and study tours 96. A total of 95 women participated in the workshops, seminars, meetings and study tours offered during the project, representing around 16.6% of the participants. According to the results of the e- survey, there were no restrictions on women’s participation in the different project activities or any other discrimination. It should be also noted that just 19% of e-survey respondents were women. This indicates underrepresentation of women in the energy sector in Central America.42 In terms of project management, the project manager and the assistant were men, but ENRU prioritized the recruitment of women (four women and one man were recruited during project implementation). Stakeholders’ perception of the gender and human rights impacts of the project 97. In the e-survey, 58% of respondents stated that the project’s consideration of human rights and gender issues was quite high. However, the comments on the question indicate some assessment biases. Indeed, the respondents indicate that the workshops did not emphasize these issues, and that the technical topics addressed by the project do not explicitly cover gender issues, doing so only implicitly. 40 The proposed topics for the regional workshops (Activity A1.1) were (a) the SE4ALL pillars (universal access to modern energy services, renewable energy, and energy efficiency), and (b) other priority issues of sustainable energy development (adaptation to climate change, gender and energy and integrated energy development). 41 The first workshop discussed project activities, while the second one provided feedback on initial drafts of the energy strategy, approved the first four regional standards on energy efficiency and institutionalized the Sectoral Council of Energy Ministers. 42 Several factors, including cultural ones, explain this. Different ministries and institutions, including the education and energy authorities, share responsibility. FINAL ASSESSMENT REPORT 42 5. CONCLUSIONS Relevance 98. The project was closely aligned with the needs and problems of participating countries, as presented in their policies, strategies and plans. The project was also closely aligned with regional needs and problems. The project contributed to strengthening regional integration in the energy sector, which was needed because the main regional road map (the 2020 Central American Sustainable Energy Strategy, which was approved in 2007), which had informed national energy planning, was becoming obsolete. Available evidence suggests that the project was relevant not only to the needs and problems of the participating countries and the region when it was formulated, but also to current national and regional needs and problems. Regional and national stakeholders were consulted during project design. Project implementation was also participative. 99. The project is line with IADGs, in particular with the 2030 Agenda for Sustainable Development. The project is mostly in tune with SDG 7, and its three targets. In addition, the project contributes to other SDGs, namely SDG 1 on poverty, SDG 13 on climate change and SDG 15 on life on land, with some links to another four SDGs. 100. The project objectives and activities were highly correlated with the ECLAC PoWs for 2014–2015, 2016–2017 and 2018–2019 with regards to the subprogramme on subregional activities in Central America, the Dominican Republic, Haiti and Mexico. The project built on previous ECLAC work. ECLAC supported the development of the SICA 2020 Sustainable Energy Strategy and has been working on energy efficiency in the region since 2012. Effectiveness 101. The level of achievement of the end-of-project targets in the project results framework is high. Four out of five had been met by project completion.43 In one case,44 the target had been exceeded. One end-of-project target had not been achieved.45 102. Although, as explained below, the results framework presents significant challenges, this high level of achievement of end-of-project targets reflects the fact that the project promoted notable changes. The technical capacity of government officials and other stakeholders, such as academia and the private sector, increased as a result of training, study tours and other project activities. A number of laws, policies, strategies and regulations advancing sustainable energy were approved in participating countries during project implementation, although this cannot be fully and exclusively attributed to the project. The support provided by the project at the regional level was significant. The project supported SICA in updating the 2020 regional energy strategy to 2030 through assessments, baseline information and technical assistance that informed the decision-making and negotiation process. The project enabled a strategy to be formulated that had a common vision and common regional objectives, while respecting national diversity in the different road maps to achieve 43 Percentage of participants in the workshops that confirm an increased understanding of sustainable energy management planning; number of participating countries that have identified national problems of sustainable energy development and have approved actions to overcome them; number of countries that have included in their national energy policies and strategies sustainable development goals; and development and discussion of a draft proposal of the regional energy strategy for eventual adoption by the SICA Energy Ministers. 44 In particular, number of countries that have included in their national energy policies and strategies sustainable development goals including universal access to modern energy services, renewable sources and energy efficiency that comply with international commitments to sustainable development and are consistent with the Regional Energy Strategy. 45 In particular, number of beneficiary countries that have approved programmes and/or projects to promote sustainable regional energy development. The final report claims that this target has been met, but it is not clear whether two of the initiatives that are considered projects are actually projects. Details are provided in the main text. FINAL ASSESSMENT REPORT 43 these objectives. In addition, with support from the project, the countries approved the first four regional standards on energy efficiency and the creation of SICREEE. Moreover, the project contributed to SICA countries signing a proposal for a “regional alliance for the universalization of modern energy and decarbonization services of the SICA countries”, in December 2018. Furthermore, the project contributed to improving regional communication and coordination. The project strengthened the work of the ECLAC technical groups and provided support for the organization of in-person and virtual meetings of the Council of Energy Ministers and of the directorates. Most importantly, the project supported the process leading to institutionalization of the SICA CSME. Established as a legal body of SICA, the decisions adopted by the Council shall be binding for the eight countries of SICA. 103. By supporting SICA countries in achieving SDG 7, the project contributed to progress on ECLAC priorities. The project generated some unplanned positive outcomes, which are detailed in section 5.2.4. Interviews and the e-survey show that stakeholders were generally highly satisfied with the benefits received from the project. Efficiency 104. This complex project was extended one year (from two to three years). Administrative issues (the introduction of a new United Nations management system) and geopolitical tensions between and within countries were behind the extension. The rotating presidency of SICA contributed to the delay, while national elections in participating countries had a limited impact. Some activities were also adjusted during implementation. There was cooperation on geospatial biomass systems in seven countries, instead of in three, within the existing budget, which is positive. Moreover, there was a change in the nature of the activity A1.1, which covered gaps in activities A.2.1 and A2.4, which was not very beneficial. At May 2019, some obligations were still outstanding. Overall, cost increases exceeded savings and the project had to request technical cooperation funds from ECLAC to cover the final project meeting. Based on assumptions regarding what constitute project management costs, such costs were 5% greater than planned.46 105. The project put in place measures to promote sound financial and administrative management. It sought complementarities with other programmes, projects or initiatives, the governments of participating countries and other development partners. Internal collaboration at ECLAC was also important. Human resources for project management were limited, which was a problem rather than a virtue.47 106. The project document provides only a very generic ME plan. The definition of roles and responsibilities is broad, and the schedule vague. Monitoring roles were further developed during project implementation, with oversight from ECLAC Mexico and PPOD. The project document does not allocate resources for monitoring and allocates limited resources for a terminal evaluation. The results framework had significant gaps. The structure of the objective and expected accomplishments is a little confusing, as the scope of the project is somewhat unclear in terms of scale (national (specific countries) or regional (the SICA)) and content (types of capacity). The logical framework does not include indicators, baselines, targets or means of verification for the objective. It does not identify outputs and therefore does not provide indicators, baselines, targets or means of verification at that level. In this regard, the allocation of IAs to EAs is not coherent. All the indicators are targets rather than indicators. The logical framework does not provide baselines. The sets of indicators and means of verification are inadequate. 107. Annual progress reporting follows the outline of DA projects. Overall, the template is comprehensive and useful. However, there is room for improvement regarding information on completion of activities and finance. Overall, reports were produced in a timely manner, are complete and are of good quality. 46 This includes travel of staff and was meant to provide both coordination and technical assistance. 47 With additional tasks related to the regular programme of the office and emerging tasks, the very knowledgeable project staff worked very hard, often completing more than 10 hours a day and sacrificing weekends and holidays. FINAL ASSESSMENT REPORT 44 108. Reporting on challenges encountered and actions taken to solve them improved with time and is generally good. Overall, the project faced three main challenges: (i) implementation and further stabilization of a new management system; (ii) limited personnel for project management; and (iii) weakness and specific problems of the counterparts in the SICA countries. The first and second challenges could have been anticipated during project design, but the project document did not consider them or established mitigation actions. In contrast, the project document did foresee risks regarding instability in participating countries and SICA, and planned mitigation actions, which were then implemented, complementing them with some new ones. Specific tensions between countries could not have been foreseen. That been said, the actions taken by the project to address the challenges encountered are generally adequate, as there is little room for action to address some of the challenges encountered. 109. Monitoring at ECLAC was conducted as planned but, as noted, the plan was vague and human resources were limited. The project sought to exchange best practices, success stories, experiences or lessons from non-SICA countries through workshops and study tours. A text on energy integration in Central America was also published. However, the project did not have a strategy to systematically document and share lessons from SICA countries and the project. 110. The governments of three participating countries and ECLAC provided in-kind co-financing to the project. The project also promoted collaboration with other agencies, initiatives and regional institutions active in the energy sector. In total, the project mobilized US$ 172,500 in in-kind contributions from development partners to support project activities. Among other stakeholders, the project collaborated with OLADE, SE4ALL, IDB, IRENA, GIZ and ADEME. In most of the cases in-kind collaboration included logistic costs of organization of meetings (meeting rooms and catering), and in some cases the travel costs of participants. The project also cooperated with several United Nations agencies, such as UNIDO, during project implementation, and launched discussions with UN Environment and WHO. Sustainability 111. At the national level, stakeholders gained technical capacities on certain topics as a result of the project. However, countries do not seem to have put in place institutional mechanisms to maintain and increase technical capacities once the project has ended. This is important not only because trainees tend to forget what they do not use regularly, but also because there is typically significant staff turnover in government institutions. The project has contributed to participating countries adopting laws, policies, strategies and regulations on sustainable energy. There is national ownership of these legal frameworks, which by nature have to be implemented. The SICA 2030 Energy Strategy will also set commitments that will have to be observed by member countries. However, experience shows that legal frameworks are not always implemented, even when there is national ownership and monitoring frameworks exist. In some cases (such as Panama), conflicts with other legal frameworks may be a deterrent. Implementing energy policies is also challenging. The implementation of the legal framework can also be compromised by a limited availability of technical, institutional and financial means. Institutional progress at the national level is mixed, with regional advances contributing to national sustainability. Domestic funding seems to be limited, not only for full roll-out of policies, but also for running pilot, demonstration projects. However, a number of development partners seem to be working in the sustainable energy space in the region and are likely to support SICA countries on this matter. 112. Some technical capacities regarding sustainable energy have been also been obtained at the regional level, but a strategy to maintain and increase this capacity does not seem to be in place. The progress at policy level is likely to continue, as once the SICA 2030 Energy Strategy and the Action Matrix are approved, SICA member states will have to implement them, including the 13 pillars and more than 200 actions, in the same way that they need to implement the recently approved energy efficiency regulations. Considerable progress has been made at the institutional level with the constitution of the CSME as a legal body of the SICA. This status of CSME as a legal body of SICA will continue. Two follow-up meetings have been organized in the first half of 2019, with support from other development FINAL ASSESSMENT REPORT 45 partners. However, limited human and financial resources may compromise the ability of SICA to organize meetings, provide technical follow-up to decisions made and support resource mobilization for implementation in the medium term. Cross-cutting issues 113. The project did not factor in gender and human rights on the energy sector and has not produced a detailed assessment of these areas. The coverage of this during project implementation was limited: the project worked on biomass, which has positive gender equity impacts, and the regional Action Matrix proposes some work on this. Other development partners plan to review the SICA 2030 Energy Strategy and ensure it is gender-sensitive. Participation of women in project workshops, seminars, meetings and study tours was limited (16%). Recruitment at ENRU prioritized women. While a majority of survey respondents stated the project’s consideration of human rights and gender issues was quite high, comments clarify that this consideration was somewhat implicit. FINAL ASSESSMENT REPORT 46 6. LESSONS LEARNED 114. National sustainable development planning is a complex task that requires specific technical capacities and knowledge that are not always available in developing countries. Development partners can be instrumental in supporting this type of planning. 115. Regional integration systems, such as SICA, have a key role to play in supporting sustainable development planning at the national level, particularly on topics that are highly technical and where interdependencies between countries are important, such as energy. 116. It is important to update national and regional policies, strategies and plans, including those related to energy, when there are major changes in international development commitments, such as the 2030 Agenda for Sustainable Development and its corresponding SDGs. 117. It is key to consider previous work when selecting an executing agency. Project delivery benefited from ECLAC experience on energy and regional integration in Central America. 118. Regional sustainable development planning is a complex task. Like national planning, it requires specific technical capacities and knowledge that are not always available in developing regions. In addition, it requires significant coordination and negotiation. The institutionalization of coordination mechanisms is a factor of success. 119. Unplanned results are often crucial in development projects. One of the key benefits of workshops and study tours are the networks that are established between countries, development partners and consultants. 120. Administrative changes should be planned in advance, establishing actions to reduce likely delays to the extent possible.48 121. Regional integration is subject to instability and conflict between and within countries and within the regional integration mechanism. Some of these factors can be foreseen and mitigation actions planned and executed in advance, but there is little room for action regarding some other factors. This should be factored in when defining timeframes for regional projects. 122. Sustainable development planning at the country and regional levels may require significant funds. While complementarities with other projects should always be encouraged, each project should have enough funds to implement its main activities regardless of other projects, in case financial collaboration with them cannot be undertaken. 123. Project management requires sufficient human resources. Availability of staff has to be analysed and dealt with in advance, particularly when recruitment processes are drawn-out. In this regard, human resources should ensure a one-month overlap between outgoing and incoming officials when a person is retiring. 124. ME has a key role to play in promoting effective and efficient project delivery. ME plans need to be well developed, with clear roles and responsibilities and schedules and sufficient staff and budget. Results frameworks need to be clear and use specific, measurable, achievable, relevant and time-bound (SMART) indicators, baselines, targets and means of verification. 48 In this context, UN funded and executed projects should ensure by all means that project staff do not fund project- related activities with their personal financial resources. FINAL ASSESSMENT REPORT 47 125. Reporting should be timely and complete, including level of accomplishment of activities and financial management, indicating expenditure not only by budget code, but also per expected accomplishment and activity. 126. Projects relating to regional integration should systematically promote the exchange of lessons and best practices not only from countries outside the region, but also within the region and from the project itself. 127. For sustainability it is crucial that countries and regional integration institutions define institutional mechanisms to maintain and increase capacity after development assistance projects end. 128. The implementation of policies, strategies or plans improves when there are monitoring frameworks and teams. Implementation can however be compromised by conflicting legal frameworks, complex institutional arrangements and limited technical and financial capacities. Projects should address these potential barriers, to the extent possible. Development partners have a role to play in supporting developing countries in mobilizing international resources, particularly for pilot, demonstration projects that can catalyse investment. 129. For regional integration to be sustained (and regional strategies implemented and followed) sufficient human and financial resources are needed. Development partners have a key role to play in this. 130. Project design needs to be informed by a detailed gender and human rights assessment. Therefore, project documents need to explicitly and directly address identified barriers, to the extent possible. This should be done during project implementation. FINAL ASSESSMENT REPORT 48 7. RECOMMENDATIONS 131. Recommendation 1. ECLAC should draw lessons from project implementation and systematically disseminate them. 132. As discussed in paragraphs 79 and 80, the project did not systematically document and share lessons from SICA countries and the project. ECLAC should fill this gap. The exercise should consider the lessons presented in section 7 of this evaluation, but should also go beyond that. ECLAC should distinguish between two types of lessons. Some of the lessons would be on project design and implementation, from the project itself. Some other lessons would be on sustainable energy planning from participating countries and SICA. The dissemination strategy should distinguish between the audiences for these two types of lessons. Lessons on project design and implementation should be shared with development project designers and implementers at ECLAC, the Development Account and other development partners, as well as with consultants supporting these institutions on project design. Lessons on sustainable energy planning should be shared with energy practitioners in participating countries, SICA, other countries covered by ECLAC, other institutions in Latin America and the Caribbean, and other regions of the world. 133. Recommendation 2. ECLAC should consider the lessons learned from this project in the design and implementation of future projects. 134. As noted in paragraphs 79 and 80, lessons from the project were not the systematically documented and shared. As highlighted in recommendation 1, ECLAC should fill this gap. In addition to documenting and sharing lessons, ECLAC should make sure that the lessons learned from this project are actually used in the design and implementation of future projects. This is particularly important for projects focusing on sustainable energy, but should also apply to other topics. A way of ensuring this would be reflecting lessons in guidelines to be followed in project design, such as a checklist of compliance with good practices. 135. Recommendation 3. ECLAC should explore the possibility of promoting the institutionalization of regional bodies on other topics at SICA and in other regional integration systems on energy and other topics. 136 As noted in paragraph 47, one of the greatest contributions of this project has been the institutionalization of the CSME, which as noted means that decisions adopted by the Council shall be binding for the eight countries of SICA. The institutionalization of regional bodies would also contribute greatly to regional integration on other topics at SICA and on other regional integration systems on energy and other topics. ECLAC should explore the possibility of promoting this approach on other topics at SICA and in other regional integration systems that it supports, such as the Andean Community of Nations (CAN), the Andean Integration System (SAI), the Southern Common Market (MERCOSUR) and the Caribbean Community (CARICOM), on energy and other topics. 137. Recommendation 4. ECLAC should support SICA countries and SICA in mobilizing resources to advance in the implementation of the sustainable energy policies, strategies and plans supported by the project. 138. As noted in paragraphs 90 and 93, SICA countries and SICA have limited internal resources to advance in the implementation of the sustainable energy policies, strategies and plans supported by the project. External resources would be needed to ensure a significant implementation of said polices, strategies and plans. As noted in paragraphs 90 and 93, a number of development partners seem to be working in the sustainable energy space in the region and are likely to be ready to FINAL ASSESSMENT REPORT 49 provide some of the required funds. However, it is important to mobilize funds in programmatic, strategic ways rather than piecemeal. This would involve developing project proposals for international funds, particularly for the GCF, which is able to mobilize large amounts of funding in single proposals; other international funds, including the Global Environment Facility (GEF), the Adaptation Fund (AF), the Climate Investment Funds (CIF) and the DA should also be considered. SICA countries and SICA have limited capacity to develop these project proposals. ECLAC should support SICA countries and SICA in developing these proposals, using its expertise on project design, its partnerships with other development partners and its network of project design consultants. These proposals should build on existing initiatives including the “regional alliance for the universalization of modern energy and decarbonization services of the SICA countries”. 51 ANNEXES ANNEX 1 EVALUATION MATRIX ANNEX 2 LIST OF CONSULTED DOCUMENTS ANNEX 3 LIST OF PERSONS INTERVIEWED ANNEX 4 SEMI STRUCTURED INTERVIEW PROTOCOLS ANNEX 5 ONLINE SURVEY ANNEX 6 EVALUATOR’S REVISION MATRIX FINAL ASSESSMENT REPORT 52 ANNEX 1 EVALUATION MATRIX Evaluation questions Indicators Information source Data collection method A. Relevance 1. How in line were the objective, activities and outputs delivered with the priorities of the targeted countries? • Level of alignment between the project (objective, EAs and activities) and national needs and problems when it was developed and during the implementation • Level of national stakeholder consultation in the design process of the project • ProDoc • Other project documentation (e.g. progress reports) • Energy policy and planning documents in the eight countries • Government partners • SICA Council of Ministries of Energy and Energy Coordination Unit • ECLAC staff • Project coordinator • Private sector, Universities and CSOs • Development partners (UNDP, IADB, IRENA, OLADE, UNIDO, GIZ) • Desk review • Interviews • Surveys 2. How in line were the objective, activities and outputs delivered with the priorities of the SICA? • Level of alignment between the project (objective, EAs and activities) and regional needs and problems when it was developed and during the implementation • Level of regional stakeholder consultation in the design process of the project • ProDoc • Other project documentation (e.g. progress reports) • SICA policy and planning documents, including the 2020 Central American Sustainable Energy Strategy • Government partners • SICA Council of Ministries of Energy and Energy Coordination Unit • Energy Integration Bodies • ECLAC staff • Project coordinator • Private sector, Universities and CSOs • Development partners (UNDP, IADB, IRENA, OLADE, UNIDO, GIZ) • Desk review • Interviews • Surveys FINAL ASSESSMENT REPORT 53 Evaluation questions Indicators Information source Data collection method 3. How in line were the objective, activities and outputs delivered with international commitments on sustainable development? • Level of alignment of the project activities with the Internationally Agreed Development Goals (IADGs) and SDGs (more precisely SDG7) • ProDoc • Other project documentation (e.g. progress reports) • IADGs and SDGs • Government partners • SICA Council of Ministries of Energy and Energy Coordination Unit • ECLAC staff • Project coordinator • Private sector, Universities and CSOs • Development partners (UNDP, IADB, IRENA, OLADE, UNIDO, GIZ) • Desk review • Interviews • Surveys 4. How aligned was the proposed project with the activities and programmes of work of ECLAC, specifically those of the subprogrammes in charge of the implementation of the project? • Level of alignment of the project activities with ECLAC’s 2014-2015 and 2016- 2017 programmes of work (PoWs), and more specifically with the subprogramme 12’s strategies, activities and budget? • ProDoc • Other project documentation (e.g. progress reports) • DA project criteria • ECLAC PoWs • ECLAC staff • Desk review • Interviews B. Effectiveness 1. To what extent were the expected accomplishments met? • Level of achievement of the impact indicators from the results framework: − 70% of participants in the workshops confirm an increased understanding of sustainable energy management planning − Four of the eight participating countries have identified national problems of sustainable energy development and have approved actions to overcome them − At least 4 countries have included in their national energy policies and strategies sustainable development goals including universal access to modern energy service, renewable sources and energy efficiency that comply with international commitments to sustainable development and are consistent with the Regional Energy Strategy • ProDoc • Project progress reports • Activity reports and evaluations surveys (trainings, seminars, workshops) • Government partners • SICA Council of Ministries of Energy and Energy Coordination Unit • Private sector, Universities and CSOs • ECLAC staff • Project coordinator • Desk review • Interviews • Surveys FINAL ASSESSMENT REPORT 54 Evaluation questions Indicators Information source Data collection method − 5 beneficiary countries have approved programs and/or projects to promote sustainable regional energy development − Draft proposal of the regional energy strategy developed and discussed with national directors and Energy and Hydrocarbons for eventual adoption by the SICA Energy Ministers 2. To what extent was the overall goal of the project achieved? • Level of progress on technical capacity on sustainable energy planning and management of key stakeholders at the national and regional levels • Number and quality of national policies and strategies incorporating contributions from the project • Number and quality of regional policies and strategies incorporating contributions from the project • Number and type of regional coordination mechanisms that have been improved • ProDoc • DA project criteria • Project progress reports • Project planning documents (quarterly and annual work plans) • National and regional energy policy and planning documents • Government partners • SICA Council of Ministries of Energy and Energy Coordination Unit • Energy Integration Bodies • ECLAC staff • Project coordinator • Desk review • Interviews • Surveys 3. How has the project contributed to enhancing ECLACs programme of work/priorities and activities? • Evidence of changes in ECLAC’s programme of work, priorities and activities that can be attributed to the project • PoWs • Project progress reports • Project planning documents (quarterly and annual work plans) • ECLAC staff • Desk review • Interviews 4. Did the project generate results not reflected in the results framework? • Number and type of unplanned consequences from project activities or outputs to date • Project progress reports • Government partners • SICA Council of Ministries of Energy and Energy Coordination Unit • Energy Integration Bodies • Desk review • Interviews 5. How satisfied are project beneficiaries with the services received? • Level of satisfaction of the participating countries with the benefits received from the project • Level of satisfaction of regional stakeholders with the benefits received from the project • Perception of the quality of the supervision and guidance of ECLAC • Government partners • SICA Council of Ministries of Energy and Energy Coordination Unit • Energy Integration Bodies • Universities, CSOs and private sector • Interviews • Surveys FINAL ASSESSMENT REPORT 55 Evaluation questions Indicators Information source Data collection method C. Efficiency 1. To what extent were the services and support delivered in a timely and cost-effective manner, according to the priorities established by the project document? • Timing and sequence of outputs against work plan • Nature and total delays (in months) generated by implementation bottlenecks • Divergences between planned and actual activities and nature of delays • Level of alignment between planned and incurred project costs and nature of divergences • Level of alignment between planned and incurred project management costs and nature of divergences • Evidence of use of financially and management sound practices for project execution and management • Evidence of the project using the technical, human and other resources available in participating countries to increase efficiency
 • Existence of coordination mechanisms between the ECLAC and other cooperating agencies ensuring efficiency in delivering project outputs and coherence of response • Monitoring and reporting documents • Project planning documents (quarterly and annual work plans) • ECLAC staff • Project coordinator • Desk review • Interviews 2. To what extent was the ME plan well-conceived and sufficient to monitor results and track progress toward achieving objectives? To what extent was the ME plan effectively and efficiently implemented? • Existence of a clear and appropriate ME plan including scheduling, assignment of roles and responsibilities, and provision of adequate resources • Existence of appropriate (SMART) performance indicators, and adequate baseline information • Proportion of executed monitoring budget against planned monitoring budget • Types, number and quality of reporting materials submitted a) correctly and b) on time • Number of project management responses to issues raised in ME reports • Monitoring and reporting documents including financial reporting • Project planning documents (quarterly and annual work plans) • ECLAC’s staff • Project coordinator • SICA Energy Coordination Unit • Desk review • Interviews FINAL ASSESSMENT REPORT 56 Evaluation questions Indicators Information source Data collection method 3. The flexibility and responsiveness of ECLAC to meet the requirements of the Project and the needs of the countries involved, reducing or minimizing the negative effects of externalities (for example, those derived from important changes in the management of UN administrative processes). • Number of monitoring missions of ECLAC and meetings held • Evidence of ECLAC’s management response/changes in project strategy/approach as a direct result of information in progress reports, missions or meetings • Evidence of collection of lessons learned and good practices on project activities and dissemination to relevant stakeholders • Monitoring and reporting documents • Project planning documents (quarterly and annual work plans) • Government partners • ECLAC’s staff • Project coordinator • SICA Energy Coordination Unit • Desk review • Interviews 4. Were there any complementarities and synergies with other work being developed by beneficiary countries, ECLAC or other development partners? • Evidence of efforts to optimize synergies and avoid duplications with the other activities implemented by beneficiary countries • Evidence of efforts to optimize synergies and avoid duplications with the other activities implemented by ECLAC • Evidence of efforts to optimize synergies and avoid duplications with other initiatives developed by other development partners (e.g. SE4ALL Initiative) • ProDoc • Other project documentation • Government partners • SICA Council of Ministries of Energy and Energy Coordination Unit • Energy Integration Bodies • ECLAC staff • Project coordinator • Development partners (UNDP, IADB, IRENA, OLADE, UNIDO, GIZ) • Desk review • Interview • Surveys D. Sustainability 1. What is the likelihood of project results being incorporated in future strategies and policies in participating countries? • Perceived level of ownership of the policy documents and strategies elaborated in the participating countries • Existence of capacity building follow up strategy in the participating countries • Existence of legal mechanisms in place ensuring the implementation of the sustainability policy and strategies in the participating countries • Existence of institutional frameworks ensuring sustainability of the project results, including a knowledge sharing platform, in the participating countries • Level of dependence on future funding for the sustainability of national progress and likely availability of such resources • Project progress reports • Government partners • SICA Council of Ministries of Energy and Energy Coordination Unit • Energy Integration Bodies • ECLAC staff • Project coordinator • Development partners (UNDP, IADB, IRENA, OLADE, UNIDO, GIZ) • Desk review • Interviews • Surveys FINAL ASSESSMENT REPORT 57 Evaluation questions Indicators Information source Data collection method 2. What is the likelihood of regional cooperation efforts being sustained? • Perceived level of ownership of national and regional bodies of the regional strategy elaborated • Existence of capacity building follow up strategy at the regional level • Existence of legal mechanisms in place ensuring the implementation of the regional strategy • Existence of an institutional framework ensuring sustainability of the project results, including a knowledge sharing platform, at the regional level • Level of dependence on future funding for the sustainability of national progress and likely availability of such resources • Project progress reports • Government partners • SICA Council of Ministries of Energy and Energy Coordination Unit • Energy Integration Bodies • ECLAC staff • Project coordinator • Development partners (UNDP, IADB, IRENA, OLADE, UNIDO, GIZ) • Desk review • Interviews • Surveys E. Crosscutting issues 1. To what extent did project design, implementation, and monitoring take into consideration human rights and gender issues? • Evidence of assessment of possible gender inequality and specific human rights issues in energy access • Gender balance in participation to project workshops, seminars, meetings and study tours • Existence of a HR and GE strategy in the regional and national policies and strategies • Perception of stakeholders on gender impacts of the project • Perception of stakeholders on human right impacts of the project • ProDoc • Project progress reports • National and regional energy policy and planning documents • Government partners • SICA Council of Ministries of Energy and Energy Coordination Unit • Energy Integration Bodies • ECLAC staff • Project coordinator • Private sector, Universities and CSOs • Development partners (UNDP, IADB, IRENA, OLADE, UNIDO, GIZ) • Desk review • Interviews • Surveys FINAL ASSESSMENT REPORT 58 ANNEX 2 LIST OF CONSULTED DOCUMENTS • Programme of Work of the ECLAC System, 2014−2015, 2016−2017 and 2018−2019 • DA project criteria • Project document • Annual progress reports 2015, 2016 and 2017 • Final report • Workshops and meetings reports • Mission reports • Evaluation survey • National and regional energy policies and strategies • National and regional energy development scenarios when available • Ventura, Victor Hugo (2018): Energy in Central American Integration, in Martinez Piva, Jorge Mario (Ed.): Achievements and Challenges in Central American Integration. Insights from ECLAC, ECLAC, Santiago, Chile, pp. 297-332. FINAL ASSESSMENT REPORT 59 ANNEX 3 LIST OF PERSONS INTERVIEWED1 No. Name Position Institution Country Date 1 Victor Hugo Ventura Project Manager ECLAC - 28/02/2019 2 Fernando Diaz G. de P. Director of Electricity National Energy Secretariat Panama 28/02/2019 3 Carlos Iglesias 4 Armando Osorio 5 Giancarlo Alexander Guerrero Ministry of Energy and Mines Guatemala 28/02/2018 6 Gabriel Armando Velasquez Subdirector of Energy Ministry of Energy and Mines 8/03/2018 7 Luis Roberto Reyes Executive Secretary National Energy Council El Salvador 4/03/2019 8 Oscar de la Maza Director Directorate of Renewable Energy–Ministry of Energy and Mines Dominican Republic 8/03/2019 8 Laura Lizano Director Planning Secretariat Energy subsector Costa Rica 12/03/2019 9 Ryan Michael-Lee Cobb Director of Energy Ministry of Energy, Science and Technology and Public Utility Companies Belize 13/03/2019 10 Werner Vargas Torres Executive Director General Secretariat - SICA - 13/03/2019 1 17 people from the eight participating countries were reached out for an interview. FINAL ASSESSMENT REPORT 60 ANNEX 4 SEMI STRUCTURED INTERVIEW PROTOCOLS Government Stakeholders (Representative of ministries of energy or directorates) Chair of SICA Council of Energy Ministers Chair of SICA Coordination Unit ECLAC Staff Project Coordinator General How long have you been involved in the project and what is the nature of your involvement (specific activities)? X X X X X Relevance To what extent was the project aligned to with the priorities of your country/the targeted countries? Please explain X X X X X To what extent was the project in line with the priorities of the SICA? Please explain X X X X X Were the national and regional stakeholders consulted during the design process of the project? Please explain X X X X X How in line was the project with the international commitments on sustainable development (IADGs, SDGs, SDG7)? Please explain X X X X X To what extent was the project in line with the activities and programmes of work of ECLAC, specifically those of the subprogrammes in charge of the implementation of the project? Please explain X Effectiveness To what extent do you consider that the workshops conducted have increased understanding of sustainable energy management planning of Energy Officers? Please explain X X X X X Have you participated in any capacity strengthening or awareness raising activity of the project? How would you rate the relevance of these activities regarding your needs and expectations? To what extent have you found these training effective? Please explain X X To what extent has the project helped your country/the SICA countries identify the national problems of sustainable energy development and approve actions to overcome them? Please explain X X X X X To what extent has your country/ have SICA countries included in its/their national energy policy and strategy measures contributing to the sustainable development goals on universal access to modern energy service, renewable sources and energy efficiency? Please explain X X X X X FINAL ASSESSMENT REPORT 61 Government Stakeholders (Representative of ministries of energy or directorates) Chair of SICA Council of Energy Ministers Chair of SICA Coordination Unit ECLAC Staff Project Coordinator To what extent has your country participated in the development/have SICA countries developed any programs/or projects to promote sustainable regional energy development as a result of the ECLAC project? Have they been approved? X X X X X Has your country/have SICA countries developed national policies and strategies incorporating contributions from the project? How robust are these policies and strategies? Have they been approved? Please explain X X X X Has a regional energy strategy been drafted as a result of the project? How robust is it? Has it been approved by the SICA Energy Ministers? Please explain X X X X To what extent has the project contributed to improve the regional coordination mechanisms? Please explain X X X X X Has the ECLAC’ PoW been in any way revised as a result of the project? Please explain X Have there been any unintended results (environmental, social, economic - positive or negative) and what were they? X X X X X How satisfied are you with the benefits received from the project? X X How would assess the quality of the supervision and guidance from ECLAC? X X Efficiency Did the project implementation face any significant delays in terms of delivery of activities and disbursement? Which? Why? What were the implementation bottlenecks? X X Were any measures put in place to ensure/enhance cost and time effectiveness? To what extent did they enhance efficiency? Please explain. X X To what extent did the project use the human, technical and other resources available at country level to enhance efficiency? Please explain. X X Were ME roles and responsibilities and timing clear? Was the ME budget enough to conduct the necessary ME tasks? Please explain X X X Were the indicators SMART? Were baseline, targets and sources of verification robust? Please explain X X Was the ME plan effectively and efficiently implemented? What aspects could have been improved? Please explain X X X To what extent was technical and financial reporting timely and complete? Please explain X X X To what extent did management respond to issues raised in ME reports? Please explain X X X X FINAL ASSESSMENT REPORT 62 Government Stakeholders (Representative of ministries of energy or directorates) Chair of SICA Council of Energy Ministers Chair of SICA Coordination Unit ECLAC Staff Project Coordinator In your opinion was the oversight by ECLAC effective? Were any corrective actions taken in response to monitoring reports? Please explain X X X X To what extent was the project complementary to other existing interventions by beneficiary countries, ECLAC or development partners (e.g SE4ALL)? Were efforts coordinated to avoid duplication and optimize synergies? Please explain X X X X X Sustainability What conditions have been put in place by the project in your country to ensure the sustainability of its results after project end (ownership, capacity building follow up strategy, legal mechanisms, institutional frameworks, funding opportunities, etc.)? Do you believe they are sufficient? Please explain X X X X X What conditions have been put in place by the project in the region to ensure the sustainability of its results after project end (ownership, capacity building follow up strategy, institutional and organizational arrangement, political and social conditions, funding opportunities, etc.)? Do you believe they are sufficient? Please explain X X X X X In your opinion, to what extent are the activities and outputs from the project likely to continue after the end of the project and/or be replicated? Why? X X X X X Cross cutting issues Did the project take into consideration human rights and gender issues in its design, implementation and monitoring? How? Was it enough? What was the impact of the project at this regard? Please explain X X X X X Are there any lessons learned from the Project to be shared with other stakeholders in the region or the country? Please explain X X X X X FINAL ASSESSMENT REPORT 63 ANNEX 5 ONLINE SURVEY La CEPAL tiene el agrado de invitarle a participar en la encuesta de evaluación del proyecto de Fortalecimiento de la capacidad de los países de América Central en la elaboración de políticas y estrategias de energía sostenible, financiado por la Cuenta para el Desarrollo de las Naciones Unidas. Este proyecto fue ejecutado por la CEPAL entre Junio 2015 y Diciembre 2018 con el objetivo de: • Ampliar el conocimiento y la comprensión de los funcionarios de los ministerios de energía y de los organismos de energía del SICA en cuanto a la planificación y la gestión de la energía sostenible. • Reforzar la capacidad de los países del SICA en la formulación de estrategias y políticas de energía sostenible y cumplimiento con los compromisos internacionales de desarrollo sostenible. Los datos recogidos mediante esta encuesta serán tratados con la debida confidencialidad al contener datos personales, o sea protegiendo el anonimato de los encuestados. Completar la encuesta solo tomará unos 10 minutos. Esta encuesta estará disponible desde el XX/XXX hasta el XX/XX de 20191 1. ¿Para qué tipo de organización(es) trabajó usted durante el período en que participó en las actividades del proyecto? (Marque todas las opciones que correspondan) Organización gubernamental o institución pública de un país participante (Dirección de Energía e Hidrocarburos) / Organización de la sociedad civil (incluidas ONG, asociaciones y sindicatos) de un país participante / Institución académica o instituto de investigación de un país participante / Organización del sector privado de un país participante / Unidad de Coordinación de Energía del SICA / Entidad de Integración regional en energía / Agencia u organismo de desarrollo bilateral o multilateral (incluidos los organismos de las Naciones Unidas, Bancos de Desarrollo) / Otro (por favor especifique) 2. ¿Cuál era su posición al momento de su participación en las actividades del proyecto? Gerencia senior / Gerencia intermedia / Personal técnico / profesional / Consultor(a) / Personal administrativo / Otra (por favor especifique) 3. ¿Está satisfecho con su participación en el diseño del proyecto en general? Muy satisfecho Satisfecho Un poco insatisfecho Insatisfecho ¿Puede explicar su nivel de satisfacción? ______________________ 4. ¿Basado en su conocimiento del proyecto, el objetivo del proyecto (fortalecer la capacidad técnica del personal para que puedan diseñar políticas de energía para el desarrollo sostenible del sector de la energía y para lograr el objetivo del acceso universal a la energía) le parece alineado con las prioridades de desarrollo de su país/ de los países participantes? Totalmente Mayormente En cierto modo Para nada 1 The deadline was extended twice. The survey was open between February 21st and March 15th. FINAL ASSESSMENT REPORT 64 5. En qué medida fueron participativos el diseño e implementación del proyecto en su país/los países participantes? Muy participativos Bastante participativos Poco participativos Nada participativos ¿Puede explicar su respuesta? ______________________ 6. ¿Basado en su conocimiento del proyecto, el objetivo del proyecto le parece alineado con las prioridades regionales y las del SICA? Totalmente Mayormente En cierto modo Para nada 7. ¿Basado en su conocimiento del proyecto, el objetivo del proyecto le parece alineado con los compromisos internacionales de los países participantes sobre el desarrollo sostenible, en particular con los Objetivos de Desarrollo Convenidos Internacionalmente (ODCI) y los Objetivos de Desarrollo Sostenible (ODS, incluido ODS7 Energía Asequible y No Contaminante)? Totalmente Mayormente En cierto modo Para nada 8. ¿Ha participado Usted en un taller de reforzamiento de capacidad / de formulación de políticas o estrategia / en el desarrollo de estudios, o alguna otra actividad del proyecto? Sí No solo si la respuesta es afirmativa a la pregunta anterior: ¿En qué actividad(es) participó? ___________________________________ ¿Cómo calificaría los siguientes aspectos de la(s) actividad(es)? (Excelente / Bastante bueno / Mejorable / Muy mejorable) • Objetivo del taller / de la actividad(es) (pertinente con las prioridades de mi país) • Contenido del taller / de la actividad(es) y de las presentaciones • Ejercicios prácticos (pertinencia para mi trabajo) ¿Puede explicar sus respuestas? ______________________ 9. ¿En qué medida contribuyó el proyecto a aumentar la capacidad de los actores nacionales y regionales en la gestión y planificación sostenible de la energía? Contribuyó mucho Bastante Un poco No contribuyó ¿Puede explicar su respuesta? ______________________ 10. ¿En qué medida contribuyó el proyecto a aumentar la capacidad de las entidades no gubernamentales de desempeñar su papel de socialización de las policías energéticas? Contribuyó mucho Bastante Un poco No contribuyó ¿Puede explicar su respuesta? ______________________ 11. ¿En qué medida contribuyó el proyecto a identificar los problemas relativos al desarrollo de energía sostenible y desarrollar acciones para abordar esos problemas en su país / los países participantes? Contribuyó mucho Bastante Un poco No contribuyó ¿Puede explicar su respuesta? ______________________ 12. ¿En qué medida contribuyó el proyecto a incorporar aspectos de acceso universal a energía moderna, fuentes renovables y eficiencia energética en la políticas y estrategias nacionales? Contribuyó mucho Bastante Un poco No contribuyó ¿Puede explicar su respuesta? ______________________ 13. ¿En qué medida contribuyó el proyecto a la elaboración y aprobación de hojas de ruta para el desarrollo sostenible energético en su país / los países participantes? Contribuyó mucho Bastante Un poco No contribuyó ¿Puede explicar su respuesta? ______________________ FINAL ASSESSMENT REPORT 65 14. ¿En qué medida contribuyó el proyecto a aprobar proyectos en esa materia en su país / los países participantes? Contribuyó mucho Bastante Un poco No contribuyó ¿Puede explicar su respuesta? ______________________ 15. ¿En qué medida contribuyó el proyecto a la identificación de objetivos de largo plazo para la estrategia regional de desarrollo sostenible de energía? Contribuyó mucho Bastante Un poco No contribuyó ¿Puede explicar su respuesta? ______________________ 16. A nivel regional, ¿en qué medida contribuyó el proyecto a mejorar los mecanismos de coordinación en esa materia? Contribuyó mucho Bastante Un poco No contribuyó ¿Puede explicar su respuesta? _____________________ 17. ¿Cómo calificaría la coordinación del proyecto con las otras intervenciones existentes a nivel nacional y regional? (Excelente / Bastante buena / Mejorable / Muy mejorable) • Complementariedad de las actividades • Coordinación entre las intervenciones para evitar duplicación y optimizar las sinergias Muy alta Bastante alta Bastante baja Muy baja ¿Puede explicar sus respuestas? ______________________ 18. En su opinión, ¿cuál es la probabilidad que los resultados del proyecto se mantengan después de la terminación del proyecto? (es decir, un mayor conocimiento y capacidad de los diseñadores y creadores de políticas energéticas, una planificación de la gestión de la energía mejor informada a nivel nacional y regional, un mejor diálogo entre las oficinas gubernamentales, una mejor cooperación regional en materia de energía sostenible y el intercambio de experiencias) Muy alta Bastante alta Bastante baja Muy baja ¿Puede explicar su respuesta? ______________________ 19. En su opinión, ¿cuál es la probabilidad que los resultados del proyecto se mantengan después de la terminación del proyecto a nivel regional? (es decir, un mayor conocimiento y capacidad de los diseñadores y creadores de políticas energéticas, una planificación de la gestión de la energía mejor informada a nivel regional, un mejor diálogo entre las oficinas gubernamentales, una mejor cooperación regional en materia de energía sostenible y el intercambio de experiencias) Muy alta Bastante alta Bastante baja Muy baja ¿Puede explicar su respuesta? ______________________ 20. En su opinión, ¿el proyecto tuvo en cuenta de manera suficiente los derechos humanos y las cuestiones de género en su diseño e implementación? Mucho Bastante Poco Nada No tengo suficiente información ¿Puede explicar su respuesta? ______________________ FINAL ASSESSMENT REPORT 66 ANNEX 6 EVALUATOR’S REVISION MATRIX Evaluation of the DA Project 1415BD/ROA 312-9A “Strengthening the Capacity of Central American and Caribbean Countries in the Preparation of Sustainable Energy Policies and Strategies” Evaluation Report Feedback Form: Evaluation Reference Group GENERAL COMMENTS Report section (if applicable) Comment Evaluator’s response The assessment is very thorough and detailed, which reflects an excellent job of the evaluator. However, we believe that the document does not adequately dimension the following aspects: a) The regional dimension and the complexity of working simultaneously with 8 countries; b) The intrinsic complexity of the energy sector (8 specific situations for each country, each with its own dynamics); c) The geopolitical aspects (the SICA subregion suffered at least 3 major crises during the Project execution period); d) The UN Reform process and the adoption of the new UMOJA management system; e) The adoption (in September 2016) of the 2030 Agenda by the UN Assembly and the effort of the ENRU to fully incorporate that Agenda into the Project; f) The human resources of the ENRU (the smallest substantive Unit of ECLAC Mexico), which also had to face recruitment processes (both for mobility, as well as retirement and death of a colleague) during the Project execution period, and g) Others (an unfortunate accident that forced the temporary retirement of the main Senior Energy Economist of Fundación Bariloche FB, a think tank with which ECLAC had already reached agreements to work on the preparation of the Central American Energy Strategy). We believe that considering the dimensions and aspects above, will help to have a better overall evaluation of the Project. Points c, d and f were highlighted in sections 5.3.1, in particular pages 23-24, and 5.3.2, in particular pages 31-33. Point a seems to be related to point b: working simultaneously in 8 countries with their own dynamics. A reference to this has been added in paragraph 77. Points e and g provide additional information. References have been added in paragraphs 54, and 79 and 96, respectively. FINAL ASSESSMENT REPORT 67 SPECIFIC COMMENTS Paragraph number Comment Evaluator’s response 1. Executive Summary/8. Efficiency (page 2) “..but did not have a strategy to systematically document and share lessons from SICA countries and the project” Comment: In most of the meetings of the Project, in each topic, the starting point was always to know the experiences and the position of each country. In that sense, consensus was always sought to implement the projects activities. There is a lot of information that will be used by the ENRU team to document and share the experiences. Presenting the experiences, I assume orally, in most of the meetings is not a systematic exercise of documenting and sharing lessons. The project has now been completed. It is great that the ENRU will document and share experience after project completion, but this should have been part of the implementation of the project. Note as well that lessons and experiences are different: lessons imply a sound reflection of what worked well and less well in specific experiences. The issue of lessons learned is further discussed below. Adjustments have been made in the main text. However, the evaluator believes the finding provided in the executive summary (which has to be short) summarizes well the evidence-based discussion provided in the main text. 1. Executive Summary/9. Sustainability (page 2) “Participating countries do not seem to have put in place institutional mechanisms to maintain and increase the capacity gained as a result of the project once this has phased out” …” Besides, not much progress seems to have been made at institutional level in participating countries” Comment: Keep in mind that energy is long term. Unless it is a conjunctural problem, the implementation of regional mechanisms and solutions takes a long time (years). Complexity increases when the problem is addressed at the regional level (with many countries). Take the case of regional electricity integration as an example: a) between 1976-1980, ECLAC, with the participation of the countries, prepared the first pre-feasibility study for the Central American interconnection (6 countries); b) it was not until 1996 that the Presidents approved the Framework Treaty for the Central American Electricity Market (after 16 years of discussions and updating of studies); c) between 1997-2014 countries implemented the mandates of the Treaty (approval by the Legislative Assemblies, creation of regional entities, search for financing, studies of engineering and design of facilities, tendering, construction of infrastructure, regulatory harmonization, and approval of national regulations, all Note that the highlighted text refers to technical capacity and institutional mechanisms. The comment focuses on planning processes. The second part of the text has been adjusted to stress the relevance of CSME Regulation (which is further highlighted in the following paragraph). Information provided to the main text regarding this has also been reflected in the executive summary to ensure consistency. A justification is not provided to adjust the first part. FINAL ASSESSMENT REPORT 68 SPECIFIC COMMENTS Paragraph number Comment Evaluator’s response that took 20 years). In summary, the Central American electricity integration process, in its first phase, took 36 years (1980-2014). In this sense, the approval of the CSME Regulation (which gives life to that Ministerial Council, discussions that took place within the framework of project meetings), can be considered a MILESTONE, for the sustainability of the Project and for the follow-up and fulfilment of the SDG 7. The matrix of actions, approved in the framework of the project, forms the work agenda for the next decade. We suggested to modify this paragraph. 1.Executive Summary/10. Sustainability (page 3) “A strategy to maintain and increase gained capacity does not seem to be in place at regional level either” Comment: Within the framework of the project meetings, the countries approved a matrix of actions (including 13 major axes and more than 200 actions), which was one of the inputs for the preparation of the Strategy. The countries are using the above (the Strategy and the Regional Matrix of Actions) to work towards achieving the goals of the Strategy and the SDG 7. It will constitute the Regional Energy Agenda for the next decade. At the end of March 2019, the Energy Directors of SICA held their first meeting of this year (2019), with no sponsorship of the ROA Project. Between May 2-5 they will have a second meeting to follow up on the issue of universal energy access. ECLAC has been invited to participate in both meetings. The Dutch agency HIVOS has supported both meetings. We suggest the evaluator, refine and reformulate that paragraph, emphasizing as one of the strengths of the project, the Regional Matrix of Actions as a mechanism of consensus and prioritization. It has been clarified that the highlighted text refers to technical capacity, not to an institutional mechanism for prioritizing actions. Other sentences in the paragraph have been adjusted to highlight the Strategy and the Regional Matrix of Actions and the organization of follow up meeting with support from other institutions. 1. Executive Summary/10. Sustainability (page 3) “A strategy to maintain and increase gained capacity does not seem to be in place at regional level either” Comment: The axis 14 of the Matrix of actions is dedicated to the Strengthening of the Regional Institution of the Energy Sector. Without a doubt, it is a challenge for SICA. We suggest nuancing and reformulating this comment. The comment does not justify why the highlighted text (a finding, not a comment) should be adjusted. It has been clarified that the text refers to technical capacity. FINAL ASSESSMENT REPORT 69 SPECIFIC COMMENTS Paragraph number Comment Evaluator’s response 1. Executive Summary/ 10. Sustainability (page 3) “The project was not informed and has not produced a detailed assessment of gender and human rights on the energy sector” Comment: That as such (“detailed assessment of gender and human rights”) was not contemplated in the Project Document, therefore it was not addressed. Keep in mind that access to energy (the topic addressed in the Project) is considered (based on the 2030 Agenda) as part of basic human rights. Bear in mind that in all the activities there was participation of female officers representing the countries (in the second semester of 2017, the PPT was in charge of a woman, the vice-minister of energy of Costa Rica). Bear in mind too that, on behalf of ENRU, the recruitment of women has been a priority (during the project period, UERN recruited 4 women, one retired and another died, only one male was recruited, and it was a UN headquarters decision). The first female P-3 official recruited, had among her tasks entrusted to address the issue of gender, unfortunately, she could only collaborate for 9 months. We suggest softening the nuances of the comments and reformulating this paragraph. Please note that the executive summary has to be short. For that reason, adjustments have mostly been made in the main text (section 5.5). On the main points: The text indicates that a detailed assessment of gender and human rights was not provided nor contemplated as one the project activities in the Prodoc. This is a gap. Please note that women represented less than 17% of total workshops, seminars, meetings and study tours participants. You mention a woman made a presentation in the second semester of 2017, but this does not say much: what was the percentage of presentations made by women? Information on project structure (gender balance and a person in charge of gender for some time) has been added. Regarding human rights, while access to energy is certainly one critical aspect of energy and human rights, this is not the only one. A footnote has been added to explain this. On the executive summary: The text has been adjusted to highlight some actions on gender, clarify women’s participation in project activities and mention recruitment priorities at ENRU. 3.1. General information on the project ROA /page 5, paragraph 30 “….. These countries have been historically characterized by a high population density,” A small correction: only three countries (El Salvador, Dominican Republic and part Guatemala), have a high population density. This came from the project document. In any case, it has been adjusted. Level of achievement of the impact indicators from the results framework/65. Sustainability (page 18) 1. Given that only one programme or project can be confirmed, the target was not met. The report has been revised and the following results have been included, with which the goal has been reached: The creation of SICREEE, the norm and the tender in El Salvador and the biding process in Guatemala are relevant. The norm in El Salvador was already mentioned in paragraph 63. References have been added in the text FINAL ASSESSMENT REPORT 70 SPECIFIC COMMENTS Paragraph number Comment Evaluator’s response 1) The eight SICA countries approved (in December 2019) the creation of the Center for Renewable Energies and Energy Efficiency (SICREEE). 2) El Salvador: a) The regulatory authority approved at the end of 2017 a norm of distributed generation with renewable energies. With this legal basis, a tender for the purchase of 28 MW renewable energy began at the end of 2018. b) The first biodigester was developed and built - with financing from a cooperating country- based on the feasibility evaluation carried out within the Project (in the Agronomic School of Chalatenango (in the north western part of the country). c) In November 2016, the National Energy Council (CNE) presented during the COP 22 (Marrakech, Morocco) the project Energy Efficiency in Public Buildings, a project that was approved in 2017 by the NAMA FACILITY. In July 2018 funds were announced for 20 million dollars to develop this initiative, from the Green Climate Fund (GCF), which will be executed through the Development Bank of El Salvador (BANDESAL) 3) Guatemala. The state electric utility INDE returned to the development of renewable energies. In November 2018 the board of directors of the mentioned institution approved the Supply, construction, installation and start-up of photovoltaic solar systems up to a maximum of 110 megawatts, bidding process that will begin in 2019. The decision to contract the power supply Electricity through solar generation has the objective of diversifying the generator park and reducing the climatic risk on the water resource. Activities carried out on non-conventional variable renewable energy issues, as well as topics of vulnerability to climate change, within the ROA Project, are part of the background of this initiative. (paragraphs 73 and 62, respectively) to SICREEE and the biding process in Guatemala. However, these are not programme or projects, so while very positive, they do not count that indicator. The biodigester in El Salvador is already mentioned. The “Energy Efficiency in Public Buildings” is a project. A reference to this has been added and the text adjusted. Evidence of the project using the technical, human and other resources available in participating countries to increase efficiency/ paragraph 85, page 24 “…parsimonious use of its human resources, … However, this was a problem related to slow administrative pace and unforeseeable circumstances rather than a virtue” Comment: We believe that the final part of this paragraph is very unfavorable for the ENRU and does not give rise to any recognition The text has been adjusted to mention the work of other ECLAC officials, highlight the hard work and experience of ENRU staff and clarify, in case it wasn’t, that ENRU staff acted in accordance with staff rules and with the mandates of the general assembly. However, staffing was limited. FINAL ASSESSMENT REPORT 71 SPECIFIC COMMENTS Paragraph number Comment Evaluator’s response of the staff of our Unit (nor does it recognize the work of other ECLAC officials who collaborated in the project). It ignores the experience and knowledge of the subjects (experience of many years accumulated in the ENRU), the complexity of the issues addressed, and the difficulty in recruiting personnel with the required qualifications. ENRU officers had to sacrifice vacations, holidays, weekends and many times, to work more than 10 continuous hours a day. ENRU staff is very clear that they have acted in accordance with the Staff Rules and with the mandates of the General Assembly. Maybe it is not a virtue, but we believe that the final part of the paragraph does not value the effort of the staff of the ENRU and ECLAC Mexico, so we ask, be modified. It must be borne in mind that in addition to the activities of the Project, the ENRU officials carried out tasks of the regular program, contained in the Biannual Programs of ECLAC Mexico and many emerging tasks (requests for cooperation and assistance and techniques related to countries and requests of ECLAC Headquarters). For a project and for an institution, this is a problem (as the comment stresses, the issues addressed were complex and project staff had additional tasks) and not a virtue (even though it may be a virtue of project staff). Note that an independent evaluation report needs to present all relevant facts, regardless of whether they are favourable or not to a particular stakeholder. The difficulty of recruiting personnel with the required qualifications has been added as a reason for limited staffing. Existence of a clear and appropriate ME plan including scheduling, assignment of roles and responsibilities, and provision of adequate resources/ paragraphs 86 and 87, page 25 Comment: UERN had periodic meetings (quarterly) with the Research Coordinator (Pablo Yanez) and the Programming Officer (Liza Harake, until November 2018, and her assistant, Ana Maria Larrauri) of ECLAC Mexico, in which the progress of the Project was reported, potential problems and possible solutions were identified. When required, information and support were requested, both to the Director of ECLAC Mexico and to the DPPO. Beyond what is established in the prodoc, the size and resources of the ECLAC Mexico Units and the challenges faced to work in the period of adoption of the new UMOJA administration system must be taken into account. Remember, that according to UMOJAs philosophy, additional resources for M M should not be required (this will be a challenge in all UN projects, and a challenge for UN reform, signaled both by the UN Secretary General as by the Executive Secretary of ECLAC, in Town Hall annual meetings). The above, we suggest we suggest be considered as part of the lessons learned, and for the M E procedures of future projects. Note that the text referred to the ME plan in the project document. A new paragraph has been added on monitoring arrangements during implementation. A reference to the meetings with other staff at ECLAC Mexico and DPPO has been added in paragraph 86, when discussing the use of human resources. The size of ENRU and UMOJA related challenges have already been highlighted. A reference to UMOJA’s philosophy has been added, noting the risks that it represents. One of the lessons has been adjusted to mention this. FINAL ASSESSMENT REPORT 72 SPECIFIC COMMENTS Paragraph number Comment Evaluator’s response Existence of appropriate performance indicators, and adequate baseline information/ paragraph 88, page 25 Comment: Take into account in the definition of the indicators, the difficulties of simultaneously addressing 8 countries (which have similarities, but also large differences) and the challenge of achieving advocacy, both at the national and regional levels (to make the Agenda 2030 principle of No one must be left behind and its Goal 10, Reduce inequality within and among countries). I think part of the evaluators comments makes sense. Others do not, do not visualize the complexity of the regional dimension (many countries, each one with its specificities) and those of the energy sector. The argument is that conceptually the structure of objective and expected accomplishments does not clearly distinguish between scales (national (specific countries) or regional (the SICA)) and types of capacity. The complexities of the regional dimension and the energy sector and the differences between countries do not imply that the structure should not have distinguished in a clear way scales and types of capacity (the paragraph does not discuss the level of complexity of the regional dimension or the energy sector, or the degree of similarity between countries). /paragraphs 100 and 101, page 25 “…the retirement of another official could have been dealt with earlier, and the general recruitment need could have been addressed earlier” Comment: Yes, ECLAC Mexico and ENRU took the corresponding measures, in advance of at least 4 months. The new official was hired in record time (according to ECLAC records). Even so, the institution is far from achieving an overlap, of at least 1 month, between the outgoing officer and the incoming one, which would be ideal in order to avoid what was indicated by the evaluator. The recommendation is very valid for the human resources area “The introduction of UMOJA was probably planned well before its introduction in the second half of 2015” Comment: The evaluators comment is valid, however, it should be kept in mind that the UMOJA theme, due to its size, corresponds to another scale (it must be seen within the challenge of the UN Reform Process). After four years of UMOJA initiation, we believe that it has a very positive benefit / cost ratio. The effects to the Project must be considered as minor and unpredictable. On the other hand, the positive aspects should be valued: the organization and its officials were able to face and deal with the UMOJA Process in a good way, achieving that the counterparts (the countries that are the beneficiaries of the Project), did not have affectations caused by UMOJA. The response from ECLAC and the UN, extending the duration of the project for 1 more year (from 2 to 3 years), was of benefit to the Project (paragraph 101, summarizes the above). The additional information provided in the comment has been added to the text (paragraphs 98 and 100) and to the lessons learned (paragraph 150). The evaluation covers the UN reform process to the extent it affects the project. Whether the introduction of UMOJA has positive cost ratio is beyond the scope of this evaluation. The effects to the project were predictable: the project could have considered that the introduction of UMOJA would generate delays. A reference has been added in paragraph 97 regarding minimizing the impacts in the execution of project activities by mobilizing external funds. FINAL ASSESSMENT REPORT 73 SPECIFIC COMMENTS Paragraph number Comment Evaluator’s response Number of monitoring missions of ECLAC and meetings held /paragraph 102, page 31 Comment: the paragraph is a good criticism, but at the same time it does not recognize the strengths of the ECLAC subregional office in Mexico: the attention of 10 countries, in various topics and with limited resources. In the 13 missions referred to in the paragraph, the Chief of the ENRU also addressed topics of the Biannual Work Agenda of ECLAC Mexico. We suggest toning down the nuances of the comments. The assessment of the strenghts and weaknesses of ECLAC Mexico is beyond the scope of this evaluaiton. The report does not discuss that. The mentioned paragraph discusses the number of monitoring missions for this project. The number of countries covered by ECLAC Mexico does not explain that in any sense. It has been clarified that the chief on ENRU also addressed non- project related topics, although from a project point of view this is not particularly positive. Evidence of ECLAC’s management response / paragraph 103, page 31 “the project did not have a separate project team…” Comment: having a separate team, dedicated only to the project, is not viable, would have required many resources and therefore, would have reduced to the extreme the activities and the results of the Project. The mentioned paragraph is not assessing whether the project should have had or not a separate project team. The paragraph is indicating that ECLAC’s management response is discussed in section 5.3.2 above, where the responses of the project are discussed. Note that most development projects have a project team that is not composed by the UN programme staff. Evidence of collection of lessons learned and good practices on project activities and dissemination to relevant stakeholders /paragraph 105, page 32 “However, the project did not have a strategy to systematically document and share lessons from SICA countries” Comment: This was not reflected in the Project reports. Periodically, the ENRU reports to the Director of CEPAL Mexico, the relevant results obtained, including the success stories. In 2018, a commemorative publication of the 70th anniversary of ECLAC, included a chapter on the work of the ENRU (see: Chapter IX, https://repositorio.cepal.org/handle/11362/44590 References to regular reporting to the Director of ECLAC Mexico and the publication have been added. Note however that: 1) Lessons learned cover more than success stories, being critical to document both what works well and what works less well. Lessons have to be documented and shared with a wide range of stakeholders, not only internally at ECLAC Mexico. 2) The publication provides a very useful historical overview on energy integration in Central America and some good insights on the way forward, but it does not document the lessons learned through the process (the chapters on mobility and climate change have sections on lessons learned, but the energy one does not). Moreover, the text refers to lessons learned from SICA countries and lessons learned through the implementation of the project on which this evaluation focuses. FINAL ASSESSMENT REPORT 74 SPECIFIC COMMENTS Paragraph number Comment Evaluator’s response Evidence of efforts to optimize synergies and avoid duplications with the other activities implemented by beneficiary countries/ paragraph 106, page 32 Comment: Almost all the activities were carried out with the support of third parties (Ministries and / or cooperation agencies), for example: a) The Seminar on non-conventional and intermittent renewable energies (Panama, 2016), had the cooperation of IRENA b) Seminars on energy policy and evaluation of energy scenarios, were supported by OLADE. The support of OLADE and IADB was already mentioned in paragraph 108 (now 109). Explicit references to IRENA and AMEXID and some details have been added. c) The seminar on vulnerability to climate change of the SIEPAC regional network was co-organized with OLADE, the IDB and a regional entity (EPR) d) The study travel had the support of the Mexican Agency for International Cooperation (Amexid) and institutions of the energy sector of Mexico. e) Videoconferences for the discussion of regional energy efficiency standards were carried out with the support of the IDB. Evidence of efforts to optimize synergies and avoid duplications with the other activities implemented by ECLAC /paragraph 106, page 32 “There is no additional evidence of complementarities between project activities and other activities implemented by ECLAC” Comment: Care was taken to align most activities of ENRU and other Units and divisions of ECLAC, with the project, for example: a) In almost all the missions carried out within the Project, ENRU officers carried out coordination tasks related to the regular bi- annual program of ECLAC Mexico (update of databases and information systems, coordination of technical assistance activities, meetings of experts , studies and publications, and synergies with regional initiatives: the Mesoamerica Project, the Energy and Climate Alliance for the Americas and the sectoral councils of SICA, especially the Environment, Water and Health). b) With other Units and divisions, the following stand out: Energy and Climate Change (the first study on this topic in Latin America was developed by ECLAC Mexico, the issue of vulnerability of the electric power transmission systems was addressed within the Project), and, Energy efficiency: the BIEE base was driven by the DRNI for the countries of South America. The ENRU followed up in Guatemala, Honduras and Costa Rica. The additional information provided through the comment has been added to the report (paragraph 108) and the mentioned statement deleted. FINAL ASSESSMENT REPORT 75 SPECIFIC COMMENTS Paragraph number Comment Evaluator’s response /paragraph 106, page 32 “In most of the cases available information does not allow distinguishing the specific contribution of these stakeholders” Comment: we suggest deleting this paragraph as it is well explained, in the final lines of the referred paragraph: “In most of the cases in-kind collaboration included logistic costs of organization of meetings (meeting rooms and catering), and in some cases the travel costs of participants. ” The mentioned sentence has been deleted. Existence of capacity building follow up strategy in the participating countries/ paragraph 111, page 32 “The project document did not make reference to any strategy to maintain and increase capacity once the project phases out” Comment: Within the framework of the project meetings, the countries approved a matrix of actions (including 14 major axes and more than 200 actions). The above, in addition to the Strategy, constitute the regional commitments that will require continuing to strengthen national capacities. This was already commented in another section. It has been clarified that the text refers to technical capacity. Approving a matrix of actions including the commitment to strengthen capacities does not ensure that technical capacities will actually continue to be strengthened. The text has been adjusted to reflect this. “Panama has a market approach to energy with a focus on security where the government does not control what type of energy is used” Comment: we suggest changing the paragraph, for example: Panama has a market approach to energy with a focus on security. For this reason, the country promoted natural gas, as a transition fuel, but they will have to adapt the regulatory framework to provide better conditions for the development of variable renewable energies (solar and wind). This comes from an interview with representatives from the country. The evaluator does not have the legitimacy to indicate what Panama should or should not do regarding its regulatory framework. In any case, the point in that paragraph is that it is uncertain whether Panama would adapt its regulatory framework in the direction the comment suggests. Existence of institutional frameworks ensuring sustainability of the project results, including a knowledge sharing platform, in the participating countries / paragraph 114, page 36 “However, at institutional level, it does not seem to be much progress at the national scale.” Commentary; we suggest changing the paragraph, for example: “However, at institutional level, some countries may face specific major problems, which could lead to delays in complying with regional and international commitments (Honduras, Nicaragua and Guatemala).” “The only exception would be the Dominican Republic, where a Biomass Network was created, but this does not comprehensively cover sustainable energy”: Commentary; we suggest changing the paragraph, for example with this nuance: The comment should indicate why the text should be changed instead of proposing a different text. The suggested text does not clearly refer to institutional aspects. It is also unclear to which specific major problems the proposed text refers to. Again, the comment should justify why the sentence is not appropriate, instead of proposing a different sentence. Note that the proposed sentence does not clearly refer to the information provided in the sentence in the text. FINAL ASSESSMENT REPORT 76 SPECIFIC COMMENTS Paragraph number Comment Evaluator’s response On the contrary, institutionality in the Dominican Republic is highlighted, where in addition to the challenge of the sustainable use of biomass (for example, its Biomass Network), efforts should be made to improve the electricity services (through the so-called Electric Pact) and the promotion of renewable energies. The institutions of Costa Rica, Panama and El Salvador show a lot of solidity, stability and strength. In any case, taking into account the information provided, the text has been adjusted indicating a solid and stable institutional setting in Costa Rica, Panama and El Salvador, progress in the Dominican Republic and challenges in Guatemala, Honduras and Nicaragua. Evidence of assessment of possible gender inequality and specific human rights issues in energy access paragraphs 119 and 120, Comment: That as such (detailed assessment of gender and human rights) was not contemplated in the Project Document, therefore it was not addressed. Keep in mind that access to energy (the topic addressed in the Project) is considered (based on the 2030 Agenda) as part of basic human rights. Keep in mind that in all activities there was female participation by countries. This was already commented in another section. See above. Gender balance in participation to project workshops, seminars, meetings and study tours /paragraph 121, “This indicates an under representation of women in the energy sector” Yes, that is a reality, the same thing happens almost everywhere in the world. It is a problem of origin: in technical schools and university studies related to engineering and exact sciences, there is less participation of women. Paradoxically, it is not the energy sector where many of the gender and energy proposals should come from. In the case of the goal of achieving gender parity in the professional cadres of the energy sector, the main driving force should be the Education authorities (primary, middle, technological and higher), to break down myths rooted in the population. The evaluator does not have the data to certify whether this happens almost everywhere in the world. Several factors, including cultural ones, explain this. The responsibility is probably shared by different ministries and institutions, including the education authorities, but regarding energy, the energy authorities themselves probably have some responsibility. A footnote has been added. Note however that an in-depth analysis of the causes of this under representation is beyond the scope of this evaluation. Acronyms/page ii Include within the acronyms and abbreviations, the following: Amexid: Mexican Agency for International Cooperation ECLAC México: Subregional headquarters of ECLAC in Mexico. DRNI-ECLAC: Division of Natural Resources and Energy of ECLAC ECPA: Energy and Climate Alliance for the Americas (Regional Forum of Energy Ministers of the Americas) FB: Fundación Bariloche (think tank specialized in energy and development) References to the mentioned institutions have been added and their acronyms (except for Fundacion Bariloche –a small institution) included in the list of acronyms. ECLAC Mexico has also been added to the acronym list. FINAL ASSESSMENT REPORT 77 SPECIFIC COMMENTS Paragraph number Comment Evaluator’s response 57 The question and text refer to the PoWs of 2014−2015 and 2016−2017. However, Table 1 refers to the PoWs of 2014−2015 and 2015−2016. Why is that? Also, considering that the project was given a 1-year extension, why PoWs of the following years (2017−2018) are not taken into account. There was a typo in the table. The years were 2014−2015 and 2016−2017 (the evaluator has not found PoWs for the periods 2015−2016 and 2017−2018). References to PoW 2018−2019 have been added. Existence of a clear and appropriate ME plan including scheduling, assignment of roles and responsibilities, and provision of adequate resources/ paragraphs 86, page 25 La oficina de Programación realiza periódicamente reuniones con los Jefes de Unidades y/o Coordinadores de proyectos para monitorear el avance en la implementación del programa de trabajo tanto de presupuesto regular, como de los proyectos. En este foro se ha dado seguimiento al avance de este proyecto en particular con el Jefe de la Unidad, además de reuniones bilaterales entre el coordinador del proyecto y la Oficial de Programación. 2) La DPPO también convocó a teleconferencias con el coordinador del proyecto y el Oficial de Programación en varias ocasiones para dar seguimiento del avance. Continuamente hay coordinaciones con funcionarios de la DPPO encargados de los proyectos de Development account. Information on this has been added in paragraph 87. FINAL ASSESSMENT REPORT 78 Evaluation of the DA Project 1415BD/ROA 312-9A “Strengthening the Capacity of Central American and Caribbean Countries in the Preparation of Sustainable Energy Policies and Strategies” Evaluation Report Feedback Form: PPOD GENERAL COMMENTS Report section (if applicable) Comment Evaluator’s response General We want to thank the evaluator for the work that went into the report, which resulted into a very interesting document. We find particularly valuable the detailed analysis of the project alignment with the SDGs, and the findings and conclusions section in general. We have trouble following the link between findings, conclusions, and recommendations. Please review as to make that link apparent so that we can see that recommendations are based on findings and conclusions. Please remove mention of ROA which is an internal ECLAC codification. For example, please replace “the ROA project” with “the project” throughout The conclusions are a summary of the findings. References to paragraphs have been added in the recommendations to link these to findings. The expansion of recommendations also provides additional context. Mentions to ROA have been deleted. Lessons Identified Please rename this section lessons learned to be consistent with Evaluation TOR The section has been renamed. Recommendation The recommendation section would benefit from being revised and expanded, not necessarily in the number of recommendations, but in their specificity and depth. Please number recommendations for ease of tracking. Recommendations have been numbered and their specificity and depth expanded in section 8 on recommendations (they have not been expanded in the executive summary because this needs to be short). Executive Summary Comments have been made mostly to the body of the report, less so to the executive summary. Once the report is revised, please review executive summary to ensure that it is in line with the rest of the text. The whole document (executive summary, findings, conclusions, lessons learned and recommendations) has been reviewed once the report was revised to ensure consistency between the different sections of the report. Graphs Consider if adding additional graphs would make the report easier to read. The evaluator has not found where graphs could be added to make the report easier to read. FINAL ASSESSMENT REPORT 79 SPECIFIC COMMENTS Paragraph number Comment Evaluator’s response 11 Please clarify what is meant by “The project was not informed” It has been clarified. 17 Please see comment to paragraph 80-81 See response to those paragraphs. 35 Please rephrase. The project was implemented by ECLAC, and collaborated with various partners The sentence has been rephrased. 48 Please clarify the point made about biomass A footnote has been added to clarify this point. 54 Please correct typos: “230” should read 2030. • “SDG 7, which involves three goals” should read “three targets” Typos have been corrected. 61 Please clarify last sentence as it is unclear “Government officials also learned from project activities not directly oriented to training, such as on energy balances.” It has been clarified. The reference is to learning by doing. 62 The wording makes it sound as if the final report is the only source of information regarding the achievement of this target, please clarify if the evaluation confirmed this assessment. The wording has been adjusted. Different sources were used during the evaluation. 64 Regarding the last sentence (“the direct contribution of the project to these achievements is complex, and these achievements cannot be fully and exclusively attributed to the project”), please note that it is not a requirement in the PRODOC that those achievements are exclusively attributable to the project, on the contrary Development Account project are meant to be complementary to overall programme of work of implementing entities. The text does not say that a direct attribution was required in the Prodoc. However, it is important to clarify the attribution, not to overstate the impact of the project. I agree development projects (including Development Account projects) are meant to be complementary to other projects, the programme of work of implementing entities (e.g. ECLAC) and national efforts. Complementarities in those levels are discussed in the text. 8, 77, 101, 152 and Table 4 Please review corresponding paragraphs and table as discussed References to no reimbursing project-related costs to staff have been deleted in (former) paragraphs 8, 77, 152 and Table 4. No references to this were provided in paragraph 101. 79 On timeframe, please note that the end date of each project is fixed by Development Account guidance, therefore allocating more time to the project at its inception was not an option. The sentence has been adjusted to reflect that. 80 As written, it seems that the increase of activities is seen as a negative, while the logic of doing more with less is actually encouraged. It has been clarified. FINAL ASSESSMENT REPORT 80 SPECIFIC COMMENTS Paragraph number Comment Evaluator’s response 81 As mentioned above, the ability for the project to utilize all of its budget, to implement all planned activities and additional ones, and to leverage funds to achieve that is considered a positive achievement within the parameters of the Development Account. The conclusion has been adjusted, mentioning both that ECLAC had to cover the final project meeting and that the project was able to expand activities covering the additional costs through leveraging funds from partners. 82 Updated financial information has been provided, please review and revise accordingly. The updated financial information has been reviewed. The text has been revised accordingly, although the overall picture is very similar. Following a request sent by email, the table has been deleted. 83 It is unclear what is meant by defining project management costs and measuring their increase ex post. Please note that travel of staff was not only for the purpose of coordination, but also to deliver technical assistance, i.e. substantial to the execution of the project. Project evaluations typically assess the “Level of alignment between planned and incurred project management costs and nature of divergences”. It was agreed that this would be assess in the evaluation of this project during the inception phase. I have made reasonable assumptions to assess this. I have added a sentence clarifying that travel of staff was not only for the purpose of coordination, but also to deliver technical assistance. 86 Although it is not mentioned in the PRODOC, please note that, in addition to annual progress report, PPOD conduct monitoring meetings (by videoconference in the case of Mexico) at least twice a year with DA project implementation teams to review financial and substantive execution. A reference to this has been added in paragraph 87. 100 Please clarify what is meant by “of countries” in the phrase “Tensions between countries and of countries with the UN system”. It refers to tensions of countries with UN system (i.e. Nicaragua with international cooperation and donors). The sentence has been adjusted. 103 ECLAC is an implementing entity of the Development account, therefore the project manager is always an ECLAC official. Mention of a separate project team in this context is confusing. It has been clarified. Note that the mentioned paragraph only aims to indicate that that ECLAC’s management response is discussed in section 5.3.2 above, where the responses of the project are discussed. Note that most development projects have a day-to-day project team that is not composed by the UN programme staff. FINAL ASSESSMENT REPORT 81 SPECIFIC COMMENTS Paragraph number Comment Evaluator’s response 109 Table 4 In the table, if by ECLAC reference is made to RPTC funding regarding the $14,000, it should be labeled as such to clarify. A reference to RPTC, which I do not know what it stands for, has been added in table 4. Note that the final report indicates ECLAC. 137 Could you clarify what progress at the institutional level at the national scale would look like, as it is not clear how the project fell short in that regard. The sentence has been modified addressing a more detailed comment provide by ECLAC Mexico. 139 Please clarify what is meant by “the project was not informed” It has been clarified. 158 As noted above, the section of recommendations would be strengthened by making recommendations more specific and actionable, and by clearly linking them to findings and conclusions. Links between findings and recommendations have been clarified, by adding references to paragraphs and providing additional context. Recommendations have been expanded and made more specific and actionable. 159 This reads as too general. The point of the evaluation is to identify which specific lessons learned and recommendations should be applied to future projects. As they are written, the first 2 recommendations could be applied to any project. Recommendations have been expanded so they are useful to any future project. 160 and 161 Please clarify each recommendation by linking it to findings and conclusions, and adding explicative to make it understandable to someone without in-depth knowledge of the project. We suggest for each recommendation having the short version in bold, with numbering, and then further explication in one or a few paragraphs below. The suggested format has been used.