1995 es Corden, W. Max Corden, W. Max Una zona de libre comercio en el Hemisferio Occidental: posibles implicancias para América Latina En: La liberalización del comercio en el Hemisferio Occidental - Washington, DC : BID/CEPAL, 1995 - p. 13-40 2014-01-02T14:51:16Z hdl:11362/39404 FINAL ASSESSMENT REPORT ASSESSMENT OF DEVELOPMENT ACCOUNT PROJECT 08/09 AE Understanding Potential Economic Impacts of Climate Change in Latin America and the Caribbean August 2015 This report was prepared by Evan Green, an external consultant, who led the evaluation. Mr. Green worked under the overall guidance of Raúl García-Buchaca, Chief of the Programme Planning and Operations Division of the Economic Commission for Latin America and the Caribbean (ECLAC), and Sandra Manuelito, Officer-in-Charge of the Programme Planning and Evaluation Unit, and under the direct supervision of Irene Barquero, Programme Officer of the same unit, who provided strategic and technical guidance, coordination, and methodological and logistical support. The evaluation also benefited from the assistance of María Victoria Labra, Programme Assistant, and Carolina Trajan, Unit Intern, also of the Programme Planning and Evaluation Unit. The evaluation team is grateful for the support provided by its project partners at ECLAC, all of whom were represented in the Evaluation Reference Group (ERG). Warm thanks go to the programme managers of the Sustainable Development and Human Settlements Division of ECLAC for their cooperation throughout the evaluation process and their assistance in the review of the report, in particular Carlos de Miguel, Head, Policies for Sustainable Development Unit, and Karina Martinez, Research Assistant. The team also extends its gratitude to the programme managers of the ECLAC subregional headquarters for the Caribbean in Port of Spain, in particular to Willard Phillips, Economic Affairs Officer, and Elizabeth Thorne, Research Assistant. All comments on the evaluation report by the Evaluation Reference Group and the evaluation team of the Programme Planning and Evaluation Unit were considered by the evaluator and duly addressed in the final text of the report, where appropriate. The views expressed in this report are those of the author and do not necessarily reflect the views of the Commission. The annexes to this evaluation report have been reproduced without formal editing. Copyright © United Nations, August 2015. All rights reserved Printed at United Nations, Santiago S.15-00946 FINAL ASSESSMENT REPORT TABLE OF CONTENTS ACRONYMS ...........................................................................................................................................iii EXECUTIVE SUMMARY ...........................................................................................................................v 1. INTRODUCTION ..............................................................................................................................1 1.1. CONTEXT............................................................................................................................................................1 1.1.1. The Development Account............................................................................................................................... 1 1.2. BACKGROUND OF PROJECT ...............................................................................................................................2 1.3. OBJECTIVE AND SCOPE OF ASSESSMENT ...........................................................................................................2 2. ASSESSMENT MANAGEMENT .........................................................................................................4 2.1. ASSESSMENT STRATEGY .....................................................................................................................................4 2.2. ASSESSMENT APPROACH: DATA GATHERING AND ANALYSIS METHODS.............................................................4 2.2.1. Inception phase................................................................................................................................................. 4 2.2.2. Data collection phase ...................................................................................................................................... 5 2.2.3. Problem and objective trees and Theory of Change................................................................................. 7 2.3. LIMITATIONS .......................................................................................................................................................8 3. FINDINGS AND ANALYSIS..............................................................................................................9 3.1. ANALYSIS OF PROBLEM AND OBJECTIVE TREES AND THEORY OF CHANGE ........................................................9 3.1.1. Problem tree ..................................................................................................................................................... 9 3.1.2. Objective tree.................................................................................................................................................11 3.1.3. Logical framework .........................................................................................................................................12 3.2. RELEVANCE ...................................................................................................................................................... 14 3.2.1. Relevance of the project and activities to the region and countries .....................................................14 3.2.2. Links to the ECLAC mandate.........................................................................................................................16 3.3. EFFECTIVENESS ................................................................................................................................................ 17 3.3.1. The project’s indicators and expected results ...........................................................................................17 3.3.2. Increased awareness (EA 1) .........................................................................................................................18 3.3.3. Strengthened national capacities (EA 2)....................................................................................................21 3.3.4. Strengthened regional cooperation (EA 3)................................................................................................24 3.3.5. Methodologies used for studies ...................................................................................................................26 3.3.6. Gender mainstreaming and human rights..................................................................................................27 3.4. EFFICIENCY ...................................................................................................................................................... 29 3.4.1. Organizational factors and project management by ECLAC.................................................................29 3.4.2. Complementarities and synergies ...............................................................................................................31 3.5. SUSTAINABILITY AND REPLICATION.................................................................................................................. 33 3.5.1. Replication of activities and sustainability within participant countries ................................................33 4. LESSONS LEARNT AND BEST PRACTICES......................................................................................36 5. CONCLUSIONS..............................................................................................................................37 6. KEY RECOMMENDATIONS ...........................................................................................................40 ANNEXES .............................................................................................................................................. 45 ANNEX 1: EVALUATION DESIGN MATRIX ................................................................................................................. 46 ANNEX 2: INTERVIEW PROTOCOL ............................................................................................................................. 51 ANNEX 3: ASSESSMENT QUESTIONS......................................................................................................................... 54 ANNEX 4: SIMPLIFIED LOGICAL FRAMEWORK ........................................................................................................ 56 ANNEX 5: LIST OF STAKEHOLDERS INTERVIEWED .................................................................................................. 60 ANNEX 6: BIBLIOGRAPHY............................................................................................................................................. 61 ANNEX 7: STAKEHOLDER MAP .................................................................................................................................... 63 ANNEX 8: EVALUATOR’S REVISION MATRIX ............................................................................................................ 64 i FINAL ASSESSMENT REPORT ACRONYMS Acronym CARICOM CBD CCCCC CSR DA DFID EAs ECLAC ERG GHG HLAC IADGs IISD IPCC IR LAC LARC MDG PPBME PPOD PPEU RECC RBM SDHS ToR UNCED UNEG UWI Definition Caribbean Community Convention on Biological Diversity Caribbean Community Climate Change Centre Corporate Social Responsibility Development Account Department of International Development (UK) Expected Accomplishments Economic Commission for Latin America and the Caribbean Evaluation Reference Group Greenhouse gas High-level Advisory Committee Internationally Agreed Development Goals International Institute for Sustainable Development Intergovernmental Panel for Climate Change Inception Report Latin America and the Caribbean Latin America and Caribbean Regional Coverage Millennium Development Goal Programme Planning, Aspects of the Budget, the Monitoring of Implementation and the Methods of Evaluation Programme Planning and Operations Division Programme Planning and Evaluation Unit Review of the Economics of Climate Change Results-based management Sustainable Development and Human Settlements Division (ECLAC) Terms of reference United Nations Conference on Environment and Development United Nations Evaluation Group University of the West Indies iii FINAL ASSESSMENT REPORT EXECUTIVE SUMMARY I. Climate change, which is essentially manifested in rising average temperatures, changes in precipitation patterns, rising sea levels, reduction of the ice extent, glacier melt and alterations in the pattern of extreme events, will disproportionately affect developing countries. In Latin America and the Caribbean, the effects of these changes will be observed in many areas including agricultural activity, water availability, forests and biodiversity, sea levels, tourism and health, all of which will also have a significant impact on the region’s economies. In fact, tentative estimates indicate that if the region’s average temperature rises by 2.5°C, the economic costs of climate change will stand at between 1.5% and 5.0% of current GDP, while adaptation costs are calculated at less than 0.5% of current GDP. II. The Caribbean subregion is particularly exposed to combined phenomena, such as rising sea levels coupled with extreme climate events including drought, flooding, hurricanes and storms. In Latin America and the Caribbean, a transition is needed towards a more sustainable form of development that will preserve its economic, social and natural assets. Economic analysis of climate change in the region is therefore vital not only to identify the main channels, the scale of climate effects and the best ways of adapting to the new climate conditions, but also to formulate a long-term sustainable development strategy that combines a low-carbon pathway with social inclusiveness. III. The Development Account funded project “Understanding potential economic impacts of climate change in Latin America and the Caribbean” was implemented by ECLAC during the period 2008-2010 for a total budget of US$ 558,000. It sought to improve and harmonize the ability of Latin American and Caribbean countries to adapt and cope with the economic impacts of climate change on development. More specifically, it aimed to provide stakeholders with relevant information on climate trends for 20-50 years or more, evaluate the impacts of any changes detected through relevant studies, and disseminate the findings of such studies in a meeting. The expected accomplishments (EAs) were: • Increased awareness among Latin American and Caribbean governments, business communities and civil society of the economic impacts of climate change, as well as adaptation and mitigation options. • Strengthened national capacities to identify responses to the potential economic impacts of climate change. • Strengthened regional cooperation to address the economic impacts of climate change. IV. ECLAC conducts internal assessments of each of its Development Account projects in accordance with Development Account requirements. Assessments are defined by ECLAC as brief end-of-project evaluation exercises aimed at assessing the relevance, efficiency, effectiveness and sustainability of project activities. They are undertaken as desk studies and consist of a document review, stakeholder survey, and a limited number of telephonebased interviews. The final assessment of this project was undertaken from February to July 2015. It is retrospective and summative in nature and considers both anticipated and unanticipated key results. It looks at all project activities and, to the extent possible, at non-project activities. v FINAL ASSESSMENT REPORT V. The approach utilized for this assessment comprised three key phases: inception phase, data collection phase and the data analysis and reporting phase. The inception phase included a preliminary document and literature review, the drafting of an inception report/workplan, and the development of data collection tools for the subsequent desk review, surveys and interviews. During the data collection phase, interviews and surveys were conducted with key project stakeholders, including ECLAC project managers, partner institutions and beneficiaries. Lastly, and based on the data collected through a thorough review of all project documentation as well as the interviews and surveys with key stakeholders, the data analysis and reporting phase entailed the drafting of this final assessment report. VI. This assessment evaluates the efficiency, effectiveness, sustainability and relevance of the project in order to determine whether or not the project has achieved its objectives. Specifically, it examines how the project’s activities and results have contributed to increasing the knowledge, awareness and capacity of governments and other stakeholders in the Latin American and Caribbean region regarding the economic impacts of climate change. VII. Some of the lessons learnt and best practices identified during this assessment that could be considered in future and similar initiatives include: • Supporting multidisciplinary studies in the area of climate change in the region is critical, as they are considered relevant and contribute to national planning in the areas of mitigation and adaptation to climate change. • Technical assistance can improve the chances for success of such projects as it is appreciated and considered useful by participants. • Use of key mechanisms such as the International Advisory Panel, National Advisory Panels and Technical Teams in the management of such projects can help to enhance overall efficiency. • Wide dissemination of key products of the project can help to further awareness and enhance knowledge. • Ensuring the involvement of key regional knowledge organizations as early as the project design stage can help to ensure that available expertise is tapped into and key stakeholders who can further the project’s agenda are involved early on in a meaningful way. VIII. The key recommendations of the project are: • In the context of future projects, the logical framework should be expanded upon and improved in order to ensure accurate monitoring, evaluation and reporting throughout the project life cycle. • Future and similar initiatives should make an effort to ensure that key stakeholders in the area of climate change in the region (such as CCCCC and UNFCCC) are involved in all stages of implementation. • Reports, studies and other products should be promptly and widely distributed with a view to enhancing the overall relevance of the programmes and increasing general awareness of the issue. • Additional efforts should be made to ensure smooth coordination between multiple sectors in these types of participatory/collaborative initiatives, which require the involvement of many stakeholders. vi FINAL ASSESSMENT REPORT • Future and similar initiatives should take into account in the project approach the time and effort needed to collect the required data before conducting the studies, and further assistance should be provided to facilitate implementation of the recommended methodologies. • Existing studies should be maintained and updated periodically to advance and strengthen the evaluation and formulation of specific actions for mitigating the economic impacts identified, and to ensure that changing conditions and contexts are reflected, thereby enhancing and sustaining the relevance and usefulness of the studies produced. IX. The ability to estimate the cost that climate change will have on a country or region is fundamental, not only for planning and adaptation purposes, but also for mitigation. Understanding that in some cases the cost of avoiding the effects of climate change might be a fraction of the cost of tackling the causes of this climatic phenomenon is the first step towards the action that is needed regionally as well as globally. The purpose of these studies is to offer evidence for decision-making so that countries, governments and all relevant climate change stakeholders have the information needed to demonstrate that the disastrous effects of future climate change will not only be detrimental to individuals, groups and natural systems but will also bring a significant economic cost that has yet to be fully considered. This project contributed key studies, information and capacity-building to this area of research, providing some of the basis that is needed to begin these discussions. It can therefore be regarded as an important contribution to climate change research and knowledge and a fairly significant step towards understanding the economic impacts of climate change in the region. vii FINAL ASSESSMENT REPORT 1. INTRODUCTION 1. This draft final assessment report provides an evaluation of the Development Account project “Understanding Potential Economic Impacts of Climate Change in Latin America and the Caribbean.” The assessment was carried out by Evan Green (herein referred to as the Consultant), as commissioned by the Economic Commission for Latin America and the Caribbean (ECLAC). 2. The report is made up of the following sections: • Section 1: Introduction. This section provides an overview of the context and the background of the project as well as the scope of the assessment. • Section 2: Assessment management. This section highlights the overall assessment strategy and the methodologies used for data gathering and analysis. • Section 3: Findings and analysis. This section examines the key findings and conclusions reached on the basis of the relevant data collected for this assessment. It is based on the four assessment criteria outlined in the terms of reference (relevance, effectiveness, efficiency and sustainability) and includes an analysis of the project’s problem and objective trees together with the lessons learnt and recommendations. 1.1. Context 3. While all regions will eventually feel the effects of climate change, it will disproportionably affect developing countries, in particular poor communities that are already living at, or close to the margins of survival. The impacts of climate change on development are already being addressed in some countries in Africa and the Pacific, but the Latin American and Caribbean region is still in the initial stages of assessing the economic impacts of climate change, supported by ECLAC. 4. The small islands of the Caribbean are highly vulnerable to the impacts of climate change such as rising sea levels, coral bleaching and increased rainfall, as well as drought and disasters. Efforts are already being made to address climate change, with valuable financial support from such organizations as the Organization of American States (OAS) and the former Canadian International Development Agency (CIDA), now the Department of Foreign Affairs, Trade and Development (DFATD). This led to the establishment of the Caribbean Community Climate Change Centre (CCCCC), which has already begun to address the impact of climate change in the Caribbean subregion. It is noted that countries would greatly benefit from more international funding. It is widely felt that all efforts to address impacts will eventually have to be comparable and inclusive on a regional scale and should encourage international coordination and harmonization of implementation measures. 1.1.1. The Development Account 5. The Development Account was established by the General Assembly in 1997 as a mechanism to fund capacity development projects carried out by the economic and social entities of the United Nations. Projects financed by the Development Account aim at achieving development impact by building the socioeconomic capacity of developing countries through collaboration at the national, subregional, regional and inter-regional 1 FINAL ASSESSMENT REPORT levels. Development Account projects are implemented by global and regional entities, cover all regions of the globe, and focus on five thematic clusters. 1 Projects are programmed in tranches, which represent the Accounts programming cycle. The Development Account is funded from the Secretariats regular budget and ECLAC is one of its 10 implementing entities. 1.2. Background of project 6. The project “Understanding potential economic impacts of climate change in Latin America and the Caribbean” was implemented during the period 2008-2010 for a total budget of US$ 558,000. It sought to improve and harmonize the ability of Latin American and Caribbean countries to adapt and to cope with the economic impacts of climate change on development. More specifically, it aimed to provide stakeholders with relevant information on climate trends for 20-50 years or more, evaluate the impacts of any changes that are detected through relevant studies, and disseminate the findings of such studies in a meeting. The Expected Accomplishments (EAs) were: 1) Increased awareness among Latin American and Caribbean governments, business communities and civil society of the economic impacts of climate change, as well as adaptation and mitigation options. 2) Strengthened national capacities to identify responses to the potential economic impacts of climate change. 3) Strengthened regional cooperation to address the economic impacts of climate change. 7. In order to achieve the EAs outlined above, the primary focus of the project was to support countries and governments in the region in the development of research studies on the economic impacts of climate change. These studies were to be a tool for governments in decision-making and planning and also to contribute to a larger study that would provide an overview of the issue for the entire region. While this project was partially funded by the Development Account and supported and implemented by ECLAC, similar activities are still ongoing today. The project can thus be seen as part of a larger initiative within the region. 1.3. Objective and scope of assessment 8. This assessment aligns with General Assembly resolutions 54/236 of December 1999 and 54/474 of April 2000, which endorsed the Regulations and Rules Governing Programme Planning, Aspects of the Budget, the Monitoring of Implementation and the Methods of Evaluation (PPBME). As part of the strengthening of the evaluation function to support and inform the decision-making cycle in the United Nations Secretariat in general and ECLAC in particular and within the normative recommendations made by different oversight bodies endorsed by the General Assembly, the Executive Secretary of ECLAC is implementing an evaluation strategy that includes periodic evaluations of areas of the organization’s work. This is therefore a discretionary internal evaluation managed by the Programme Planning and Evaluation Unit of the Programme Planning and Operations Division at ECLAC. 1 Development Account projects are implemented in the following thematic areas: advancement of women; populations/countries with special needs; drug and crime prevention; environment and natural resources; governance and institution building; macroeconomic analysis, finance and external debt; science and technology for development; social development and social integration; statistics; sustainable development and human settlement; and trade. 2 FINAL ASSESSMENT REPORT 9. ECLAC undertakes internal assessments of each of its Development Account projects in accordance with Account requirements. Assessments are defined by ECLAC as brief end-ofproject evaluation exercises aimed at assessing the relevance, efficiency, effectiveness and sustainability of project activities. They are undertaken as desk studies and consist of a document review, stakeholder survey and a limited number of telephone-based interviews. The final assessment of this project was undertaken from February to July 2015. It is retrospective and summative in nature and considers both anticipated and unanticipated key results. It looks at all project activities and, to the extent possible, at non-project activities. Specifically, it seeks to: 1) Analyse the design of the project as well as the relevance of its stated goals to the thematic area and region within which it operated. 2) Assess the project’s level of efficiency in implementing its activities, including its governance and management structures. 3) Take stock of the results obtained by the project and evaluate the extent to which it achieved its objectives. 10. This assessment evaluates the efficiency, effectiveness, sustainability and relevance of the project in order to assess whether or not the project has achieved its objectives. Specifically, it examines how the project activities and results have contributed to increasing the knowledge, awareness and capacity of governments and other stakeholders in the Latin American and Caribbean region with regard to the economic impacts of climate change. 11. The target audience and principal users of the evaluation include all implementing partners and beneficiaries, the Programme Manager of the Development Account, as well as other Regional Commissions and agencies of the United Nations system. The unit of analysis for this assessment is the project itself, including its design, implementation and effects. The assessment will be conducted in line with the norms, standards and ethical principles of the United Nations Evaluation Group (UNEG). 2 Finally, as agreed, the Consultant has not treated this assessment as a fully fledged evaluation; nevertheless, the ECLAC guiding principles to the assessment process were applied as far as possible. 2 Standards for Evaluation in the United Nations system, UNEG, April 2005. http://www.unevaluation.org/document/detail/22. Norms for Evaluation in the United Nations system, UNEG, April 2005. http://www.uneval.org/document/detail/21. UNEG Ethical Guidelines for Evaluation, UNEG, March 2008.http://www.unevaluation.org/document/detail/102. 3 FINAL ASSESSMENT REPORT 2. ASSESSMENT MANAGEMENT 12. This section provides an overview of the strategy used during this assessment process as well as details of the data gathering and analysis methods. 2.1. Assessment strategy 13. The assessment strategy focused on addressing the evaluation questions presented in the terms of reference in a timely manner (these questions are found in Annex 3). By adopting this strategy, an independent, participatory and professional approach has been taken. The Consultant worked independently but received organizational support from ECLAC in setting up interviews and managing the online survey. 14. The methodological principles set forth below helped to guide the assessment process. • Participative evaluation: The Consultant fully adheres to the UNEG guidelines stating that: “...evaluations should be carried out in a participatory and ethical manner...” The consultant made every effort possible to involve many of the key stakeholders linked in one way or another to the project, within the very relevant constraints of time and resources. • Triangulation of observations and findings: The assessment applied the information and data triangulation method to answer the evaluation questions. To the extent possible, the Consultant ensured that all findings through each line of inquiry were comprehensively cross-checked with one another (desk research, interviews and the electronic surveys) in order to comprehensively answer the evaluation questions. • Participative and iterative management: To ensure the mandate was aligned with the proposed and adapted timeline, the Consultant took a flexible approach to managing the assessment process. • Application of knowledge: Throughout the mandate, the evaluation Consultant exerted his theoretical and practical knowledge in the areas of evaluation, climate change, human rights, results-based management, UN/ECLAC programming, report writing and all other relevant domains. 2.2. Assessment approach: data gathering and analysis methods 15. The approach utilized comprised three key phases: inception phase, data collection phase and the data analysis and reporting phase. This approach and these phases are outlined below 2.2.1. Inception phase Step 1: Preliminary document and literature review 16. The purpose of this phase was to for the Consultant to familiarize himself with the project, the main stakeholders and partners, the beneficiaries, and the results achieved to date. This entailed a desk review of all relevant project documentation as well as a stakeholder mapping of key actors. 4 FINAL ASSESSMENT REPORT Step 2: Inception report/workplan development of data collection tools 17. A key element of the assessment was the inception report. This provided an overall approach to the project’s assessment and helped to build up a picture of the assignment’s activities so that the necessary staff, time and resources could be identified and made available. The Consultant prepared a draft inception report that described the methodology to be used, including an evaluation matrix and a detailed workplan. 18. To ensure a rigorous approach to assessment planning, an evaluation design matrix was developed that outlined evaluation criteria, key questions, sub-questions, indicators, and sources from which data would be collected to address the issues. The matrix served as an overarching tool to guide the preparation of the data collection tools and efforts to implement them. The evaluation design matrix is presented in Annex 1. 19. This assessment encompassed the design, implementation and results stages of the Development Account project and was structured around four UNEG standard evaluation criteria: relevance, efficiency, effectiveness and sustainability (the impact of the project was not addressed in this assessment). To assess performance according to the criteria (and subcriteria), a combination of content analysis along with electronic survey and semi-structured interview (by telephone/Skype) results were triangulated. Please see Annex 3 for a list of the assessment questions that were outlined in the terms of reference. These illustrate how the evaluation criteria and key guiding questions for the stakeholders were grouped. Given the large number of questions that could capture the same or very similar responses, the evaluation design matrix aimed to avoid repetition and lengthiness with fewer but more encapsulating questions. 2.2.2. Data collection phase 20. On the basis of the evaluation matrix, data gathering tools were developed and focused on the essential issues pertaining to the assessment’s guiding questions. This assessment utilized a mixed methods approach to data collection and analysis and used the following data collection methods to investigate responses to the aforementioned list of assessment questions. 1. Desk review 21. The Consultant reviewed and identified relevant sources of information and conceptual frameworks that fell within the framework of the project. A number of documents were reviewed, including: allotment advice, redeployments, the project document, progress reports, the final report, meeting reports, workshop-related documents, studies, and terms of reference for different consultancy works. Both gender and human rights were considered, particularly when reviewing information on meetings and events that were organized, as well as the key participants. Please refer to Annex 6 for a full list of documents reviewed. 22. Furthermore, the main stakeholders of the project were mapped, including managers, implementing partners inside and outside the United Nations system, and programme beneficiaries. Please refer to Annex 7 for the stakeholder map. 2. Interviews and surveys 23. A limited number of interviews (semi-structured, in-depth, key informant) were carried out via telephone and Skype with project stakeholders to capture the perspectives of a sample of managers, beneficiaries, participating ministries, departments and agencies, and/or other counterparts. 5 FINAL ASSESSMENT REPORT 24. Similarly to the interviews, electronic surveys were designed for the collection of data from key stakeholders. The Consultant aimed to survey a sample of stakeholders from participating countries such as policymakers in government ministries, other government officials from the economic, environment and finance ministries, and project managers at ECLAC and other partners who were involved in the project at the planning, implementation or other phase. In order to take into consideration the extent to which the project addressed human rights, survey questions were formulated to gauge the involvement of civil society, youth and ethnic minorities. The Programme Planning and Evaluation Unit administered the surveys and consolidated the data for use by the Consultant. 25. The key informants of the interviews and surveys can be grouped as follows: • ECLAC project managers: This group consisted of ECLAC representatives who were involved in the project from the Port of Spain, Santiago and Mexico offices. • Beneficiaries: This group includes representatives from government ministries and institutions as well as other participants involved in project activities. • Partner institutions: This was quite a varied group and included partners from other regional and international organizations as well as from universities, research centres and NGOs. 26. The activities undertaken as part of the project can also be split into three categories: • Events and seminars: These consisted of regional and high-level meetings that were held to discuss and share methodologies on the economic impacts of climate change. • Publications and studies: These were the country-level and regional studies conducted and published that examined the issue. • Technical assistance: This consisted of the support provided by ECLAC to countries in terms of building capacity and helping to implement particular methodologies. 27. This is useful for understanding the way in which the project has been formulated as well as any disaggregation that was performed throughout the draft final assessment report on the basis of these categories. Table 1 presents the number of stakeholders that were contacted and the overall response rate. Interviews Number of stakeholders contacted Number of stakeholders interviewed Surveys Number of stakeholders contacted Number of survey responses Table 1 Response rate Beneficiaries Partner institutions 10 4 51 63 86 6 26 ECLAC project managers 4 4 14 5 Total 18 10 163 36 6 FINAL ASSESSMENT REPORT 28. Overall, 10 key stakeholders were interviewed and 36 surveys were completed. The lowest response rate was recorded from beneficiaries and therefore the sample of responses from this group is relatively small. It should be noted, however, that the beneficiary and partner institution stakeholder groups are not always easily discerned. Given that many government institutions in the partner countries were contributing to these studies in a participatory manner, it became apparent that they were often both beneficiaries and partner institutions. Furthermore, a number of non-governmental partner institutions that contributed to the project also received capacity-building and experienced some increased awareness of key issues, and thus could also be considered beneficiaries. In this way, these partner institutions may have helped with the implementation of key activities, such as drafting of the reports, while also receiving capacity-building or technical assistance. This may, in fact, be the reason why the number of beneficiaries interviewed and surveyed appears to be low, as some partner institutions can also be considered beneficiaries. A list of stakeholders consulted through interviews may be found in Annex 5. 2.2.3. Problem and objective trees and Theory of Change 29. The project document includes both a problem tree and an objective tree. The Consultant assessed these simplified representations of reality and the development hypotheses behind them. This was done by logically reconstructing the Theory of Change, identifying original weaknesses, gaps, and/or any unintended effects (both positive and negative). Please see the analysis that was done in this respect in section 3.1. 3. Data analysis reporting phase 30. Recalling that this was not a fully fledged evaluation but a modest assessment integrating evaluation criteria, this consultancy utilized the aforementioned data collection methods to judge whether meanings and assertions from the data sources were trustworthy. The Consultant worked to distil patterns in the data, be they consistencies or co-variations. To maintain simplicity, this review has included: 1. A content analysis of findings from the document review to the furthest extent that they provide answers to the evaluation criteria questions. 2. A content analysis of stakeholder interview responses to tease out any details, gaps and uncertainties to questions that were not clarified by the documentary evidence. For questions that were answered through the documents, responses were cross-checked with the responses from interviewees for consistency. 3. A review of the results from surveys provided by the Programme Planning and Evaluation Unit to check for (1) internal consistency between the three sets of respondents (ECLAC project managers, beneficiaries and partner institutions), and (2) external consistency between the survey results and the findings from the other two lines of evidence, document review and stakeholder interviews. 31. The triangulation process also included completion of the data evaluation matrix to highlight the findings for each of the evaluation questions and demonstrate the links between (1) the information and data collected through each line of inquiry and (2) each performance indicator. 7 FINAL ASSESSMENT REPORT 32. The data collected through interviews and electronic surveys are presented throughout this assessment using a simple, straightforward and efficient benchmark: • All respondents said…; • The majority of respondents said… (~ more than 75%); • Many respondents said… (~ more than 50%), • Some respondents said… (~between 25% and 50%), • A few said… (~ less than 25%); and • One respondent said… 33. Following this analysis, the reviewer was able to draw reliable and credible conclusions and better formulate valid recommendations and lessons learnt for the draft final assessment report, within the contextual limitations. 2.3. Limitations 34. Best efforts notwithstanding, any assessment has a finite scope at the onset and encounters limitations during the engagement. This does not impede assessment of the project but does shape the outcome. Some limitations encountered during this assessment include: • Scope: Given that this is not a fully fledged evaluation, the overall scope of the assessment is smaller and more limited. The time and resources provided for undertaking the assessment were very limited and therefore the assessment’s depth and scope are also somewhat limited. This does not negate the findings in this report; however, it should be taken into consideration that a greater scope would have enhanced the overall findings. • Collection of survey data: Survey data were collected by the Programme Planning and Evaluation Unit and hence timing, follow-up and consolidation of the data was out of the hands of the Consultant. While this saved time, it also means that certain techniques that may have been used by the Consultant to enhance response rates were not applied. • Project formulation and specificity: Given that the project was implemented in collaboration with other partners and donors, and similar activities are still ongoing, it was often difficult to clearly identify what constituted the efforts of this particular project, within its specific time frame. In addition, some stakeholders were not necessarily aware that the activities in which they had participated were a part of this specific project. This meant that assessing the project as a singular unit was somewhat difficult and data had to be collected per activity to ensure that stakeholders were providing accurate information. • Timing of the assessment: It should also be noted that this project ended in 2010. Given that five years have passed since the project ended, it is possible that this has influenced the data collected for this assessment. It is probable that respondents have moved on from their original positions or do not remember all the details of the project. Furthermore, there have also been changes in ECLAC staff, as some of those who were project managers for this project, and may have provided input to this assessment, are no longer with the organization. This should be taken into consideration when reviewing the results found in this report. 8 FINAL ASSESSMENT REPORT 3. FINDINGS AND ANALYSIS 35. This section outlines the key findings and analysis that have been derived from the assessment of the data collected through document review, interviews and surveys. Each of the evaluation criteria (relevance, effectiveness, efficiency and sustainability) is addressed throughout this section as well as an analysis of the problem and objective trees and an overview of recommendations and lessons learnt. 3.1. Analysis of problem and objective trees and Theory of Change 3.1.1. Problem tree 36. The project document includes a problem tree that outlines the effect that climate change has on various sectors of an economy. This problem tree demonstrates that, through anthropogenic and natural factors, greenhouse gas emissions are increased, which has an impact on global temperatures and hence the overall climate and environment. These impacts include rising sea levels, higher ocean temperatures, increased flooding and drought. As demonstrated in the problem tree, this changing environment and climate can affect various sectors of the economy (infrastructure, agriculture, forestry, tourism, housing etc.) in a number of different ways (jobs losses, threats to food security, reduced production of food and other goods, loss of income and cash flow, fluctuations in tax income, etc.). Figure 1 below presents the project’s problem tree in full. Figure 1 Problem tree 9 FINAL ASSESSMENT REPORT 37. The Overseas Development Institute (ODI) explains that the process of producing a problem tree analysis (also known as a situational analysis or problem analysis) provides a clear prioritization of factors, more focused objectives, a better understanding of the problem, an outline of key actors and a gap analysis in terms of what further information and resources are needed3. Furthermore, the process of building the tree itself is often participatory and can help to build a shared sense of purpose and action among participants. 38. While the problem tree for this project provides a brief overview of climate change and the effects it may have on a country’s economic sectors, it lacks a number of details in terms of providing a thorough analysis of such a complex issue. It is however, understood that the “cause” of the problem is climate change and the “effects” may be seen in the various sectors outlined. It should be noted, first and foremost, that there is no accompanying narrative to explain the process of development for the problem tree or the logic behind it. In terms of understanding how climate change affects the various sectors, the problem tree is somewhat vague. For example, it is clear from the diagram that in terms of infrastructure, climatic variations may cause a decrease in productivity, transport and jobs. It also appears that it is through damage to roads that transport is reduced; however, it is not clear how both jobs and productivity are affected (presumably it is also through damage to roads). This is partially due to the fact that these “effects” are fairly vague and written as titles for each sector instead of a more specific or precise wording, such as “damage to infrastructure”, which would provide a better explanation of the cause and effect nature of the diagram since it more clearly denotes a problem. It is understood that these effects were explored in greater detail within each of the country-level studies, nevertheless more information as to how this original problem tree was developed and the thinking behind it would have enhanced understanding at this early stage. 39. In addition, key areas appear to be missing within the tree. For example, issues related to water resources and the health sector are not addressed, although they present themselves quite frequently in the country-level studies. While one cannot expect all issues related to the economic impacts of climate change to be identified at the design stage, it is important that the process is thorough from the beginning. 40. In terms of the design of the diagram itself, the ‘changing environments climate’ level could have been further expanded to outline the key climatic and environmental effects of climate change and link these with the various sectors found above. An additional box for the different sectors to feed into at the highest level also appears to be missing. A box at this level would identify the key problem that the project is aiming to address, namely: the negative impacts of climate change on the economies of Latin America and the Caribbean. Furthermore, in terms of looking at the effects that climate change and climatic fluctuations may have on the economy, it would also be useful to address more explicitly the effect that the economy has on the climate. 41. Overall, further elaboration, clarification and unravelling of the issues would have strengthened problem analysis during the project design phase. Nevertheless, the logic of the problem tree is fairly clear and does provide an overview of some of the key sectors that would be affected. Given that these sectors are very present within the country reports that were supported by this project, it does appear that this analysis provided a basic foundation in terms of focusing the studies on specific sectors in the countries. 3 http://www.odi.org/publications/5258-problem-tree-analysis. 10 FINAL ASSESSMENT REPORT 3.1.2. Objective tree 42. The objective tree corresponds directly to the problem tree discussed above. The analysis of objectives includes the following three steps: (1) modifying the problems from the problem tree into positive statements of what is to be achieved; (2) verification of the hierarchy of objectives; and (3) a visual representation (in the form of a diagram) of the means-end relationships. Figure 2 Objective tree 43. As can be seen above, the objective tree demonstrates that, through analysis of data from climate modelling to predict future economic impacts and identification of policies to address these issues, both anthropogenic and natural causes of climate change may be reduced. Furthermore, through increased awareness of the economic impacts of climate change, potential adaptation, mitigation measures to address these impacts and a regional policy to address climate change, various economic sectors in the country can be assisted and potential negative financial impacts can be avoided. 44. With regard to the overall coherence of the objective tree, first and foremost, the issues identified above for the problem tree should be highlighted (lack of clear statements of change, absence of key thematic components, lack of details, missing narrative etc.). A slightly more robust problem tree would have clearly translated into a more robust objective tree. 45. In addition to this, a useful technique that could have been applied to the objective tree is to take it one step further in terms of clustering the objectives and identifying potential interventions that could address these issues. This step is often necessary for showing that a certain problem can be addressed through a particular intervention in order to achieve a desired objective. It would include a number of different interventions above and beyond what was covered by this project; however, a case would be made for focusing specifically on generating knowledge based on climate modelling for improving understanding and 11 FINAL ASSESSMENT REPORT awareness of the economic impacts of climate change. This would have provided a true analysis of the situation that examines more than just the planned project activities and would have clearly presented the project scope within the larger context of climate change in Latin America and the Caribbean. 46. Overall, an accompanying narrative and analysis would have complemented both of the trees, and an additional diagram to outline the interventions that would contribute to the planned objectives would have provided a much clearer picture of the situation as a whole as well as how the project aimed to contribute. 3.1.3. Logical framework KEY FINDINGS 3.1.3a: While the logical framework for the project has served its purpose and was up to the required standard, additional efforts could have been made to align the results-based management used by the project with international best practices in this area by, for example, using neutral indicators, baselines and targets, including output level results, and formulating a more precise articulation of expected results and relevant indicators that reflect said results. 47. Although the project was somewhat small in scope and budget, the importance of a strong logical framework cannot be understated. Robust results-based management is essential for demonstrating what has been achieved and for facilitating ongoing reporting and information sharing among stakeholders. It ensures that realistic development results are transparent and accountable and that they support ongoing learning and adjustments to improve interventions. 48. With regard to this project, there are a few areas where the “Simplified Logical Framework”, found in the project document and in Annex 4 of this assessment, could have been improved. First of all, a logical framework, as a tool for monitoring and reporting, is useful as a basic outline of expected results, indicators, means of verification and risks and assumptions and is often used in project proposals and as a communication piece for partners. Nevertheless, after the proposal stage, it is useful to expand on the logical framework and create a performance measurement framework, which includes additional important details that help to orientate the monitoring and reporting of a project and thoroughly explain the methodology that will be used for collecting data. Unlike the logical framework, the performance measurement framework includes a baseline, targets, data sources, frequency and responsibility. In this way, each indicator includes not only the means of verification but also the baseline at the start of the project, the targets to be achieved by the end of the project (or any specified time frame), the actual source from which the data will be collected (stakeholders, project documentation etc.), the frequency with which this data will be collected, and the person who is responsible for the data collection. While the project document includes all the information and details that were requested for the logical framework section and is therefore sufficient for this purpose in terms of basic requirements, further expansion on the monitoring and reporting framework by the project coordinators would have improved these processes. Using a performance measurement framework during the life of the project, for example, would have helped to facilitate the data collection process and define a clear methodology which the project stakeholders, and specifically ECLAC, could have followed for monitoring and reporting. 12 FINAL ASSESSMENT REPORT 49. With regard to the indicators found within the logical framework, it should be noted that they have been formulated to include the targets for the project’s expected results, which means that they are essentially key performance indicators (KPIs). There has been a recent trend in results-based management to move away from KPIs towards the more robust methodology seen in a performance measurement framework , where a neutral indicator is identified, accompanied by a baseline (to demonstrate a starting point from which to measure change) and targets (to specify a particular value for performance). This is preferable to KPIs since a KPI is formulated with a target integrated within it and does not include a baseline (which is essential as a benchmark for measuring results). Due to the lack of baseline and target, it is more difficult to measure ongoing progress against a KPI, while the methodology found in a performance measurement framework clearly demonstrates the achievement of change and assists in monitoring these results while the project is ongoing in order to make adjustments to project activities as necessary. 50. In addition, the indicators that have been formulated to measure the project’s EAs are not necessarily relevant to the measurement of the proposed expected results. For example, EA 1 is “Increased awareness in Latin American and Caribbean governments, business, communities and civil society of the economic impacts of climate change, as well as adaptation and mitigation options.” To measure this result, the following indicator was used: “At least 20 news items are published/broadcast in mass media discussing contributions of the project.” This indicator does not provide a significant measurement of its corresponding EA and would be much better suited as an output indicator (as is the case with a number of the other indicators found in the framework). Ensuring that 20 news publications are produced and broadcast does not necessarily signify that there is increased awareness among Latin American and Caribbean governments, business, communities and civil society of the economic impacts of climate change. It does demonstrate that some awareness raising has occurred which has been picked up by the media and in the context of the project, in 2009, there were not many news items pointing to the involvement of the governments of the region in climate change activities, particularly ministries of finance. It does, therefore, provide an indication of progress within the region with regard to awareness; however, the EA itself is not fully represented by this indicator. A more suitable and relevant indicator would be, for example, “Number/type of governments, businesses, communities and civil society reporting that they are more knowledgeable/aware about the economic impacts of climate change in Latin America and the Caribbean.” This indicator would be better since it directly reflects the expected results (increased awareness) rather than the process by which this information was spread. Given that data for this suggested indicator would be more difficult to collect and that the scope and size of the project was small, it is easy to understand why a more easily measurable indicator was chosen. Nevertheless, it should be noted that the indicators were not entirely relevant to the expected results and greater efforts should have been made to measure the objectives outlined in the logical framework or to modify the EAs to ensure that they were at a level that the project would be able to measure. 51. As mentioned previously, a number of the indicators would be better suited as output indicators. Nevertheless, the logical framework has only two levels of results: objective and expected outcomes. Generally, a logical framework would include three levels (impact, outcomes and outputs). For this reason, it should also be noted that the logical framework does not have output level results. This additional level (which is defined as the products and services which result from the completion of activities within a development intervention) would have facilitated monitoring and reporting as it would have demonstrated how the 13 FINAL ASSESSMENT REPORT activities undertaken have produced outputs (products and services) that have resulted in the more significant change seen at the EA level. This would have also assisted in ongoing monitoring to ensure the project was on track. 52. Generally speaking, the activities outlined were adequate for attaining the planned EAs; however, further activities were undertaken that are not outlined in the logical framework. 53. In summary, the logical framework could have been improved by: 1. Despite the fact that this wasn’t an ECLAC or Development Account requirement, expanding on the logical framework to create a performance measurement framework for more comprehensive and better planned monitoring and reporting would have facilitated data collection and analysis. 2. Using neutral indicators along with baselines and targets instead of KPIs to clearly demonstrate change from project outset to project completion. 3. Formulating indicators for the EAs that were more clearly reflective of the actual expected results at that level or formulating EAs that were more easily achievable. 4. Including outputs and output level indicators to reflect the products and services that were produced by project activities and the ways in which these products and services contributed to the EAs. 3.2. Relevance 3.2.1. Relevance of the project and project activities to the region and countries KEY FINDINGS 3.2.1a: Overall, the data indicate that the project and its subject matter were relevant for the context within the Latin American and Caribbean region and participating countries. 3.2.1b: Interview and survey respondents generally considered the studies produced by this project to be pioneering with regard to analysing and understanding the economic impacts of climate change. 3.2.1c: According to data collected from key stakeholders, greater coordination with and the participation of key actors and institutions could, in some cases, have increased the relevance of project activities. 3.2.1d: Some respondents felt that better dissemination of the studies would have also contributed to greater relevance of the project within the region. 54. Climate change, which is basically manifested in rising average temperatures, changes in precipitation patterns, rising sea levels, reduction of the ice extent, glacier melt and alterations in the pattern of extreme events, will disproportionately affect developing countries. In Latin America and the Caribbean, the effects of these changes will be observed in many areas including agricultural activity, water availability, forests and biodiversity, sea levels, tourism and health, all of which will also have a significant impact on the economies of the region. In fact, tentative estimates indicate that if the region’s average temperature rises by 2.5°C, the economic costs of climate change will stand at between 1.5% and 5.0% of current GDP, while adaptation costs are calculated at below 0.5% of the current GDP. The Caribbean subregion is particularly exposed to combined phenomena such as rising sea levels coupled with extreme climate events such as drought, flooding, hurricanes and storms. In Latin America and the Caribbean, a transition is needed towards a more sustainable form of development that will preserve the region’s economic, social and natural assets. As such, 14 FINAL ASSESSMENT REPORT economic analysis of climate change in the region is vital not only to identify the main channels, the scale of climatic effects and the best ways of adapting to the new climate conditions, but also to formulate a long-term sustainable development strategy that combines a low-carbon pathway with social inclusiveness. 55. Review and analysis of the project’s logical framework and the associated activities indicates that the project’s design was relevant to the overall objective of improving the ability of the Latin American and Caribbean countries to adapt to, and cope with, the economic impacts of climate change. In particular, the key project activity of supporting the development of both national and regional studies to better understand these impacts was extremely relevant, given that it was a relatively new area of study. With regard to increasing national capacities, technical assistance was planned and undertaken to help support countries in the formulation of these reports as well as the use of methodologies. Furthermore, the project was designed with a view to increasing regional cooperation and the aim of key steps such as regional meetings and the development of a policy at a regional level was to increase coordination through knowledge-sharing and cooperation. Generally speaking, the design of the project, including its overall objective, EAs and activities, was relevant as a first step towards addressing the potential economic impacts of climate change. 56. The data suggest that the activities carried out within the framework of the project and the project itself were also relevant to the regional and country-level contexts. In particular, most respondents felt that the purpose and objectives were particularly important to many of the countries involved and the region as a whole. For example, one interview respondent said that the project was relevant to their country and government, given that they had been working on issues related to climate change for many years, but they had never studied it from an economic standpoint. Furthermore, it was stated that “once you put numbers on something like climate change, people start to better understand the scope and problematic.” Survey respondents also echoed similar thoughts, particularly since no similar studies had yet been completed and these publications were considered basic reference documents that could serve as the starting point for further studies of this nature in the region. Furthermore, the importance of taking a multidisciplinary approach to climate change was also highlighted and the project was seen as having contributed to this effort. 57. Those who felt that project activities were not entirely relevant to the context provided a variety of reasons. For example, it was stated that the regional document “La economía del cambio climático en América Latina y el Caribe. Síntesis (2009)” was not relevant given that it did not have an impact on public policies following publication and dissemination. Other respondents felt that the timeliness and dissemination of key documents could have been better, and more publicity would have helped to boost the accessibility of the information produced in the reports, thereby enhancing the project’s relevance. For example, one respondent from the Plurinational State of Bolivia said that since their country-level publication took a number of years to be finalized, they did not have the opportunity to publicize that study more broadly during project implementation.4 4 It should be noted that the publication and dissemination activities associated with the study for the Plurinational State of Bolivia depended on funds from the Inter-American Development Bank, which was the main donor for this activity. It is included in this assessment because it is a part of the larger RECC (Review of the Economics of Climate Change) initiative, but this particular study was not supported by ECLAC alone. 15 FINAL ASSESSMENT REPORT 58. In addition, some respondents felt that particular stakeholders who were key to implementation were not sufficiently involved in the project activities. For example, it was stated that “…in designing and implementing [a] climate change project they [ECLAC] should include the major stakeholders such as the Focal Points to UNFCCC.” Furthermore, a key partner institution in the Caribbean subregion also felt that they were not sufficiently involved and that in essence they were brought in “on the back end”, which defeated the purpose of coordination. Thus, while the subject matter of the project is indisputably relevant to the countries in the region, a lack of proper coordination with key institutions led to a project and project activities that were somewhat less relevant than they could have been had some of the relevant stakeholders in climate change in the region been better involved. 59. Overall, the data indicate that identifying and understanding the economic impacts of climate change are essential for the region, hence this project is considered quite relevant. In terms of having a better overall understanding of the impacts of climate change in Latin America and the Caribbean, the studies produced by this project will serve as a good basis and starting point regarding an issue that needs to be better understood and studied into the future. Furthermore, future and similar initiatives would want to make an effort to ensure that key stakeholders are involved at all stages of implementation and that reports are promptly and widely distributed, with a view to enhancing the overall relevance of the activities. 3.2.2. Links to the ECLAC mandate KEY FINDINGS 3.2.2a: The project was and still is very much in line with the ECLAC mandate and it contributed significantly to the organization’s climate change work. 60. With regard to the ECLAC mandate, this project is clearly in line with a number of the organization’s key priorities, including undertaking studies, research and other support activities; promoting economic and social development; gathering, organizing and disseminating data and information; and promoting development cooperation activities and projects of both a regional and subregional scope.5 In addition, the project was designed to contribute to ECLAC subprogrammes 8 (sustainable development and human settlements) and 12 (subregional activities in the Caribbean). 61. The Sustainable Development and Human Settlements Division at ECLAC aims to “contribute to evaluation of progress in public policies, instruments and institutions, with a view to promoting more inclusive economic activity with a smaller environmental footprint in the countries of LAC.” 6 It focuses on the three dimensions of sustainable development: the environmental, social and economic. Within the context of this ECLAC project, it is clear that both the environmental and economic sectors have been taken into consideration (financial impacts of climate change) with less of an emphasis on the social aspects. In fact, all ECLAC respondents felt that the project was very much in line with the mandate and priorities of ECLAC. One respondent stated that this project had contributed significantly to the topic of climate change within the organization and helped to open up a new area of work in 2008 5 http://www.cepal.org/en/mandato-y-mision 6 http://www.cepal.org/en/work-areas/sustainable-development-and-human-settlements 16 FINAL ASSESSMENT REPORT (the beginning of this project). Furthermore, it made an important contribution to this division and to many other divisions working on climate change activities across ECLAC. 3.3. Effectiveness 3.3.1. The project’s indicators and expected results KEY FINDINGS 3.3.1a: Overall, and based on the logical framework designed and the indicators that were chosen, most of the project’s expected results were accomplished. 62. In order to determine the effectiveness of the project in terms of its contribution to improving countries’ understanding of and ability to adapt to the economic impacts of climate change, a review of the project indicators and their results is presented in the table below. The overall objective of the project was to “to improve the ability of Latin American and Caribbean countries to adapt to, and cope with, the economic impacts of climate change on development.” The following EAs and indicators were anticipated in order to achieve this objective: Intervention logic Expected accomplishment 1 Increased awareness in Latin American and Caribbean governments, business communities and civil society of the economic impacts of climate change, as well as adaptation and mitigation options Expected accomplishment 2 Strengthened national capacities to identify responses to the potential economic impacts of climate change Indicators At least 20 news items are published/broadcast in mass media discussing the project’s contributions. At the end of the project, at least 25 professionals from different governmental and nongovernmental institutions of at least six countries have been trained in the methodologies for assessing the physical and economic impacts of climate change. In the Caribbean, at least two countries prepare national reports on impacts and policy options, as inputs for the high-level meetings on the project. In Latin America, at least two countries prepare national reports on the economic impacts and decide to adopt a policy framework in at least one of the identified vectors of impact. Results More than 150 news items were published from 2008 to 2011 in international media promoting the results of the two regional studies as well as those of national studies for countries such as Chile, Ecuador, Uruguay and Paraguay. More than 50 professionals from different governmental and nongovernmental institutions from eight South American countries and from the Central American and Caribbean subregions have been trained in methodologies to assess the economic impacts of climate change. In the Caribbean, two countries have adopted policy frameworks in at least one of the identified areas of impact. In Latin America, Chile and Uruguay completed national reports on the economic impact of climate change and two regional reports on the economic impact of climate change have been prepared in the framework of two international conferences COP 15 (Copenhagen, 2009) and COP 16 (Cancun, 2010). 17 FINAL ASSESSMENT REPORT Intervention logic Expected accomplishment 3 Strengthened regional cooperation to address the economic impacts of climate change. Indicators A subregional policy framework to address climate change is produced and endorsed by governments in the Caribbean. Results High-level meetings were held with participation from both Caribbean and Latin American countries and ECLAC in the Caribbean continued to interact and work closely with the Caribbean Community Climate Change Centre (CCCCC) in preparation of the regional policy, although a subregional policy framework was never developed. 63. As can be seen above, for EA 1, the expected results of the indicators (20 news items and 25 trained professionals) have been exceeded by the project activities. EA 2, which aimed for two countries in each subregion (Latin America and the Caribbean) to have published reports and adopted policy frameworks, has also been achieved to a certain extent, although it was expected that policy frameworks would be adopted in the Latin American countries but they were actually adopted in the Caribbean countries. As for EA 3, a subregional policy framework to address climate change was never produced, although efforts were made to collaborate regionally on this area of work. Overall, and based on the logical framework designed and the indicators chosen, most of the project’s expected results were accomplished. 3.3.2. Increased awareness (EA 1) KEY FINDINGS 3.3.2a: The data collected suggest that the awareness and knowledge of beneficiaries and the public of the economic impacts of climate change in the region was increased through the publication, dissemination and use of the reports. This was validated in both project reports and by participants. 3.3.2b: According to the data collected, the type of activity that was most useful for increasing awareness was technical assistance from ECLAC, an example of which was the provision of support to national teams on climate change issues and methodologies. 3.3.2c: Generally speaking, data indicate that better publicity and distribution of results could have improved awareness of the economic impacts of climate change in the region even further. 64. As stated in the final report for the project, there were approximately 400,000 website downloads of both the regional and national studies from November 2009 to December 2011. In addition, more than 150 news items were published during the 2008-2011 period in international media promoting the results of the studies in countries such as Chile, Ecuador, Uruguay, Colombia and Paraguay. It should be noted that these news items were collected specifically for the timing of the project’s report and many of them were related to the event launches. Similarly, beneficiaries also noted increased awareness as a success of the project. For instance, when asked about the ways in which project activities contributed to awareness of the economic impacts of climate change in the region, one respondent stated that they were particularly important for raising public awareness: 18 FINAL ASSESSMENT REPORT The results were first page of the newspapers of the country, particularly the newspapers that have a national coverage. In terms of public awareness, the fact that it had such a great media coverage means that it raised the public awareness. This was the first time we came up with a number in terms of GDP.7 65. To a certain extent, this demonstrates that the project and its many studies contributed to greater public awareness of the economic impacts of climate change in Latin America and the Caribbean. Nevertheless, data suggested that there could have been greater dissemination of results to both the public and beneficiaries through, for example, further national and regional seminars and workshops. This is also in line with the conclusion that was highlighted above in section 3.2, whereby some respondents felt that the timeliness and dissemination of key documents could have been improved and more publicity would have helped to increase the accessibility of the information found in the reports. While it is clear, therefore, that the project and its results were covered in the national media and the documents produced have been accessed, further distribution and publicity would have and could still aid in ensuring that the results are widely disseminated. 66. With regard to whether or not the project and its activities contributed to increased awareness and knowledge among beneficiaries, figure 3 below summarizes the survey responses provided by partner institutions and ECLAC in this regard: Figure 3 Extent to which partner institutions and ECLAC feel that project activities contributed to beneficiaries’ awareness of the economic impacts of climate change (surveys) 20 15 11% 10 55% 5 0 Strongly agree 4% 22% 4% Agree Disagree Partner institutions ECLAC 4% Strongly disagree 67. As depicted above, the majority of partner institutions and ECLAC survey respondents agreed that the activities of the project helped increase awareness of the economic impacts of climate change among beneficiaries. For example, when asked about the greatest achievements of the project, one partner institution explained that it had contributed to putting the issue on the agenda of the Latin American and Caribbean countries and to highlighting the need for adaptation and mitigation options. 7 It is likely that this actually refers to a study that was published in 2014, which fell within the framework of Development Account project ROA-145. Nevertheless, given that the project referred to in this report was essentially phase one of continued work in the area of assessing the economic impacts of climate change, it is also relevant to mention it here. 19 FINAL ASSESSMENT REPORT 68. Although the survey response rate from beneficiaries was low, figure 4 below summarizes respondents’ views regarding the extent to which particular activities contributed to their awareness in this area. Figure 4 Extent to which beneficiaries feel that the activities in which they participated contributed to an increased awareness of the economic impacts of climate change (surveys) 2,5 100% 2 1,5 33%33% 1 0,5 0 Strongly agree 33%33% 33% 33%33% Agree Disagree Strongly disagree Technical assistance Studies / Reports Events / Seminars 69. As seen in figure 4 above, many of the beneficiaries felt that the activities in which they participated had contributed to their own knowledge of this area of climate change. For example, in one interview, the respondent explained that project activities had an important connotation for both the public and private sector, given that the national study had been cited by many different actors in the country and had become an important reference document. The influence of the national studies on the public and private sectors is further demonstrated in the final report for the project, which states that “completion of the studies of Chile and Uruguay has motivated participation of the business community as well as NGOs” clearly indicating greater awareness of these issues. In addition, it was noted that the study in the Plurinational State of Bolivia had helped to boost awareness, particularly among government officials at both a national and municipal level, regarding water scarcity as well as disasters and flooding in the Amazon. In contrast, at least three respondents felt that neither the studies nor the events had increased their awareness. For example, one participant said that the event they attended had not contributed to their knowledge of the issue, as they felt that the consultants involved were not overly aware of the particular impacts of climate change on the Caribbean subregion. However, in terms of the technical assistance provided by ECLAC to countries on climate change issues and methodologies, both respondents who had participated in this type of activity felt that it had significantly increased their knowledge and awareness of the subject matter. 70. It is clear from the data collected that the project and its activities have contributed to increased awareness and knowledge of the economic impacts of climate change as well as options for adaptation and mitigation within Latin America and the Caribbean. Generally speaking, it is evident that the project contributed to increased awareness among both the public and beneficiaries. Furthermore, one of the project’s greatest achievements was its contribution to analysing climate change impacts through an economic lens as opposed to an exclusively environmental one, and hence providing the basis for an examination of the financial costs associated with mitigation and adaption versus the economic costs of inaction. This was supported by respondents, who referred to the studies as a “pioneering analysis” 20 FINAL ASSESSMENT REPORT and the first attempt to put the issue of the financial costs of climate change on “the table”. Overall, the activities of this project have served to introduce or to further this issue among a number of different actors, thus contributing to knowledge and awareness of the economic impacts of climate change in the region. 3.3.3. Strengthened national capacities (EA 2) KEY FINDINGS 3.3.3a: The data suggest that project activities have contributed sufficiently to the strengthening of national capacity for identifying solutions to address the economic impacts of climate change. This was supported by both the project documentation and interviews and surveys. 3.3.3b: Key contributions have been made to strengthening the capacity of some governments to incorporate the results of the studies into national plans, policies and documentation. 3.3.3c: According to survey data, a significant challenge of the project was its interdisciplinary and multisectoral nature, as this required coordination between various different sectors and ministries. 3.3.3d: It was felt that, in some cases, there was limited knowledge and capacity at the country level to undertake a study of this nature due to a lack of qualified specialists in the specified thematic areas. 71. The project’s second EA aimed to strengthen national capacities to identify responses to the potential economic impacts of climate change. This included training government officials to assess the impacts of climate change, producing reports that could be used for policies, research and decision-making, and improving national coordinating strategies and political coordinating mechanisms for designing responses to policies related to climate change. 72. With respect to this EA, the data suggest that some capacity-building did occur. For example, the final report for the project states that more than 50 professionals from different governmental and non-governmental institutions in eight South American countries and from the Central American and Caribbean subregions had been trained in methodologies to assess the economic impacts of climate change since the start of the project. Furthermore, in the Caribbean, a training workshop on climate modelling and the economic valuation of climate change impacts was convened in November 2009, attended by 28 professionals from governments, the private sector and civil society. Ninety-two per cent of participants rated the workshop positively in terms of having built capacity in the areas of climate modelling and the economics of climate change. 73. Moreover, survey responses also indicated that the capacity of beneficiaries to identify solutions to the economic impacts of climate change was increased either ‘sufficiently’ or ‘a lot’ through the implementation of project activities, as depicted in figure 5 below. 21 FINAL ASSESSMENT REPORT Figure 5 Extent to which project stakeholders feel that the activities in which they participated contributed to the capacity of beneficiaries to identify solutions to the economic impacts of climate change (surveys) A lot 20% 8% 60% Sufficiently 60% 46% 20% A little 27% 20% Not at all 20% Not enough information to… 0 19% 5 10 15 Beneficiaries Partner institutions ECLAC 20 74. While 24 of the stakeholders surveyed felt that beneficiaries were better able to identify solutions to the financial impacts of climate change (either ‘sufficiently’ or ‘a lot’), eight stakeholders felt that project activities contributed only ‘a little’, while one felt that they did not contribute at all. Overall, while the capacity of beneficiaries (in this case, government institutions) to identify solutions does seem to have been increased, data suggest that it is to varying degrees. This could be in part due to the fact that many countries and stakeholders were involved and therefore different results were obtained based on country-level factors, such as the changeover of staff in the participating ministries, the overall country capacity to engage in activities, political buy-in, etc. 75. With regard to how the project activities and studies have contributed to identifying solutions, one respondent mentioned that it had provided countries with the ability to measure the impact of climate change in economic terms and hence contrast it with national and regional GDP. Another respondent explained that the issue had been clearly documented by the studies, meaning it cannot be negated by national governments. In the case of Uruguay, it was mentioned that these studies contributed to the development of the “Sistema Nacional de Respuesta al Cambio Climático y la Variabilidad” which is essentially a new form of non-compartmentalized institutionalism to promote horizontal coordination and integration of national technical capacities with relation to climate change across the country. This system is a national coordinating strategy that has been developed to help identify solutions to climate change. The development of this system demonstrates that project activities contributed not only to countries’ abilities to identify solutions to address the economic impacts of climate change, but also to national coordinating strategies (as also depicted in figure 6 below). 22 FINAL ASSESSMENT REPORT Figure 6 Extent to which stakeholders feel that the project activities have contributed to improved national coordinating strategies for climate change (surveys) A lot 8% 60% Sufficently 25% 29% A little 25% 37.5% Not at all 25% 20% Not enough information to respond 25% 25% 0246 Beneficiaries Partner institutions ECLAC 20% 8 10 76. Further data also confirmed that the studies and their results have contributed to plans and policies at the national level. For instance, an interview respondent from Colombia explained that their country-level study was part of the national coordination process on how to approach policy design in climate change, and the results have been used in climate policy. More concretely, Colombia has been drafting a national adaptation plan that uses the results of the study to contextualize and analyse the challenges and the country’s strategic response. Similarly, a respondent from Chile mentioned that the results also contributed to their country’s adaptation plans. For example, these results have been cited in a number of different documents by many actors and the study itself has become an important reference document for the country. Furthermore, in Chile as well as in Argentina it was mentioned that the study had also contributed to their National Communications on Climate Change for UNFCCC. 77. Despite these accomplishments, it was mentioned that a significant challenge for the project was the inability to make progress on the actual implementation of actions and interventions identified in the study. While this is somewhat outside of the scope of the project, it could be considered in terms of the next steps. 78. According to the data, one of the biggest challenges in terms of effectiveness, which contributed to complications when it came to developing the studies, was their multidisciplinary and intersectoral nature, which required the participation and coordination of actors from different sectors and ministries. At least nine respondents mentioned this as one as the greatest challenges of the project, explaining that it was difficult to involve all sectors and have a continuous and clear conversation across disciplines. For example, one respondent mentioned that collaboration between climatologists and economists posed a challenge. Another respondent explained that the development of this type of study was challenging since it applied both scientific and technical knowledge that was completely novel and unknown and which spanned a number of thematic areas. In some cases, respondents cited coordination among ministries (particularly those that do not usually work on climate change issues) as also complicating the process. While strategies such as the use 23 FINAL ASSESSMENT REPORT of the National Advisory Panel were employed, for future studies that take a multi-sectorial or interdisciplinary approach, additional techniques should be considered for ensuring smooth coordination and integrated results. For example, a longer time period for the project could help ensure that interdisciplinary teams (such as economists and environmentalists) develop a common language and knowledge to discuss and research the issues presented, or that opportunities for training in the specified sectors are identified. 79. An additional challenge mentioned by a number of respondents was the limited knowledge and capacity at the country level to undertake a study of this nature. This finding was also echoed in the final report for both the South American and Caribbean subregions. In South America, it was mentioned that it was sometimes difficult to identify qualified sector specialists, while in the Caribbean it was stated that the limited range of skills and experience in various modelling techniques slowed implementation. Furthermore, it was found through both interviews and document review that another recurring challenge in terms of capacity-building was the changeover of government staff, which caused substantial delays. With regard to the difficulty of involving government officials in the study in the Plurinational State of Bolivia, it was recommended that perhaps this type of study would be improved by working more closely with universities and researchers from universities, particularly since a number of the consultants were from the academic field. 80. The general consensus derived from the data was that the project and its activities had contributed in certain key ways to the strengthening of national capacities with regard to identifying and addressing the economic impacts of climate change. The most notable contributions had concerned increasing the capacity of governments to incorporate these issues into national plans and documentation and assisting in the identification of key solutions. In terms of challenges, the interdisciplinary nature of the project did pose some problems in terms of being able to effectively ensure coordination, collaboration and integration of all the varying views, and in some cases national capacity was lacking to carry out certain activities. 3.3.4. Strengthened regional cooperation (EA 3) KEY FINDINGS 3.3.4a: Data suggest that regional cooperation was strengthened in some ways and to a limited extent through a number of regional events which were supported by the project and which gave the project stakeholders the opportunity to collaborate. 3.3.4b: The most significant output of the project in terms of supporting regional cooperation is the document “La economía del cambio climático en América Latina y el Caribe, Sintesis.” This document provided an overview of the economic impacts of climate change in the region, and was noted as the first regional attempt to document this issue. 3.3.4c: The expected result of a subregional policy for the Caribbean was not accomplished; however, some effort was made to document information that could contribute to this policy in the future. 81. The project also aimed to strengthen regional cooperation to address the economic impacts of climate change. As per the project document, this included holding a number of high-level meetings for sharing information on methodologies and methods for assessing the economic 24 FINAL ASSESSMENT REPORT impacts of climate change, as well as developing and endorsing a subregional policy framework to address climate change in the Caribbean. 82. With regard to the subregional policy for the Caribbean, unfortunately this was never developed. Nevertheless, in June 2010, a meeting was held entitled “towards development of a climate change policy for the Caribbean”, and 13 countries were invited to present their approach to climate change. An ECLAC stakeholder said that, while a subregional policy had not been developed, efforts had been made to identify and document elements of national policies that could potentially contribute to a subregional policy at a later stage. 83. At the regional level, a number of high-level events were organized to promote cooperation, such as the side events that were held at both COP15 in December 2009 and COP16 in December 2010 to discuss the economic impacts of climate change, and a highlevel event in Santiago in August 2009 to present and discuss the preliminary results of the studies. There were thus various opportunities for regional counterparts to come together to discuss the economic impacts of climate change as well as the progress of both the national and regional studies. This indicates that efforts were made to promote regional cooperation and some opportunities to discuss regional issues and collaborate were available to partner institutions and beneficiaries. 84. In addition to this, one of the biggest outputs of the project was the regional study: “La economía del cambio climático en América Latina y el Caribe. Sintesis” which was first published in 2009 and included preliminary information that had been collected by the study teams and by ECLAC. It was published again in 2010, containing enhanced results and further analysis and information. This was the first document that aimed at exploring the economic impacts of climate change in the Latin American and Caribbean region. It is a consolidation of all the national level studies and provides a regional overview. While it was the first of its kind, one ECLAC stakeholder did mention that one of the biggest challenges of the project was consolidating all the national level studies into one document. Nevertheless, many respondents said that the regional nature of this study and the project in general was one of the project’s greatest achievements, particularly since it was the first study on this subject matter for Latin America and the Caribbean. Overall, this study was a significant output of the project and provided a basis for regional cooperation in terms of issues related to the financial costs associated with climate change. Figure 7 Extent to which partner institutions and ECLAC feel that project activities have contributed to strengthened regional cooperation 14 12 10 60% 8 6 4 50% 2 0 Strongly agree 39% Agree Partner institution 20% 11% Disagree ECLAC 20% Strongly disagree 25 FINAL ASSESSMENT REPORT 85. Survey data also support the conclusion that regional cooperation was strengthened through project activities. In fact, only four respondents either disagreed or strongly disagreed with this. Of note is the fact that, of the respondents who disagreed or strongly disagreed, two were from ECLAC, indicating that from an institutional and internal perspective, the project may have been lacking with regard to the promotion of regional cooperation. In contrast, other respondents said that this project had put the topic on the public agenda and helped to facilitate coordination between ministries or levels of government and other political actors across and among countries. It was also mentioned that the project had provided opportunities to participate in meetings and other activities within other countries of the region and to collaborate with regional counterparts. Furthermore, it was stated that there had been a significant increase in collaboration and coordination among the researchers of the project, as a number of academics across the region had participated and have continued to work together. 86. Beneficiaries who had participated in technical assistance activities strongly agreed that this activity had contributed to regional cooperation. For those who participated in events and seminars, opinions were divided equally; while some respondents felt that the activities had contributed to regional cooperation, an equal amount felt that they had not. Although the beneficiary response rate for the surveys that contributed to this project assessment was quite low, data suggest that technical assistance appears to have been the most satisfactory activity in terms of results. 87. While the expected subregional policy for climate change in the Caribbean was not produced, some opportunities for regional cooperation were seized. This included the regional events organized at the COPs as well as regional opportunities to review and share information on methodologies for understanding the economic impacts of climate change. Furthermore, while the publication of a regional report that synthesizes the impact of climate change on the region’s economies does not necessarily indicate increased regional cooperation, it can be seen an important tool in this regard. 3.3.5. Methodologies used for studies KEY FINDINGS 3.3.5a: Data suggest that, while the methodologies used for the studies were deemed as sufficient by some, a few others felt that they were not necessarily representative of their context or that they required further support in order to implement them. 3.3.5b: A key challenge in implementing the methodologies that was identified by respondents and during the document review was the lack of data in some countries for climate modelling. In addition, future initiatives should provide countries with further support in order to collect the required data for climate models as well as with regard to developing and implementing the methodologies. 88. Additional challenges that were noted in terms of the effectiveness of the project and its studies concerned the methodologies that were used and the availability of information. Since the studies produced were based on complicated climate models for the assessment of the financial impacts of climate change, this led to difficulties in terms of identifying the right methodologies for all national contexts and applying this model on the basis of the data that were available at the country level. 26 FINAL ASSESSMENT REPORT 89. As mentioned in the final report for the project, key problems that were encountered in the South American subregion in this regard stemmed from the methodological approach for each study, the lack of information in some sectors, and the complex identification of qualified sector specialists. Similarly, in the Caribbean it was stated that “it proved difficult to obtain data to run the modelling exercises for which the best available data should be used; the time and effort to collect such data needs to be incorporated into the project approach.” This was also reiterated in survey and interview responses. For example, a number of respondents said that utilizing the methodologies provided and receiving sufficient support to apply them presented challenges. Similarly, one interviewee felt that the methodologies and the results that they produced were not representative of the actual realities in the country and were hence not relevant for that particular context. In addition, a number of respondents noted that the information needed as input for the climate models was not always available or sufficiently robust to ensure that the results provided proper projections. For example, in the Plurinational State of Bolivia it was stated that better country-level data was needed to improve future projections, while for the Caribbean subregion it was mentioned that data represented a big challenge given that relevant data was often not collected, not properly collected, not documented or not disseminated, despite researchers’ best efforts. At least one person mentioned that further support in terms of applying the methodologies would have been useful. 90. In contrast, a few respondents mentioned that the results were acceptable for future research and that the methodology was adequate for the completion of their study, indicating that whether the methodology was appropriate or successful depended on the particular country. For example, a country with greater capacity and more advanced data would be much more likely to produce an accurate and representative study, while a country without these necessary inputs would be hard pressed to complete a study of this nature. 91. In order to ensure the production of subsequent studies that address the economic impacts of climate change at a country level, further support is needed to ensure that the countries involved are collecting the data required to run the climate models, and additional technical support should be provided wherever possible. Nevertheless, given that at least a few of these studies have produced relevant results that have been integrated into national policies and plans, in some cases the methodologies used and data collected do appear to have been sufficiently sound. 3.3.6. Gender mainstreaming and human rights KEY FINDINGS 3.3.6a: It is clear that women did participate in the project and were somewhat represented across the activities. Nevertheless, additional efforts to demonstrate that gender mainstreaming was a conscious strategy throughout the life of the project would have been beneficial. 3.3.6b: The data demonstrate that while civil society, youth and ethnic minorities were somewhat involved in project implementation, these groups primarily participated in the dissemination of results, since much of the work undertaken consisted of desk research and was performed by government officials and academics. 27 FINAL ASSESSMENT REPORT 92. Given that the main outputs of this project were the studies on understanding the potential impact of climate change in Latin America and the Caribbean, a focus on mainstreaming gender is important in terms of involvement of participants in specific activities but less so in terms of the actual results of the studies. Generally, throughout the documentation it is clear that both men and women were engaged in formulating the studies and in capacity-building or informational events. 93. With regard to participation in events, according to the participant lists that were provided, there appears to have been a consistent balance in terms of women and men attendees as well as participants in panels / presenters. In some cases, however, women were significantly less represented than men. For example, four expert group meetings were held in the Caribbean as part of the project with participants from 13 countries; of a total of 68 participants, only 19 were women. This means that 27% of participants were female and 73% were male. However, this is not always something within the control of the project, since the gender of particular experts may be less important than their experience in many cases. Nevertheless, given that a clear indication of how gender was considered throughout project implementation was not entirely evident from the data collected, some additional efforts to demonstrate that gender was taken into consideration could have helped to demonstrate a more equitable approach to implementation. 94. In terms of the extent to which the project involved specific relevant and vulnerable groups, such as civil society, youth and ethnic minorities, it was generally felt that something of an effort was made, as depicted in figure 8 below. Figure 8 Extent to which partner institutions and ECLAC project managers feel that civil society, youth and ethnic minorities were involved in the project (survey) 14 12 10 8 6 4 2 4% 20% 0 A lot 54% 12.5% 40% 29% 40% Sufficiently A little Not at all Partner institutions ECLAC Without sufficient information to respond 95. Overall, 54% of partner institutions who responded to the survey felt that the project involved civil society, youth and ethnic minorities a little, while 40% of ECLAC survey respondents felt the same. These data demonstrate that the project took into consideration these other groups of stakeholders to a certain extent but it by no means points to a strong and coherent strategy in this regard. Generally, it was felt that, since the project was mostly 28 FINAL ASSESSMENT REPORT focused on the studies and these consisted essentially of desk research, the opportunity to involve these groups did not always present itself. In addition, while the technical side of research and writing primarily focused on working with the government and academics, it was during the dissemination of results that all actors affected by climate change were involved, with a particular emphasis on civil society. 3.4. Efficiency 3.4.1. Organizational factors and project management by ECLAC KEY FINDINGS 3.4.1a: According to the data collected, project procedures and processes were sufficiently implemented and key mechanisms (such as the International Advisory Panel and National Advisory Panels) were created and implemented to support project activities. 3.4.1b: Overall, project beneficiaries were satisfied with ECLAC project management and felt that the support provided was very useful. 96. Overall and as seen in the previous section on effectiveness, it is clear that the project achieved a number of important results within a short time frame and with limited resources (the total budget for the project was US$ 558,000). These resources were divided between the two ECLAC offices (Santiago and Port of Spain), both of which provided capacitybuilding on climate change modelling methodologies and offered key support to a number of countries for conducting the studies on understanding the economic impacts of climate change. Considering what was accomplished in three years with a relatively small budget split between two offices, it is evident that project activities were efficiently implemented. 97. The data suggest that a number of organizational and management processes were put in place to support project implementation. For example, many of the survey respondents (64%) from both ECLAC and partner institutions agreed that sufficient procedures and processes were established at the beginning of the project (the other 36% of respondents stated that they did not have enough information to answer the question). In terms of their effectiveness, of the 20 survey respondents who acknowledged that procedures and processes had been established, many (70% or 14 out of 20) replied affirmatively that they were effective while only one person replied negatively and five others were unsure. 98. Data collected from ECLAC stakeholders through interviews as well as document review both support these findings. For example, data from these sources pointed to a number of mechanisms that were established to support implementation of the project, such as: • An International Advisory Panel consisting of expert consultants in the areas of mitigation, adaptation and climate modelling to assist with establishing methodological and strategic lines for the studies. • National Advisory Panels in which two to three members were designated from each country to support project activities. • Technical teams created at the country level to support the development and formulation of the reports; these consisted of a national coordinator to organize the study and various consultants in specific thematic areas. 29 FINAL ASSESSMENT REPORT 99. With regard to the actual activities that were implemented within the framework of this project, table 2 below summarizes survey responses from beneficiaries. Table 2 Beneficiaries’ perceptions of the efficiency of activities Type of activity Events / seminars Quality rating High: 4 respondents (100%) Satisfaction with activity and support of ECLAC Sufficiently satisfied: 2 Respondents (50%) A little satisfied: 1 respondent (25%) Do not have enough information to answer: 1 respondent (25%) Effective and efficient implementation of activities Yes: 4 respondents (100%) Studies / reports Technical assistance High: 3 respondents (60%) Skipped the question: 2 respondents (40%) N/A Very satisfied: 1 respondent (20%) Sufficiently satisfied: 2 respondents (40%) Skipped the question: 2 respondents (40%) Very satisfied: 1 respondent (50%) Sufficiently satisfied: 1 respondent (50%) N/A Yes: 2 respondents (100%) 100. Beneficiaries were clearly quite satisfied with the quality and efficiency of the activities as well as the support provided by ECLAC. This was also echoed in interviews, where beneficiaries positively rated ECLAC support, explaining that it was essential to the success of the studies and the project as a whole. For example, it was noted that: “the truth is that I am completely satisfied. It was a very good experience and it was a great contribution that was provided by ECLAC.” This appeared to be the overall consensus among respondents in this regard. 101. With respect to the project’s financial efficiency, as noted in the final report for the project, the budget expenditures indicate that the funds were properly allocated to their expected allotment areas with a final implementation rate of 86% and a balance remaining of US$ 79,739.86. It is further explained that a planned event had been rolled into an existing meeting that was taking place, which resulted in lower costs for the project; this explains the unused funds. In this way, it appears that the financial management of the project was also quite efficient and the budget allotments were adhered to. This conclusion is also supported by survey responses from ECLAC project managers wherein the majority of respondents felt that the invested resources were used efficiently and effectively to produce planned results and that these resources were available on time. Nevertheless, it was noted that greater flexibility from the Development Account in terms of the funds associated to the budget lines would have been useful. 30 FINAL ASSESSMENT REPORT 102. Overall, based on the data collected, it is evident that this rather small project contributed to a number of important meetings, events, studies and opportunities for countries to work together, and used funds generally as planned. This cumulatively demonstrates that the resources provided were utilized in an efficient way. Furthermore, a number of processes and procedures for implementation that were put in place to ensure project efficiency were appreciated and successful. Overall, data suggest that stakeholders were satisfied with the support of ECLAC and the activities implemented. 3.4.2. Complementarities and synergies KEY FINDINGS 3.4.2a: The data indicate that this project created complementarities with other international projects and partners as well as synergies among and within national government ministries and sectors. 3.4.2b: Based on the data collected, collaboration within ECLAC and between the two implementing offices was fairly good. 103. Based on the documents reviewed and responses from the surveys and interviews, it is apparent that a number of complementary initiatives emerged during project implementation which contributed to various synergies in the area of climate change through the expansion of research matters that complemented the other work being done by ECLAC. For example, the final report highlights a complementary study that was catalysed by the project, which is an analysis of the effects of climate change on the Latin American and Caribbean coast. This project was financed by the Government of Spain in collaboration with ECLAC and contributed to understanding the overall impacts of climate change in the region. Furthermore, in the Caribbean subregion, supplementary funds were generated for a parallel project entitled “Review of the Economics of Climate Change in the Caribbean (RECCC)” in collaboration with the Department for International Development (DFID) of the United Kingdom. This project provided significant funds to conduct national sectoral studies, which in turn informed the regional studies and macroeconomic assessment of the project. It should also be noted that additional funding was provided by other donors, which extended beyond the life of this project. 104. This is in line with what was reported, as it was explained that there had been a conscious effort to create synergies with other similar projects so that they were not working in parallel but rather contributing in a complementary fashion to one overall initiative which was being supported by various donors and continued well after the particular project being evaluated. This can be seen as a number of complementary initiatives have been undertaken with support from various partners (such as the Governments of Denmark, Germany, Spain and the United Kingdom) and ECLAC now considers this research area to be a permanent and increasingly demanding area of work for the organization. Consequently, despite the short time frame of this project (2008-2011), activities aimed at producing national studies on the economic impacts of climate change continued with the support of ECLAC and other donors and are ongoing to this day. For example, another Development Account project being implemented by ECLAC, which aims to improve the management of financial resources for sustainable development, continued to carry out support activities for national studies on the economics of climate change in the region. 31 FINAL ASSESSMENT REPORT Figure 9 Extent to which partner institutions and ECLAC feel that project activities were effective in creating synergies with others (surveys) 14 12 10 8 6 80% 4 2 13% 0 Very effective 54% Sufficiently effective 20% 8% A little effective Not effective Partner institutions ECLAC 25% Without sufficent information to respond 105. As can be seen from the figure above, survey respondents from partner institutions and ECLAC generally felt that the project had been very or sufficiently effective in terms of creating synergies with other partners and projects. This supports the data obtained from the documents and the interviews. For example, two interview respondents said that the project was successful at least in part because a number of different areas of the government and sectors in the country had been involved, hence ensuring sustainability and coherence through close cooperation with key actors. In this way, not only was the project implemented in synergy with other international donors and relevant projects, there was also close collaboration with various ministries and sectors at the country level. 106. With regard to coordination within ECLAC, and in particular coordination between the two regional offices (Port of Spain and Santiago), data suggest that this was fairly good. For example, it was stated that both subregions participated in certain meetings and events held for the project as a whole, and that there were key exchanges between the two offices, particularly with regard to the methodologies used for the studies. Representatives from the Port of Spain office felt that they had full support from the Santiago office if and when they needed it, yet they were left to implement their project activities and work based on their own needs and requirements. It appears that most of the coordination between the two offices concerned the technical methodology used for the studies, particularly during the early phase of the project. 107. Overall, the project appears to have created synergies among projects and partners working in the area of climate change throughout the region. Coordination within ECLAC itself was also fairly good. However, as mentioned in section 3.2.1, some respondents did feel that particular key institutions such as UNFCCC and other regional institutions could have been better included in project activities. Nevertheless, the data demonstrate that some significant complementarities and synergies were created with project stakeholders, existing projects and even country-level government and sectors. 32 FINAL ASSESSMENT REPORT 3.5. Sustainability and replication 3.5.1. Replication of activities and sustainability within participant countries KEY FINDINGS 3.5.1a: Overall, the data collected demonstrate that the project activities are highly replicable and have in fact been replicated in the years since this project came to an end. In addition, the studies that have been completed should be periodically updated to ensure that changing conditions and contexts are taken into consideration. 3.5.1b: Some respondents said that the project’s efforts had contributed to sustainable results in their countries; however, in a number of countries, support continues to be provided by the international community and has not been sustained nationally. 108. The premise of the project’s activities, which focused on the production of country-level studies for understanding the economic impacts of climate change in the region, lends itself well to replication. Not every country in the region was successful in producing a study and many would require subsequent studies in order to update the data and information. Replication of the project activities is therefore not only feasible in the region but also necessary. This was supported by the data collected through both surveys and interviews as well as the documents. For example, when survey respondents were asked about whether or not the project activities could be replicated, many indicated that this would be likely and necessary, as indicated in figure 10 below. Figure 10 Number of survey respondents who felt that the project activities could be replicated 25 20 83% 15 10 67% 17% 5 33% 0 50% Yes 25% No 25% Do not have sufficent information to answer Beneficiaries Partner institutions ECLAC 33 FINAL ASSESSMENT REPORT 109. Overall, the data suggest that the activities of the project are highly replicable. Not only could the studies be replicated in other countries or regions, but existing studies could also be updated. For example, a few survey respondents said that it would be useful to replicate these studies in the same countries in which they had already been completed. Updating the data in these reports would provide for analysis and comparison between previous studies and current studies in order to understand how the situation in the country might be changing. This could be done periodically and it would be beneficial if more countries were to do the same, which would lead to greater collaboration, new information and new participants and contributions. For example, a respondent from the Caribbean subregion said that, since the publication of these reports, a new model had been developed to obtain better information on water, which uses a smaller grid in order to improve resolution. This is particularly important in the Caribbean where some countries are so small that the current grid (25 km) would include large areas of sea. If studies were updated with this new grid, it would be very relevant to see whether the projections would be more accurate. 110. Another possible area of replication that came to light in the interviews, surveys and essentially the project itself was the identification of priorities and sectors in each country, which could be expanded on and examined in greater detail. It is important not only to have a country or regional overview but also to perform a closer inspection of particularly important sectors at a national level, especially as they and the context evolves. For example, in Colombia, a study was undertaken to better understand the impact that climate change and climate variability will have on agricultural production and hence the economic impacts for the country as a whole. Other sectoral studies were undertaken in Argentina, Chile, Uruguay and the Caribbean. These types of studies could also be replicated. 111. With regard to sustainability, the greatest indicator of this would be the fact that work towards identifying and understanding the economic impacts of climate change in the region is still ongoing today, and that studies and other products developed by the project are being utilized. This subsequent work has been supported by ECLAC and other donors but continues as an important aspect of adaptation and mitigation work in the region. While international support demonstrates that this continues to be a priority both regionally and nationally, over time one would expect to see countries and their governments continue to study this area on their own. 112. While this may not be the case in all places, at least a couple of interview respondents and beneficiaries did identify continued efforts that were being made at the country level without substantive support from donors, which bodes well for the sustainability of the project in those countries. For example, the Departamento Nacional de Planeación (DNP) in Colombia recently drafted a plan for the next few years in which one of the objectives was to continue to try to understand the economic impacts of climate change at a national level. It was also mentioned that the efforts of this project had contributed to an ongoing examination in Chile of the economic impacts of climate change through expansion of the analysis to other areas such as the sanitation, mining and power sectors. 34 FINAL ASSESSMENT REPORT 113. In addition, one respondent from Trinidad and Tobago who works in the academic sector said that the results of the studies and general research findings had been incorporated into university courses in the areas of ecotourism and environmental economics. Furthermore, this academic has presented on these research findings at relevant conferences in the country and supported a graduate student in the use of the data and the approach for their Master’s project, hence demonstrating that this research has contributed to further research in a similar area, although on a smaller scale. 114. Overall, the data clearly demonstrate that, despite the fact that this particular project ended four years ago, aspects of the project or elements initiated by it continue today, which is a positive sign of sustainability. This can in part be attributed to the efforts of ECLAC and the Development Account through this project. Many of the project activities can and are being replicated, demonstrating that the project has had a fairly significant impact on understanding of the economy of climate change in the region, which can be built on even further into the future. 35 FINAL ASSESSMENT REPORT 4. LESSONS LEARNT AND BEST PRACTICES 115. What has been achieved by the project provides for some lessons learnt and best practices that could be given consideration in subsequent and similar projects implemented in the area of climate change in the region and potentially elsewhere. Some of the key lessons learnt and best practices include the following: • For many countries, the studies produced by this project were the first efforts in the region to achieve a better understanding and assessment of the economic impacts of climate change. These efforts have therefore been noted to be extremely relevant for the region and for the participating countries in terms of providing a basic reference document that can serve as a starting point for further studies of this nature. Furthermore, the data indicated that in some cases these studies have been used to assist in planning at a national level to ensure that the future economic impacts of climate change are taken into consideration by governments. Given this conclusion, it is evident that supporting multidisciplinary studies in the area of climate change in the region is critical, as they are considered relevant and contribute to national planning in the areas of mitigation and adaptation to climate change. • Overall, a number of beneficiaries felt that technical assistance as a project activity was extremely useful, efficient and relevant as it provided a more direct one-on-one approach to capacity-building and knowledge transfer. A lesson learnt is therefore that technical assistance can improve the chances for success of such projects as it is appreciated and considered useful by participants. • The data suggested that, in terms of the project’s efficiency, organizational and management procedures and processes were sufficiently implemented and the key mechanisms that were created and implemented to support project activities were effective in improving the efficiency and quality of the expected outcomes. An additional lesson learnt is that the use of key mechanisms such as the International Advisory Panel, National Advisory Panels and Technical Teams in the management of such projects can help to enhance overall efficiency. • It was generally noted that the timeliness and dissemination of key documents produced by the project could have been improved and that more publicity would have helped to increase the accessibility of the information found in the reports. Given that EA 1 of the project aimed at increasing the awareness of key stakeholders in the region of the economic impact of climate change, a key lesson from the project is that wide dissemination of key products of the project can help to further awareness and enhance knowledge. • It was felt by some that greater collaboration with key stakeholders, such as the Focal Points to UNFCCC and key Caribbean partner institutions, would have boosted the project’s relevance in the region. The lesson learnt or best practice from this conclusion is that ensuring the involvement of key regional knowledge organizations as early as the project design stage can help to ensure that expertise is tapped into and key stakeholders who can further the project’s agenda are involved early on in a meaningful way. 36 FINAL ASSESSMENT REPORT 5. CONCLUSIONS 116. In spite of various initiatives, Latin American and Caribbean countries continue to experience challenges in their endeavours to meet the Millennium Development Goals (MDGs). Additionally, the impending economic impacts of global climate change on developing countries, and in particular on small-island developing economies, are cause for concern. In this regard, the importance of data in planning and adapting to the expected economic impacts of climate change has been recognized. As impacts evolve, it is of critical importance to develop data-collecting activities and systems, with a view to ensuring that adequate sources of information are available to inform development planning. 117. This project aimed to build on past and current initiatives directed towards estimating the cost of adapting to and mitigating climate change based on climate change scenarios for the next 20-50 years or more, and to identify and assess the expected economic impacts that may result from the projections. The objective of the project was to ensure that key decision-making stakeholders in Latin America and the Caribbean are aware of the projected impacts on their economies and are therefore empowered to take collective action in planning for these impacts. This assessment evaluated the four dimensions of relevance (effectiveness, efficiency and sustainability) in order to evaluate whether the project was relevant to the countries of the region, effective in attaining its objectives, implemented efficiently, and sustainable in terms of continued efforts in this area and capacity-building. RELEVANCE 118. Given that climate change is a very real and prominent threat, and one whose impact is already starting to be felt by countries, it is not surprising that this project would be considered extremely relevant to the Latin American and Caribbean region. Overall, the data that were collected from this assessment suggested that the project and its subject matter were relevant for all participating countries and stakeholders. In fact, it was felt that the studies produced by this project on the economic impacts of climate change in specific Latin American and Caribbean countries were pioneering, given that they were some of the first of their kind to be undertaken in the region. Although greater coordination in terms of disseminating the results and engaging certain regional climate change actors could have potentially improved the project’s relevance, the subject matter, design and purpose of this project were undoubtedly relevant to the region and beyond. EFFECTIVENESS 119. One of the main objectives of the project was to increase the awareness of government officials and other stakeholders regarding costs and other issues that may be predicted to arise due to climate change. Generally, the data that were collected for this assessment suggest that the awareness and knowledge of beneficiaries and other stakeholders did improve due to the activities and efforts of this project (i.e. through the studies and other publications, meetings and conferences as well as the technical assistance that was provided). In terms of effectiveness, this is probably one of the areas in which the project was the most effective. Given that this area of study is relatively new to the region and certainly new to a number of the participating countries, it is evident that there is now greater knowledge and awareness of this very important issue. Furthermore, the reports that have been produced will serve as reference documents for future initiatives seeking to 37 FINAL ASSESSMENT REPORT better understand the realities of these countries as well as the methodologies that can be used to undertake these types of measurements. Once again, further distribution and dissemination of the results of these studies could have also improved awareness and new ways of sharing information should be considered for future initiatives. 120. In addition to increasing awareness, the project also aimed to strengthen national capacities among government officials and other key stakeholders. It was generally found that the project activities had contributed somewhat to strengthening national capacity to identify solutions for addressing the economic impacts of climate change, and that some governments had made efforts to incorporate the results of the studies into national plans, policies and documentation. This demonstrates that not only have beneficiaries become better informed regarding the issues, allowing them to use that knowledge to identify interventions and methods for alleviating those impacts, but in some cases they are also using the results and findings from these studies to influence national policy. While this may not be the case in every country, it is a significant accomplishment for some. Nevertheless, some of the data did indicate that one of the challenges that the project faced was a lack of capacity and knowledge at the country level to undertake the studies due to a shortage of qualified specialists. While this is something beyond the control of the project, and efforts were made to provide capacity-building wherever possible, it should be considered in future and similar studies. 121. As for strengthened regional cooperation, this was accomplished to a limited extent through a number of regional events which were supported by the project and which gave project stakeholders the opportunity to collaborate. The most significant output of the project in terms of supporting regional cooperation was the regional document “La economía del cambio climático en América Latina y el Caribe, Sintesis.” This document provided an overview of the economic impacts of climate change in the region, and was noted to be the first regional attempt to document this issue. Unfortunately, the expected subregional policy for climate change in the Caribbean was not produced; however, efforts were made to identify and document some elements of national policies that could potentially contribute to a subregional policy at a later stage. 122. With regard to gender mainstreaming, it is evident that women did participate in the project and were somewhat represented throughout the activities. Nevertheless additional efforts to demonstrate that gender mainstreaming was a conscious strategy throughout the life of the project could have been beneficial and should be considered for future initiatives by, for example, reporting on gender in the project reports. 123. In general, the project was fairly effective in achieving its expected results, as also demonstrated by the project’s logical framework indicators, and it should be commended as not only one of the first efforts in the region to address and understand the economic impacts of climate change but also as a small initiative that achieved a considerable amount despite a short time frame and limited resources. EFFICIENCY 124. Overall, and based on the data collected, it is evident that this rather small project contributed to a number of important meetings, events, studies and opportunities for countries to work together, and generally used funds as planned. This cumulatively demonstrates that the resources provided were utilized in an efficient way. Furthermore, a number of the implementation processes and procedures followed to ensure project 38 FINAL ASSESSMENT REPORT efficiency were appreciated and successful. The data suggest that stakeholders were satisfied with ECLAC support and the activities implemented. Internal coordination, between the two regional offices, was also fairly good in terms of collaboration on revising and refining the methodologies used for the studies. Generally, and based on the sheer amount of activities and outputs, it is evident that the project was efficient in its use of resources and effective in terms of being implemented within a short period of time. SUSTAINABILITY AND REPLICATION 125. The data suggest that the project activities are highly replicable and have in fact been replicated over the years since this project came to an end. The studies could not only be replicated in other Latin American countries but also in other parts of the world. For countries that have successfully employed the methodologies, there is also the possibility of updating these studies in the future on the basis of more current data, which would lend itself to the sustainability of this initiative. Overall, the data clearly demonstrate that, despite the fact that this particular project ended four years ago, aspects of the project or elements initiated by it continue today, which is a positive sign of sustainability. This can in part be attributed to the efforts of ECLAC and the Development Account through this project. Many of the project activities can and are being replicated, demonstrating that the project has had a fairly significant impact on understanding of the economy of climate change in the region, which can be built on even further into the future. OVERALL CONCLUSION 126. The ability to estimate the cost that climate change will have on a country or region is fundamental, not only for planning and adaptation purposes, but also for mitigation. Understanding that in some cases the cost of avoiding the effects of climate change might be a fraction of what it would cost to tackle the causes of this climatic phenomenon is a first step towards the action that is needed regionally as well as globally. The purpose of these studies is to offer evidence for decision-making so that countries, governments and all relevant climate change stakeholders have the information needed to demonstrate that the disastrous effects of future climate change will not only be detrimental to individuals, groups and natural systems but will also bring a significant economic cost that has yet to be fully considered. The project contributed key studies, information and capacity-building to this area of research, providing some of the basis that is needed to begin these discussions. It can therefore be regarded as an important contribution to climate change research and knowledge and a fairly significant step towards understanding the economic impacts of climate change in the region. 39 FINAL ASSESSMENT REPORT 6. KEY RECOMMENDATIONS 127. The assessment’s recommendations are based on the key findings detailed in the report and are as follows: RECOMMENDATION 1 128. Key finding 3.1.3a: While the logical framework for the project has served its purpose and was up to the required standard, additional efforts could have been made to align the results-based management used by the project with international best practices in this area by, for example, using neutral indicators, baselines and targets, including output level results, and formulating a more precise articulation of expected results and relevant indicators that reflect said results. 129. Recommendation 1: Robust results-based management is essential for demonstrating what has been achieved and for facilitating ongoing reporting and information sharing among stakeholders. It ensures that realistic development results are transparent and accountable and support ongoing learning and adjustments to improve interventions. In the context of future projects, the logical framework should be expanded upon and improved in order to ensure accurate monitoring, evaluation and reporting throughout the project life cycle. 130. Implementing recommendation 1: This is a recommendation for ECLAC, as project implementer, to be put into practice in collaboration with other key stakeholders. In order to improve the use of results-based management in future projects, this recommendation could be implemented by refining the project indicators and results during the formulation phase and by ensuring that the indicators are neutral and reflect the expected results and a target and baseline are identified. In addition, after the formulation phase, a further expansion of the logical framework into a full performance measurement framework would greatly help facilitate monitoring and reporting, despite the fact that this may not be required by the Development Account or ECLAC. Implementing a more robust results-based management system would have to be taken into consideration during all phases of project implementation and key stakeholders would need to be engaged. RECOMMENDATION 2 131. Key finding 3.2.1c: According to data collected from key stakeholders, greater coordination with and the participation of key actors and institutions could, in some cases, have increased the relevance of project activities. 132. Recommendation 2: As mentioned previously, certain stakeholders at the regional and global level who were extremely relevant to the implementation of this project were not sufficiently included. Key stakeholders in the area of climate change are essential to the verity and coordination of these studies and for this reason, future and similar initiatives should make an effort to ensure that key stakeholders in the area of climate change in the region (such as CCCCC and UNFCCC) are involved in all stages of implementation. 40 FINAL ASSESSMENT REPORT 133. Implementing recommendation 2: This would be a recommendation to be implemented by ECLAC, as the implementer of the project. In order to take this recommendation into consideration, a more consistent and thorough involvement of key regional stakeholders in the area of climate change would be needed. This could entail periodic meetings/phone conferences to ensure consistent engagement with the specified stakeholders, soliciting their feedback on material produced, and ensuring that their knowledge, advice and expertise is incorporated into all activities. It would also help to avoid the duplication of efforts in potential research areas and facilitate knowledge-sharing. RECOMMENDATION 3 134. Key finding 3.2.1d: Some respondents felt that better dissemination of the studies would also have contributed to greater relevance of the project in the region. 135. Recommendation 3: This assessment has concluded that further dissemination and distribution of the studies’ results would have increased awareness among key stakeholders in the region and would have improved the project’s overall outcomes and results. Reports, studies and other products should be promptly and widely distributed with a view to enhancing the overall relevance of the programmes and increasing general awareness of the issue. 136. Implementing recommendation 3: Implementation of this recommendation would be the responsibility of ECLAC and all partner institutions. It could be achieved by identifying key audiences and user needs and utilizing a variety of dissemination methods such as community, university or private meetings and presentations, online outlets (web sites, blogs, e-mail networks), the press (newspapers, radio, television), professional and academic conferences and one-on-one meetings with government officials. Although the press was used to disseminate the results, the inclusion of other methods to help reach a greater audience, the allocation of additional resources to results dissemination and the development of a dissemination plan would all assist in the implementation of this recommendation. RECOMMENDATION 4 137. Key finding 3.3.3c: According to survey data, a significant challenge of the project was its interdisciplinary and multi-sectoral nature, as this required coordination between various different sectors and ministries. 138. Recommendation 4: According to the data, one of the biggest challenges in terms of effectiveness, and which contributed to complications with regard to developing the studies, was their multidisciplinary and intersectoral nature. Combining the thematic areas of the environment and economics to assess future potential impacts of climate change was relatively new in the region and created some difficulties in terms of coordination and understanding among stakeholders, and in particular between ministries at the national level. Given this conclusion, additional efforts should be made to ensure smooth coordination between multiple sectors in these types of participatory/collaborative initiatives, which require the involvement of many stakeholders. 41 FINAL ASSESSMENT REPORT 139. Implementing recommendation 4: The implementation of this recommendation would be the responsibility of ECLAC. It should be mentioned that, despite the various efforts made to ensure coordination (such as the creation of National Advisory Panels and the assignment of a lead or co-lead ministry to coordinate and centralize information), it was still felt by a large number of respondents that the biggest challenge in undertaking a study of this nature was the fact that it spanned disciplines. To ensure even greater coordination in this area, it may be useful to allow for additional time so that teams from different sectors (economics versus climate change for example) have the opportunity to develop a common language and knowledge for discussing the issues. Furthermore, additional opportunities could be provided for broadening the training of sector specialists so that they are proficient in the sector that is not their area of expertise. RECOMMENDATION 5 140. Key finding 3.3.5b: A key challenge in implementing the methodologies that was identified by respondents and during the document review was the lack of data in some countries for climate modelling. In addition, future initiatives should provide countries with further support in order to collect the required data for climate models as well as with regard to developing and implementing the methodologies. 141. Recommendation 5: It is important that the country-level data needed for implementing the project methodologies be available and that, above and beyond building capacity to use the methodologies, further support is offered to stakeholders in terms of actually implementing the climate scenarios. As such, future and similar initiatives should take into account in the project approach the time and effort needed to collect the required data before conducting the studies and further assistance should be provided with regard to facilitate the implementation of the recommended methodologies. 142. Implementing recommendation 5: This is a recommendation for ECLAC, as the implementers of the project. In order to ensure that countries have the data they need to undertake the studies, time and effort would need to be allocated to this endeavour. It would need to be factored into the project timing and activities, as a first and fundamental phase, or alternatively a separate project could be run before undertaking the studies in order to ensure that each country has the data required to run the climate models. Furthermore, this data would need to be updated over time, meaning that it is also fundamental to ensure that the country has the capacity to do so. This recommendation can be justified by the fact that running climate models with poor or inadequate data would produce skewed results that will not benefit the country or future decision-making in any significant way. RECOMMENDATION 6 143. Key finding 3.5.1a: Overall, the data collected demonstrate that the project activities are highly replicable and have in fact been replicated over the years since this project came to an end. In addition, the studies that have been completed should be periodically updated to ensure that changing conditions and contexts are taken into consideration. 42 FINAL ASSESSMENT REPORT 144. Recommendation 6: With regard to replication and sustainability of the project, these studies examined the economic impacts of climate change in Latin America and the Caribbean and are the first efforts to assess these effects in the region. There are therefore a number of areas and countries where this work could be easily replicated. Not only could these studies be replicated in other countries of the region and/or sectors that have yet to have undertaken this research, but existing studies should be maintained and updated periodically to advance and strengthen the evaluation and formulation of specific actions for mitigating the economic impacts identified, and to ensure that changing conditions and contexts are reflected, thereby enhancing and sustaining the relevance and usefulness of the studies produced. 145. Implementing recommendation 6: This recommendation should be implemented by the country partners and those who have taken ownership of the studies that have been developed. In the absence of such ownership and the capacity to update studies into the future, ECLAC could develop an additional project and/or fund to maintain the information produced by the studies. This would entail updating the data collected at the country level so that it can be fed into the climate models and comparisons can be made from year to year. 43 ANNEXES FINAL ASSESSMENT REPORT ANNEX 1 Evaluation Design Matrix ANNEX 2 Interview Protocol ANNEX 3 Assessment Questions ANNEX 4 Simplified Logical Framework ANNEX 5 List of Stakeholders Interviewed ANNEX 6 Bibliography ANNEX 7 Stakeholder Map ANNEX 8 Evaluator’s revision matrix 45 FINAL ASSESSMENT REPORT ANNEX 1 Evaluation Design Matrix Questions and sub-questions Indicators Data Collection Methods Information and Data Sources RELEVANCE: The extent to which the project and its activities are suited to the priorities and policies of the region and countries at the time of formulation and to what extent they are linked or related to the ECLAC mandate and programme of work. • To what extent was the project and its activities suited to the priorities and policies of the region and countries at the time of formulation? • Quality of pre-design economic impact needs assessment • Degree of involvement of national climate change experts, economists and consultants in the design phase of the project. • Level of consistency between objectives, activities and outputs and the attainment of the expected accomplishments (EAs). • Document review • Electronic survey • Interviews • Project Document • Annual Progress Reports • Final Project reports • High Level Meeting Reports • ECLAC Project Managers • Beneficiaries o Did the objectives analysis demonstrate the logic and plausibility of the means- end relationship? • Correlation between problem analysis and country problem conditions • Correlation between project objectives and accomplishments with countries’ needs and priorities. • Document • Project Document review • Annual Progress Reports • Electronic survey • Final Project reports • Interviews • ECLAC Project Managers o To what extent were Project activities related to the ECLAC mandate and sub-programmes? • Correlation of activities and outputs with ECLAC mandate • Extent of inclusivity of civil society, youths, and ethnic minorities • Document review • Electronic survey • Interviews • ToR (consultancies) • ECLAC Project Managers • Meeting reports • Level of satisfaction with ECLAC representatives 46 FINAL ASSESSMENT REPORT EFFECTIVENESS: the extent to which the activities attain its objective and EAs. • Did the project contribute to strengthened national capacities to identify responses to the potential economic impacts of climate change? (EA 2) • Number of professionals in government ministries and departments in the region who report applying acquired skills from training to assess physical and economic impacts of climate change. • Document review • Electronic survey • Interviews • Number of reports/studies and policies integrating formal economic models and methods into climate change impact analyses. • Evidence of a change in the behavior/ attitude/ skills/ performance of local resources • Did the project contribute to strengthening regional cooperation to address the economic impacts of climate change? (EA 3) • Number of high-level meetings held for sharing or integrating economic methodologies and models with other countries. • Extent to which Sub-regional policy framework to address climate change has been produced and endorsed by Governments in the Caribbean. • Document review • Electronic survey • Interviews • Did the project contribute to increased awareness in Latin America and Caribbean of the economic impacts of climate change, as well as adaptation and mitigation options?(EA 1) • Level of satisfaction of beneficiaries with the • Number of professionals in the region taking up training to assess physical and economic impacts of climate change. • Number of events aimed at sensitizing communities to economic aspects of DRR and climate change; with emphasis on marginalized groups. • Perception of the quality and timeliness of the project outputs and services • Document review • Electronic survey • Interviews • Business communities • Beneficiaries • ECLAC Project Managers • International and regional partners • High Level/other Meeting Reports • National and Valuation Reports • Advocacy groups/materials on economic impacts of climate change • Beneficiaries • ECLAC Project Managers • International and regional partners • High Level/other Meeting Reports • National Reports • Beneficiaries • International and regional partners • ECLAC Project Managers • High Level/other Meeting Reports • National Reports 47 FINAL ASSESSMENT REPORT • To what extent did the project apply a gender mainstreaming and human rights-based approach to • Extent to which there is a planned or developed policy for integrating human rights, gender mainstreaming and RBM • Document review • Electronic survey • Interviews programming? within government departments/ministries. • Number of reports and publication materials demonstrating gender mainstreaming and participation of civil society, youths and ethnic minorities EFFICIENCY: measurement of the outputs (qualitative and quantitative) in relation to the inputs. • Did organizational factors contribute to effective implementation of the project and support partner coordination? • Evidence of narrative or organogram of governance and management structures for Project • Evidence of an implementation arrangement/ management plan • Document review • Electronic survey • Interviews incorporating partnership as a modality for coordination with universities, the CCCCC etc. • Extent of clarity in definition of roles and responsibilities of cooperating agency and partners with regard to ECLAC’s procedures and reporting requirements. • Extent to which there were processes and procedures in place to support timely implementation of EAs. • Extent to which there was a planned or developed RBM policy for the project. • Project Document • Annual Progress Reports • Final Project reports • High Level Meeting Reports • ECLAC Project Managers • Project Document • Annual Progress Reports • Final Project reports • High Level Meeting Reports • ECLAC Project Managers • International and regional partners 48 FINAL ASSESSMENT REPORT • Were there • Evidence of joint programming with complementarities or other development partners, CSOs, synergies with others private sector, research centres etc. working in the same thematic • Evidence of public and/or private area of economic impacts sector support resulting from • Document review • High Level Meeting Reports • Electronic survey • ECLAC Project Managers • Interviews • International and regional partners from climate change? the project. • Evidence of active involvement civil society, youths and ethnic minorities • Evidence of internal coordination within ECLAC between subprogrammes and offices • Level of satisfaction of ECLAC with internal coordination for project implementation • Have the invested resources produced the planned EAs? • Timeliness of resources allocation for national capacity strengthening, awareness building activities • Implementation delays due to lack of resource allocation timeliness • Document review • ECLAC Project Managers • Electronic survey • Project Document • Interviews • Annual Progress Reports • Final Project reports • Responses and actions taken to expedite processes • Adequacy of implementation resources to avoid cost overruns SUSTAINABILITY: the extent to which the benefits of the project are likely to continue after funding has been withdrawn. • To what extent did the design of the project take into account the issue of sustainability? • Evidence of an Exit Strategy or plan addressing various risks to sustaining Project benefits. • Level of satisfaction of beneficiaries with their involvement in the implementation of the project. • Document review • ECLAC Project Managers • Electronic survey • High Level Meeting Reports • Interviews • Project Document • Annual Progress Reports • Final Project reports • Beneficiaries • To what extent did beneficiary countries implement measures to enhance sustainability of EAs of the project? • Number of country measures and/or policies that address systemic risks (financial, legal, management, institutional) that can impact sustainability of Project EAs. • Document review • Final Project reports • Electronic surveys • High Level Meeting Reports • Interviews • ECLAC Project Manager/ representative • International and regional partners • Beneficiaries 49 FINAL ASSESSMENT REPORT • Was the project sustainable in terms of financial resources to carry on since project closure? • To what extent was Project hand-over to beneficiary country Governments preceded by a capacity development programme? • Was there the potential for scaling-up of the project implemented with DA resources? • Extent to which project design and implementation factored in strengthening local ownership and commitment among key stakeholders • Evidence of improved and harmonized ability of the LAC countries to adapt and cope with the economic impacts of climate change on development • Resources committed by ECLAC, beneficiary countries and other partners to perpetuate Project benefits. • Quality of partnerships with new donors or partners to improve after-project financial capacity • Evidence of an Exit Strategy or similar that contains plans for strengthening capacity (local, national levels) to perpetuate Project benefits • Perception of an enabling environment by government officials to carry on after the project ends. • Scaling or replication plans • Budget for scaling out to other locations • Quality of consultations with civil society or other groups in targeted geographies. • Document review • Final Project reports • Electronic surveys • High Level Meeting Reports • Interviews • ECLAC Project Managers • International and regional partners • Beneficiaries • Document review • ECLAC Project Managers • Electronic surveys • International and regional • Interviews partners • Beneficiaries • Document review • International and regional • Electronic surveys partners • Interviews • ECLAC Project Managers • Meeting Reports • Beneficiaries 50 ANNEX 2 Interview Protocol FINAL ASSESSMENT REPORT ECLAC Project Manager Beneficiaries Partners from International and Regional Organization Questions To what extent the project and its activities are suited to the priorities and policies of the region and countries at the time of formulation and to what extent they are linked or related to the ECLAC mandate and programme of work? 1. Did the project design properly address the issues identified in the region? a. Did the project design include both gender and human rights considerations from the onset? 2. Were the objective and accomplishments relevant to the     countries’ development needs and priorities? 3. To what extent do you consider the activities in which you  participated were relevant to your country context? 4. Were the objective and accomplishments aligned with  ECLAC’s mandate and the relevant sub-programmes? To what extent did the project activities attain their objectives and expected results? 5. Did the project contribute to increased awareness about the   economic impacts of climate change, as well as adaptation and mitigation options? 6. Did the activities in which you participated provide you with  increased awareness about the economic impacts of climate change, as well as adaptation and mitigation options? 7. To the best of your knowledge:  Did the project contribute to strengthen national capacities to identify responses to the potential economic impacts of climate change? 8. Do you feel that the activities in which you participated  provided you with increased capacity to identify responses to the potential economic impacts of climate change? 9. To the best of your knowledge:  Did the project contribute to strengthen regional cooperation to address the economic impacts of climate change? 10. Do you feel that participation in the activities increased  regional cooperation for addressing the economic impacts of climate change? 51 FINAL ASSESSMENT REPORT 11. To what extent were you satisfied with the quality and  timeliness of the activities and publications produced in collaboration with ECLAC and other partners? 12. To the best of your knowledge:  What factors have contributed to achieving or not achieving the project’s intended outcomes and results? 13. To the best of your knowledge:  What factors have contributed to or impeded the achievement  of key results? (Publications, studies, events, meetings, regional coordination etc.)? 14. Was a human rights-based approach understood and  pursued throughout the project? a. Were gender considerations mainstreamed throughout the implementation of the project? Measurement of the outputs (qualitative and quantitative) in relation to the inputs 15. In your opinion, did the governance and management  structures contribute to effective implementation? a. Were roles and responsibilities clearly established at the beginning of the project? b. Were any processes or procedures established to improve implementation? c. Was there a Results-based Logical Framework? d. Did the project use results-based monitoring and reporting? 16. Was the project successful in creating a continuous capacity  strengthening process, jointly with regional/country authorities, over the lifetime of the project? 17. Through the activities in which you were involved do you  feel that a continuous capacity strengthening process was created? 18. Did Project procedures contribute to or jeopardize the  effective implementation of the project? 19. Were the needed resources available in a timely manner  and utilized as planned? 20. Were project activities delivered in a timely manner and  outcomes achieved on time? 21. How effective was the project at bringing partners together?  a. Which partners did the project bring together? b. To your knowledge, did this project develop any complementarities or synergies with other work that was being developed (i.e. other environmental financial management work, sustainable development work, regional work etc.)? c. To what extent was civil society, youths and ethnic minorities involved in the project?       52 FINAL ASSESSMENT REPORT 22. How satisfied were you with the coordination between the  different sub-programmes and offices within ECLAC? a. How could this coordination have been improved? b. Did coordination within ECLAC contribute to the achievement of the project outcomes? Why or why not? The extent to which the benefits of the project are likely to continue after funding has been withdrawn. 23. To what extent has the design of the project taken into  account the issue of sustainability? 24. Were you involved in more than one activity? (yes or no)  a. If so, then do you feel that a continuous capacity strengthening process, jointly with country authorities was provided by ECLAC? 25. What were the factors that enhanced or undermined  long-term continuity of the outcomes achieved or likely to be achieved? a. Is the project sustainable in terms of financial resources to carry on since Project closure? b. To what extent is Project hand-over to beneficiary country Governments preceded by a capacity development programme? 26. Was there the potential for scaling-up of the project  in other locations or regions? 27. In your opinion, do any of the activities in which you have  participated demonstrate potential for replication and scale- up of successful practices? a. What do you see as being some of the key elements that could be replicated or expanded upon? b. Do you have any recommendations for future activities?     53 FINAL ASSESSMENT REPORT ANNEX 3 Assessment Questions Assessment Question: Assessment Sub-Question: Assessment Question: Assessment Sub-Questions: 1. EVALUATION CRITERION: RELEVANCE R1) To what extent the project and its activities are suited to the priorities and policies of the region and countries at the time of formulation? To what extent they are linked or related to the ECLAC mandate and programme of work? R1.1) Were governance and management structures of the project effectively established? R1.2) Were the objectives and accomplishments relevant to the countries’ development needs and priorities? R 1.3) Did the objectives and accomplishments remain relevant throughout the implementation? R1.4) Were the objective and accomplishments aligned with ECLAC’s mandate and the relevant sub-programmes? R1.5) Did the design properly address the issues identified in the region? R1.6) Were these structures appropriate to the objective, accomplishments and activities? R1.7) Were the activities and outputs consistent with the objective and the attainment of the EAs? R1.8) Did the problem analysis define the initial situation with sufficient precision? R1.9) Did the problem analysis define the major problem conditions with sufficient precision? R1.10) Did the problem analysis identify realistic cause-effect relationships among problem conditions? R.11) Did the objectives analysis demonstrate the logic and plausibility of the means- end relationship? 2. EVALUATION CRITERION: EFFECTIVENESS E1) To what extent did activities attain their objectives and EAs? E1.1) To what extent did the project achieve the EAs outlined in the Project Document? E1.2) Did the project contribute to increase the awareness of LAC Governments, business communities and civil society about the economic impacts of climate change, as well as adaptation and mitigation options? E1.3) Did the project contribute to strengthening national capacities to identify responses to the potential economic impacts of climate change? E1.4) Did the project contribute to strengthening regional cooperation to address the economic impacts of climate change? E1.5) To what extent are the project’s main beneficiaries satisfied with the quality and timeliness of the outputs and services? 54 FINAL ASSESSMENT REPORT Assessment Question: Assessment Sub-Questions: Assessment Question: E1.6) What factors have contributed to achieving or not achieving the intended outcomes? E1.7) What factors contributed to effectiveness or ineffectiveness? E1.8) To what degree were approaches such as a human rights based approach to programming, gender mainstreaming and results-based management understood and pursued in a coherent fashion? E1.9) Has the project made any difference in the behavior/ attitude/ skills/ performance of the clients? E1.10) How effective were the project activities in enabling capacities and influencing policy-making? E1.11) Are there any tangible policies that have considered the contributions provided by the project in relation to the project under evaluation? 3. EVALUATION CRITERION: EFFICIENCY EF1) Was Project coordination and implementation timely, cost-effective and supportive to continuous capacity strengthening? EF1.1) Did governance and management structures of the project contribute to effective implementation and coordination of partners? EF1.2) Was the project successful in creating a continuous capacity strengthening process, jointly with regional/country authorities, over the lifetime of the project? EF1.3) Did project procedures contribute or jeopardize the effective implementation of the project? EF1.4) Which partners did the project bring together? EF1.5) Have the invested resources produced the planned outcomes? EF1.6) Were the needed resources available in a timely manner and utilized as planned? EF1.7) Were the project EAs/outcomes achieved on time? EF1.8) Was the project implemented in the most efficient way compared to alternatives? EF1.9) Were there any complementarities and synergies with the other work being developed? 4. EVALUATION CRITERION: SUSTAINABILITY S1) What is the extent to which the benefits of the project are likely to continue after funding has been withdrawn? S2) Are the project outcomes expected to have a lasting impact on beneficiaries’ access to knowledge and technical capacity in the medium- to long term? S3) To what extent has the project contributed (or will it contribute) to improve and harmonize the ability of LAC countries to adapt and cope with the economic impacts of climate change on development? S4) Has follow-up support after the end of the activities been discussed and formalized? S5) Does the project demonstrate potential for replication and scaleup of successful practices? 55 FINAL ASSESSMENT REPORT ANNEX 4 Simplified Logical Framework Intervention Logic Indicators Means of Verification Risks and Assumptions Objective: The overall objective of this project is to improve the ability of Latin American and Caribbean countries to adapt to, and cope with, the economic impacts of climate change on development. Expected accomplishment 1 At least 20 news items are The results of the meetings It is assumed that governments Increased awareness in Latin published/broadcast in would inform preparation of would support this initiative American and Caribbean mass media discussing publication materials which through the national and Governments, business communities contributions of the would be obtained from reports regional information services. and civil society of the economic project. of the meetings. Preparation of The challenge here is in impacts of climate change, as well as At the end of the project, publication materials would be reaching rural communities adaptation and mitigation options at least 25 professionals done by meeting facilitators in through the normal media from different collaboration with ECLAC channels. This however could be governmental and non- sustainable development and addressed through the governmental institutions public relations focal points. The convening of meetings in these of at least 6 countries publications would be in the areas at convenient times. have been trained in the form of news briefs and would methodologies to assess focus on different aspects of the physical and economical topics that were discussed at the impacts of climate change. meetings. A 1.1 One workshop on climate change modeling. A 1.2 One workshop on economic valuation of impacts of climate change 56 FINAL ASSESSMENT REPORT Intervention Logic Indicators Means of Verification Risks and Assumptions Expected accomplishment 2 In the Caribbean, at least The documents on climatic It is assumed that countries are 2 countries prepare modeling and potential committed to addressing climate Strengthened national capacities to national reports on economic impacts of climate change on a sustained basis. It identify responses potential impacts impacts and policy change prepared by the is expected that countries would change of to the economic climate options, as inputs for the consultants would be used as begin the national reports but High-Level meetings of the reference material for completion would be a project. In Latin America, preparation of national reports. challenge. Follow-up activities at least 2 countries Government environmental and to the project and collaboration prepare national reports planning and policy focal points with the stakeholders, such as on the economic impacts would lead this process with CCCCC should address this. and decide to adopt a support from ECLAC. ECLAC policy framework in at would co-opt the required least one of the identified expertise to vet the reports in vectors of impact. order to promote the highest standards. 2.1 One study that will assess the potential physical impact of climate change. This would involve a modeling exercise for the Caribbean Sea; 2.2 One study on methodologies which allow to identify and compare the potential production changes due to temperature and water stress on tourism, agriculture and forestry, cattle, mining, urban productivity and health analyzing anticipated effects on regional international trade, supply and prices, fiscal income, fiscal expenditure and investment at the national level in at least 3 Latin American countries. 2.3 One valuation study to estimate the economic impacts of potential changes on variables such as production and trade, prices and income in the Caribbean. The study will also identify policy options to address climate change (adaptation and mitigation). Particular attention will be given to Caribbean SIDS as part of the MSI strategy; 2.4 One valuation study to estimate the potential integrated response measures for water management in rural and urban areas, fiscal income stability, gradual responses in public expenditure, price fluctuations mitigation, and in general an economic stabilization policy related to climate change in Latin America. The study will also identify policy options to address climate change (adaptation and mitigation). 57 FINAL ASSESSMENT REPORT Intervention Logic Expected accomplishment 3 Strengthened regional cooperation to address the economic impacts of climate change. Indicators A sub-regional policy framework to address climate change is produced and endorsed by Governments in the Caribbean. Means of Verification Reports of all high level meetings as well as consultants reports of studies will build awareness among senior policy makers thereby obtaining their commitment to formulation of a regional policy on climate change. Of particular importance here is the realization by policy makers of the value of such a policy to addressing the impacts of climate change on the economies of these countries. This would be verified by presentation of data obtained from the project meetings at these high level meetings at which senior policy makers would be present. Following on this, is obtaining written commitment of policy makers to formulation of a regional climate change policy and the appointment of a team to oversee its preparation and finalization. ECLAC would lead this process in collaboration with governmental disaster
environmental focal points in the region. Risks and Assumptions The assumption here is that countries are committed to development of a regional climate change policy. The risk is lack of agreement on the elements of such a policy. This could be addressed through the hosting of focus group meetings to address any concerns. 58 FINAL ASSESSMENT REPORT A 3.1 One expert meeting will be held, for the Caribbean Sea area where consultants, government officials and academics assess findings of the above studies, perform comparative analysis of impact of climate change and deliberate on adaptive policy responses. The result will be a proposed policy framework that identifies elements for coordination of policy responses to address the impact of climate change that incorporates elements of adaptation and mitigation. A3.2 One expert meeting will be held, for the entire LAC region, where consultants, government officials and academics assess findings of the above studies, perform comparative analysis of methodologies on modeling physical and economic impacts of climate change and discuss on policy responses. The result will be a proposed policy framework that identifies elements for coordination of policy responses to address the impact of climate change that incorporates elements of adaptation and mitigation. A3.3 One high-level meeting will be organized on the impact of climate change on the Caribbean Sea. A3.4 One high-level meeting will be organized on the economic impacts of climate change, in the entire LAC region. 59 FINAL ASSESSMENT REPORT ANNEX 5 List of Stakeholders Interviewed Nombre Carlos de Miguel Willard Philips Elizabeth Thorne Karina Martinez Sebastián Vicuña Silvia Calderón Luis Santos Mark Bynoe Ruben Mamani-Paco Sharon Hutchinson Cargo Oficial de Asuntos Ambientales Economic Affairs Officer Coordinador Nacional RECCS Subdirección de Desarrollo Ambiental Sostenible Dirección Nacional de Medio Ambiente Equipo ERECC Caribe Director de Operaciones Lecturer Department of Agricultural Economics and Extension País POS POS Santiago Chile Colombia Uruguay Bolivia Trinidad and Tobago Institución CEPAL CEPAL CEPAL CEPAL Centro Interdisciplinario de Cambio Global UC DNP Ministerio de Ambiente CCCCC Fundación Ingenieros en Accion University of West Indies Mail Carlos.demiguel.cepal.org Willard.philips@eclac.org Elizabeth.thorne@eclac.org Karina.martinez@cepal.org svicuna@gmail.com ; svicuna@uc.cl Silvia.lcalderon@gmail.com ; scalderon@dnp.gov.co lsantos@cambioclimatico.gub.uy m_bynoe@yahoo.co.uk Ruben775@yahoo.com Sharon.hutchinson@sta.uwi.edu 60 FINAL ASSESSMENT REPORT ANNEX 6 Bibliography Evaluation Templates • Expected Accomplishment (EA) 1 - (ECLAC, 2014) • Expected Accomplishment (EA) 2 - (ECLAC, 2014) • Expected Accomplishment (EA) 3 - (ECLAC, 2014) Reports • Annual Development Account Progress Report (ECLAC, 2009-2010) • Development Account Terminal Report (ECLAC, 2011) • An Assessment of the Economic Impact of Climate Change on the Macroeconomy in the Caribbean (2011) • Análisis de la vulnerabilidad del sector hidroeléctrico frente a escenarios futuros de cambio climático en Chile (2012) • Climate change and the macro-economy in the Caribbean basin: analysis and projections to 2099 (2010) • Desarrollo de una función agroclimática para estimar la productividad de los cultivos agrícolas en Colombia (2013) • Disponibilidad futura de los recursos hídricos frente a escenarios de cambio climático en Chile (2012) • Efecto del cambio climático en el costo de los eventos de interés en salud pública en Colombia: estudio de caso sobre malaria y dengue (2013) • Escenarios hidrológicos de caudales medios del río Paraná y Uruguay (2013) • Evaluación de impactos de cambio climático sobre la producción agrícola en la Argentina (2013) • Evaluación de los impactos del cambio climático sobre el ecosistema natural y la biodiversidad, Argentina (2013) • Expert Group Meeting on Macro-econometric Modelling in the Caribbean: New challenges and new direction (2009) • La economía del cambio climático en América Latina y el Caribe, Síntesis (2009) • La economía del cambio climático en América Latina y el Caribe, Síntesis (2010) • La economía del cambio climático en Chile. Síntesis (2009) • La economía del cambio climático en Chile (2012) • La economía del cambio climático en el Ecuador (2012) • La economía del cambio climático en el Paraguay (2014) • La economía del cambio climático en el Uruguay, Síntesis (2010) • La economía del cambio climático en la Argentina, primera aproximación (2014) • Panorama del cambio climático en Colombia (2013) • Regional Climate Modeling in the Caribbean (2010) • Regional Report on the Impact of Climate Change on the Tourism Sector (2010) 61 FINAL ASSESSMENT REPORT • Report of the fourth expert group meeting of the project: Understanding the potential economic impact of climate change in Latin America and the Caribbean: Methodological issues in modelling the macroeconomic impacts of climate change in the Caribbean (2011) • Report of the meeting of the high-level advisory committee (HLAC) of the project: a review of the economics of climate change (RECC) (2009) • Report of the Roundtable: Towards Development of a Climate Change Policy for the Caribbean: Understanding the Potential Economic Impacts of Climate Change in Latin America and the Caribbean (2010) • Report of the third expert group meeting of the project understanding the potential economic impact of climate change in Latin America and the Caribbean (2011) • Report of the twenty-third session of the Caribbean development and cooperation committee (2010) • Review of the Economics of Climate Change in the Caribbean Phase II – The Water Sector (2010) Other • Allotment Advice (2009) • Complementary Material for Project Evaluation (Updated up to 25th November 2014) • Notes from Meetings, Conferences and Presentations (2008-2014, Annexes 1-12 and Fonoconferencias) • Project Document • Publications (2009-2014) • Contracts (28) 62 ANNEX 7 Stakeholder Map FINAL ASSESSMENT REPORT HIGH IMPORTANCE How influential How affected Beneficiaries: 1. Government officials 2. Universities/Academics 3. Civil society (women, girls, minorities, youths) 4. Private Sector ECLAC Project Managers LOW Other regional and international partners: - CARICOM Secretariat/CCCCC - BID -UKAID -CBD LOW SUPPORT HIGH In summary, categories of respondents include members of the following stakeholder groups:  ECLAC’s Project Managers;  Beneficiaries (Policy makers, National Energy Agencies, universities, research centers and the CCCCC).  Regional and international partner organizations 63 FINAL ASSESSMENT REPORT ANNEX 8 Evaluator’s revision matrix A. COMMENTS ERG SUSTAINABLE DEVELOPMENT AND HUMAN SETTLEMENTS DIVISION (SDHSD) GENERAL COMMENTS REPORT SECTION (if applicable) COMMENTS ERG EVALUATOR’S RESPONSE It should be further empathized the additional funding provided by other donors, which was extended beyond the timing of the evaluated project. This has been included in paragraph 102. Also it is recommended to mention the limited amount of resources invested by the project in terms of the results obtained for the entire region. Also it is crucial to indicate the project’s distribution of resources, which were divided by the offices of Port of Spain and Santiago. This is mentioned in paragraph 95. 2.3 Limitations It is suggested to further explain the context of the project and the timing of evaluation. Therefore, it has influenced on the surveys as many of the stakeholders have been removed from their original positions, or even they do not remember all the details of the project. This is included in the limitations section (2.3). This project started in 2008 and the evaluation was done until 2015. After 7 years and several changes of government officials and even ECLAC staff, it is difficult for all the interviewees to describe or determine the relevance of many activities that occurred at the beginning of the project. 3.2 Relevance This section must underlines that a huge effort was done by coordinating and involving most of the key stakeholders in the countries. Furthermore, in the South American countries the national governments suggested to ECLAC the focal points for the National Panel. Efforts in coordination have been included in paragraph 84. 64 FINAL ASSESSMENT REPORT SPECIFIC COMMENTS PARAGRAPH NUMBER COMMENTS ERG EVALUATOR’S RESPONSE Section 3.1.1 par 3 Section 3.1.2 par 5 Section 3.1.3 par 1 Section 3.1.3 par 4 The problem tree was a “tailor made methodology” for each national study, according to their needs, conditions and availability of information. The relationships among the cause and effects of climate change that the consultant mentions as “not clear” are fully described into the national studies. In the report, it is referring specifically to the problem tree that is found in original project document. The point being made is that in the original project document this could have been further elaborated on or more clear. It is not referring to the national studies, which came afterwards. As such, this has been elaborated to be clearer in the corresponding section. Several stakeholders were consulted during the process of This has been revised. project’s elaboration, as ECLAC has a close relationship with the environment and finance’s ministers of the region. The project document includes all the information and details requested for the logical framework’s section, so it was not “understated” as it is mentioned in the evaluation. This has been clarified. We were not saying that it was done wrong, only providing suggestions of how it could be improved throughout implementation. The indicator of news items has been significant since most of them underlined the work of the ministries on climate change activities. This results most be contextualized as in 2009 there were not many news items pointing to the involvement of the governments of the region on climate change activities, particularly the ministries of Finance. The indicator suggested by the consultant “Number/type of Governments, businesses, communities and civil society reporting that they are more knowledgeable/aware about the economic impacts of climate change in LAC”, may be more adequate but is not feasible to demonstrate. This indicator has been removed. 65 FINAL ASSESSMENT REPORT SPECIFIC COMMENTS PARAGRAPH NUMBER COMMENTS ERG EVALUATOR’S RESPONSE Section 3.1.3 par 6 Section 3.2.1 par 5 Section 3.3.1 par 3 Section 3.3.2 par 9 The logical framework was “updated and revised” along the project implementation, as the countries and their situation were changing. Another reason to complement this context is the participation of other sponsors who contributed to the initiative RECCS and their schedule of participation. Particularly, the study of Bolivia was completed many years after the project ROA 137 was completed. This result was reported as it was part of the RECCS initiative. However, it must be explained that the publication and dissemination activities depended on the reception of the funds from the IADB, which was the main donor of this activity. Most of the countries continue disseminating the study on their climate change web pages, as part of the reference documents on this arena. It is crucial to state that the news items reported were collected specifically for the timing of the project’s report. Many of these items were related to the launching events. However, from time to time, ECLAC staff has identified that the studies are mentioned in news related to climate change activities of the countries. Additionally, during activities such as seminars or workshops in Santiago or abroad, the SDHSD promotes the results of the studies among many different groups to ensure they are widely disseminated. These activities involve awareness of the results for further beneficiaries and stakeholders. The strategies suggested by the consultant such as assigning a lead or co-lead ministry were carried-on during the project. This was the role of the National Advisory Panel. ECLAC requested to the governments of the South American countries appointing at least 2 focal points from the ministries of Environment and Economy/Finance. These ministries centralized the information and invited/coordinated the required activities with other ministries. Reference to not having updated and revised the logical framework has been removed. A footnote was added. This has been added in this section. The strategies suggested have been revised. 66 FINAL ASSESSMENT REPORT SPECIFIC COMMENTS PARAGRAPH NUMBER COMMENTS ERG EVALUATOR’S RESPONSE Section 3.3.2 par 10 Section 3.3.3 par 3 Section 3.3.3 par 5 Section 3.3.3 par 6 Section 3.3.4 par 5 Section 3.4.1 par 9 The recommendation of working “more closely with universities and researchers from universities” was applied during the project. As it was mentioned by the respondent, the majority of the consultants were academics in almost all countries. The involvement of the government officials was crucial for their appropriation of the study as it was one of the objectives of the project. This recommendation has been removed/revised. It must be clarified that the objective of the sub-regional policy was only considered by the Caribbean region, not for the South American countries. This has been clarified. The first regional document was “La economía del cambio climático en América Latina y el Caribe, Sintesis 2009”. This document included preliminary information collected by the study teams and by ECLAC. Consequently, the following publication in 2010 enhanced the results with further analysis and information. This has been revised. It is important to mention in some part of the report that also in ECLAC has been changes of staff during the life of the project. Noted in Limitations section (2.3). The two negative responses from ECLAC about promotion on regional cooperation might be due to lack of information from colleagues who do not identify all the activities implemented since the beginning of the project between the consultants and government officials from the participant different countries and ECLAC officials. Also it must be pointed that the project promoted the jointly work among offices (Port of Spain and Santiago), during and after the project. The positive responses from the beneficiaries reinforce the result that the project strengthened the regional cooperation. Indeed, at least for the South American countries the methodologies were “tailored” according to the needs and information available of each country. This has been clarified in section 3.3.5. One of the main considerations for the South American studies was the ownerships of them from the government officials who have been the main beneficiaries. Several examples of this ownership were included in the complementary material for the project evaluation. This has been noted in section 3.5.1. 67 FINAL ASSESSMENT REPORT SPECIFIC COMMENTS PARAGRAPH NUMBER COMMENTS ERG EVALUATOR’S RESPONSE Section 4.1 par 4 Section 4.2 par 2 These studies have been used by the national governments of Chile and Uruguay to continue with similar activities, the IPCC authors have used them as inputs for the assessment reports and also have been used by the sectoral experts as academic material. This conclusion must be delimited and expanded with further details and examples. Only one respondent mentioned in section 3.3.1 that “…they felt that the consultants who were involved were not overly aware of the particular impacts that climate change has on the Caribbean region”. This was removed. All the consultants hired (national and international) who developed the studies were knowledgeable about the effects of climate change in their particular country or region. The national consultants were experts with broad background on their respective areas, and the international consultants belong to the group of IPCC experts. As the project report indicated it was not difficult to identify consultants who knew the effect of climate change on their particular countries. The difficulties to identify experts were related with the expertise and sufficient knowledge on the application of methodologies to evaluate the cost of the impacts of climate change. These methodologies were applied for the first time into the region and there are only few international references. A needs assessment document was the first task for each of the national coordinators of the South American studies, in order to identify the specific needs of each country. Based on this assessment each national coordinator identified the three requirements: capacity building, data collection and the tailored methodology. This recommendation with regards to a needs assessment was removed. 68 FINAL ASSESSMENT REPORT GENERAL COMMENTS REPORT SECTION (if applicable) COMMENTS ERG EVALUATOR’S RESPONSE Section 4.2 par 8 As it was mentioned on the comments for Section 3.3.2 par 9, the strategies suggested by the consultant such as assigning a lead or co-lead ministry was carried-on. This was the role National Advisory Panel. These were revised. The difficulties in terms of coordination among stakeholders and ministries were variable between countries. ECLAC suggested to the governments a scheme of collaboration among ministries but each country had their own dynamics of work. As it is mention on the project report, there were many government changes, not only focal points but creation of new ministries which replaced the previous ones, for example in Bolivia. SUBREGIONAL HEADQUARTERS FOR THE CARIBBEAN (POS) GENERAL COMMENTS REPORT SECTION (if applicable) COMMENTS ERG EVALUATOR’S RESPONSE The overall evaluation represents a fair assessment of the elaboration and implementation of the project in the SROCaribbean. It also provides a good learning perspective on the actual achievement of the project on the ground. Thank you! SPECIFIC COMMENTS PARAGRAPH NUMBER COMMENTS ERG EVALUATOR’S RESPONSE 1-4 Page 15 Specifically, I do concur with the observations made in respect of the Logical Framework…., Noted. Page 16 and the recommendations for improvement of the LF for future projects. Noted. 69 FINAL ASSESSMENT REPORT B. COMMENTS PPOD GENERAL COMMENTS REPORT SECTION COMMENTS PPOD (if applicable) Favor incluir un resumen ejecutivo de 2-4 páginas al inicio del informe. Favor asegurarse que tanto en los hallazgos como en las conclusiones y/o recomendaciones (si es aplicable) se haga mención específica al cumplimiento o no de la transversalización de género en el proyecto bajo evaluación. Favor numerar cada párrafo del informe. En general, se solicita este formato en todo informe de evaluación para facilitar la lectura y los comentarios vinculándolos al número de párrafo. Falta incluir una sección de conclusiones, después de la sección de hallazgos y la de lecciones aprendidas. Estas deben dar respuesta a las “grandes preguntas evaluativas” basándose en un análisis general por parte del evaluador de los principales hallazgos. En general, por la forma en que fue redactado el informe, da la impresión de que muchas de los hallazgos presentados se basan en una o dos respuestas de parte de los entrevistados, lo cual pareciera ser un poco riesgoso al presentar hallazgos sin el suficiente respaldo o evidencia. Esto especialmente se da en aquellas acotaciones que hacen referencia a “one respondent stated….””more tan one respondent”, etc. De lo cual se encuentran varios ejemplos en el informe. Se les agradecería revisar y sustentar apropiadamente o dejar en claro que si bien no son hallazgos concluyentes o afirmaciones generalizadas si hubiera ciertos comentarios interesantes de algunos entrevistados que se recomienda sean tomados en cuenta. Recomendamos homologar todas las gráficas con los resultados de las encuestas utilizando porcentajes y no número de respuestas como sucede en varios casos. 3.3 Effectiveness Recomendamos incluir en esta sección un análisis de los resultados del proyecto, no solamente en relación a los resultados esperados (EAs), pero si no también a los indicadores establecidos para medir dichos resultados esperados. 3.4 Efficiency Hace falta incluir un análisis y los principales hallazgo relacionados a los niveles de cooperación y coordinación de la DDSAH y la oficina de POS, a lo interno de la CEPAL con otras Divisiones de la CEPAL y a lo externo con otras instituciones, en la implementación del proyecto. EVALUATOR’S RESPONSE This has been included. A section on gender has been added to the report (Section 3.3.6). Each paragraph has been numbered. A conclusions section has been added (Section 5). Any findings that were based on too few respondents have been modified to be represented as just interesting commentary that could be noted. In other places, where multiple respondents had similar feedback, this has been revised to properly represent this number. Overall, some findings were either removed or slightly modified to ensure that they are supported by sufficient evidence. Percentages have been added to all graphics. This has been included (section 3.3.1). A short analysis has been included to section 3.4.2. 70 FINAL ASSESSMENT REPORT 4.2 Key Recommendations Favor asegurarse de que en la presentación de cada recomendación se haga referencia específica al hallazgo y conclusión con que se vincula, para lo cual sería conveniente identificar cada hallazgo específico con un número, así como asignar un número a cada recomendación. Each finding has been numbered and the recommendations at the end of the document are linked to this numbering. 4.2 Key Recommendations Favor explicitar a quien está dirigida cada recomendación o quien estaría a cargo de su implementación. La DPPO, CEPAL en general DESA como Project manager del Development Account para temas más transversales o la División sustantiva a cargo del proyecto o tema. In the recommendations section, each recommendation has a paragraph entitled “implementing recommendation No.”. This provides an overview of who would be responsible for implementation of each recommendation. 4.2 Key Recommendations Se recomienda explicitar un poco sobre como el consultor considera pudiesen ser implementadas las recomendaciones incluidas en el informe. No debe ser más de uno o dos párrafos por recomendación. In the recommendations section, each recommendation has a paragraph entitled “implementing recommendation No.”. This also provides an overview of how the recommendation could be implemented. 4.2 Key Recommendations Llama la atención el hecho que no se haya incluido ninguna recomendación relacionada con el proceso de diseño del programa y su marco lógico, siendo que se mencionan muchas deficiencias del mismo en la sección de hallazgos. This has been added based on the findings in the report (Recommendation 1). SPECIFIC COMMENTS REPORT SECTION COMMENTS PPOD (if applicable) EVALUATOR’S RESPONSE Página 6. Sección 1.3 Objective and scope of the evaluation Favor mencionar de manera explícita, el propósito y objetivo de la evaluación, y cuál fue el alcance y objeto de la evaluación detallando el proyecto evaluado. Incluir asimismo el período en el cual se desarrolló la evaluación. This has been added to section 1.3. Página 7. Sección 2.1 Assessment Strategy Página 9. Tabla 1. Favor incluir una descripción de cómo se incorporaron las perspectiva de género y derechos humanos en el proceso de evaluación. Agradeceríamos incluir dos filas al final de cada sección (entrevistas y encuestas) detallando los porcentajes de respuesta. This has been integrated throguhout section 2.2. We were unable to do this. The reason for this is that each section in the report for project 137 is split up by EAs and includes a number of different graphs and questions that were asked during interviews and surveys to which a different number of persons have responded in each case. Given 71 FINAL ASSESSMENT REPORT Anexo 5 Página 10. Section 2.3 Limitations Párrafo 3. Collection of Data Página 18. Sección 3.2.2 Links to ECLAC Mandates Página 27. Tabla 2 Página 33. Sección 4.2 Key Recommendations 1.III that there are more than one question included in each section, to which a varying number of answers were provided, it is impossible for us to provide a summary of percentages of answers per section. Each graph now has the percentage as well as the number of respondents, since each graph corresponds to a question that was often answered by a different number of people during the surveys or interview. That is where you can see the number of respondents for each stakeholder group as well as the percentage for each answer, per stakeholder group. Favor actualizar la información presentada en el anexo 5, This has been updated. ya que todavía incluye solamente el listado de las 8 personas originalmente entrevistadas. Consideramos que se debería de hacer mención al hecho de que el tiempo pasado entre la implementación del proyecto y el momento de la evaluación, pudo haber sido un factor importante que influyó en el bajo nivel de respuesta de las encuestas. This was included in the Limitations Section (2.3). Favor incluir los objetivos y resultados esperados de la División de Desarrollo Sostenible (subprograma 8) y de la Oficina de POS (Subprograma 13) establecidos en el Programa de Trabajo de la CEPAL (documento oficial) como marco de referencia al momento de analizar qué tan vinculado esta el proyecto con el programa de trabajo de la División, y no solamente hacer referencia a la información incluida en su página web. This has been added to section 3.2.2. Por favor buscar otra manera de presentar esta tabla. Como esta presentada actualmente con número de respondientes sin compararlos con el total, no permite realizar un verdadero análisis de los resultados o su significado. The percentage of respondents has been added to the table to provide an overview of total respondents. Nos gustaría que se analizara un poco mas esta recomendación tomando en consideración que si bien es cierto, es importante poder la adaptar las metodologías a las realidades y contextos en cada país, también es importante mantener una comparabilidad entre países. Aspecto de gran relevancia para instituciones como CEPAL cuyo mandato es más bien a nivel regional. This recommendation was modified and revised and some of what was found there can be seen in recommendation 5. 72