1995 es Corden, W. Max Corden, W. Max Una zona de libre comercio en el Hemisferio Occidental: posibles implicancias para América Latina En: La liberalización del comercio en el Hemisferio Occidental - Washington, DC : BID/CEPAL, 1995 - p. 13-40 2014-01-02T14:51:16Z hdl:11362/40162 FINAL ASSESSMENT REPORT Assessment of Development Account Project 08/09 Z Strengthening the capacity of National Statistical Offices (NSOs) in the Caribbean small island developing States to fulfil the Millennium Development Goals (MDGs) and other Internationally Agreed Development Goals (IADGs) February 2016 This report was prepared by Nicole Hazel, an external consultant, who led the evaluation. Ms. Hazel worked under the overall guidance of Raúl García-Buchaca, Chief of the Programme Planning and Operations Division of the Economic Commission for Latin America and the Caribbean (ECLAC), and Sandra Manuelito, Officer in Charge of the Programme Planning and Evaluation Unit, and under the direct supervision of Irene Barquero, Programme Officer of the same unit, who provided strategic and technical guidance, coordination, and methodological and logistical support. The evaluation also benefited from the assistance of María Victoria Labra, Programme Assistant, and Leslie Cuellar, Unit Intern, also of the Programme Planning and Evaluation Unit. The evaluation team is grateful for the support provided by its project partners at ECLAC, all of whom were represented in the Evaluation Reference Group (ERG). Warm thanks go to the programme managers of the ECLAC subregional headquarters for the Caribbean in Port of Spain, including Johann Braithwaite, Associate Programme Officer, Sheila Stuart, Social Affairs Officer, and Dale Alexander, Computer Information Systems Assistant. All comments on the evaluation report by the Evaluation Reference Group and the evaluation team of the Programme Planning and Evaluation Unit were considered by the evaluator and duly addressed in the final text of the report, where appropriate. The views expressed in this report are those of the author and do not necessarily reflect the views of the Commission. The annexes to this evaluation report have been reproduced without formal editing. Copyright © United Nations, February 2016. All rights reserved Printed at United Nations, Santiago S.16-00177 FINAL ASSESSMENT REPORT TABLE OF CONTENTS ACRONYMS................................................................................................................................................... v EXECUTIVE SUMMARY..................................................................................................................................vii 1. INTRODUCTION ........................................................................................................................................ 1 1.1. Statement of the problem and project objectives...........................................................................................1 1.2. Assessment objectives and scope........................................................................................................................2 1.2.1. Assessment objectives....................................................................................................................................2 1.2.2. Assessment scope ...........................................................................................................................................2 2. ASSESSMENT FRAMEWORK AND METHODOLOGY................................................................................. 3 2.1. Assessment framework..........................................................................................................................................3 2.2. Assessment methodology......................................................................................................................................3 2.3. Triangulation and data validation .....................................................................................................................3 2.4. Implementation of the assessment ......................................................................................................................6 2.4.1. Inception phase ..............................................................................................................................................6 2.4.2. Sample population and selection criteria.................................................................................................6 2.4.3. Data collection tools......................................................................................................................................7 2.4.4. Data collection................................................................................................................................................8 2.4.5. Strategies to increase stakeholder participation ....................................................................................8 2.4.6. Response rates ...............................................................................................................................................8 2.4.7. Brief profile of respondents ........................................................................................................................9 2.4.8. Data analysis..................................................................................................................................................9 2.5. Limitations............................................................................................................................................................. 10 2.6. Assessment principles and ethical issues......................................................................................................... 10 3. ASSESSMENT FINDINGS.......................................................................................................................... 11 3.1. Assessment findings by criterion ...................................................................................................................... 11 3.1.1. Project beneficiaries................................................................................................................................... 11 3.1.2. Relevance of the project ........................................................................................................................... 11 3.1.3. Relevance of the activities and the outputs........................................................................................... 13 3.1.4. Effectiveness of the project....................................................................................................................... 18 3.1.5. Efficiency of the project............................................................................................................................. 25 3.1.6. Cross-cutting themes................................................................................................................................... 27 3.1.7. The sustainability of the project and/or its interventions ................................................................... 28 4. GENERAL CONCLUSIONS AND RECOMMENDATIONS .......................................................................... 29 4.1. Relevance ............................................................................................................................................................. 29 4.2. Effectiveness......................................................................................................................................................... 29 4.3. Efficiency .............................................................................................................................................................. 30 4.4. Sustainability ....................................................................................................................................................... 31 4.5. Cross-cutting themes........................................................................................................................................... 31 5. KEY LESSONS LEARNED .......................................................................................................................... 32 iii FINAL ASSESSMENT REPORT ANNEXES ..................................................................................................................................................... 33 ANNEX 1 Data Collection Tools ................................................................................................................... 35 ANNEX 2 List of Key Informants .................................................................................................................. 61 ANNEX 3 Assessment Terms of Reference.................................................................................................... 62 ANNEX 4 Evaluator’s revision matrix ........................................................................................................... 76 List of tables Table 1: Assessment questions by method and data source......................................................................................4 Table 2: Proposed sample population by data collection method ..........................................................................7 Table 3: Survey responses by target group .................................................................................................................8 Table 4: Interview response rates by target group ....................................................................................................9 Table 5: National and regional workshops................................................................................................................ 15 List of figures Figure 1: The perceptions of national representatives regarding the relevance of the project ..................... 11 Figure 2: The contributions of development agencies to the project .................................................................... 12 Figure 3: Partner perspectives of the work with ECLAC.......................................................................................... 13 Figure 4: Recipients of ECLAC support........................................................................................................................ 14 Figure 5: Perceptions of the relevance of the training to national priorities....................................................... 16 Figure 6: The role of development partners and experts....................................................................................... 16 Figure 7: Relevance of the Caribbean MDG report to national interests ........................................................... 17 Figure 8: The satisfaction of national officials with the meetings and workshops.............................................. 18 Figure 9: National officials and the perceived benefits of the expert group meetings and/or training workshops .......................................................................................................................... 19 Figure 10: Reported use of knowledge and skills in the last six months .............................................................. 20 Figure 11: Levels of satisfaction with the assistance provided by ECLAC ........................................................... 20 Figure 12: Opinions of national officials regarding the quality of the MDG report ........................................ 21 Figure 13: Satisfaction with ECLAC support in the preparation of the Caribbean MDG report.................... 21 Figure 14: Participation in the ECLAC project and improvements in data collection, analysis and use at the country level ...................................................................................................... 22 Figure 15: Opinions on improvements in reporting on the MDGs and other IADGs.......................................... 23 Figure 16: The continuity of improvements over the long term .............................................................................. 23 Figure 17: Collaboration between ECLAC Port of Spain and headquarters...................................................... 26 iv ACRONYMS FINAL ASSESSMENT REPORT CARICOM CBD ECLAC IADGs KIs KIIs MDGs NSOs OECS SDGs ToR UN UNDP UNEP UNFPA UNICEF UNSD Caribbean Community Caribbean Development Bank Economic Commission for Latin America and the Caribbean (United Nations) Internationally agreed development goals Key informants Key informant interviews Millennium Development Goals National statistical offices Organization of Eastern Caribbean States Sustainable Development Goals Terms of reference United Nations United Nations Development Programme United Nations Environment Programme United Nations Population Fund United Nations Children’s Fund United Nations Statistics Division v FINAL ASSESSMENT REPORT EXECUTIVE SUMMARY The following report outlines the results of the assessment of Development Account project ROA 122 entitled “Strengthening the capacity of National Statistical Offices (NSOs) in the Caribbean small island developing States to fulfil the Millennium Development Goals (MDGs) and other Internationally Agreed Development Goals (IADGs)”. This three and a half year project was implemented by the United Nations Economic Commission for Latin America (ECLAC) subregional office in Port of Spain, with the objective of improving the capabilities of the staff of 15 national statistical offices to improve the quality of reporting on the MDGs and other IADGs.1 The project’s activities therefore focused on capacity-building through national and regional workshops, the provision of technical assistance to member and associate States by ECLAC staff and partners, and the publication of the first regional report on progress towards the achievement of the MDGs. The data collection phase of the assessment ran from 14 September 2015 to 12 October 2015 and the criteria of relevance, efficiency, effectiveness and sustainability served to provide the overarching framework for this work. In keeping with the United Nations emphasis on the inclusion of considerations to gender and human rights, where possible these principles were part of the assessment’s design and execution. Furthermore, the assessment also examined the extent to which these concepts formed part of the Development Account project. The assessment targeted (a) representatives of the 21 project countries, (b) current and former ECLAC staff members/consultants, and (c) development partners/experts. To this end, 242 surveys were sent to the three groups and 15 national representatives and five ECLAC staff members were contacted about participating in an interview. It was noted that, since four years had passed since the end of the project, not only was a low response rate a possibility but respondents may not be able to recall the events in detail. Strategies such as notification by ECLAC, tailoring the design of the instruments and periodic reminders were utilized to increase participation rates. Seventy one stakeholders took part in the assessment, by either completing a survey and/or sharing their perspectives in a semi-structured interview. Consequently, the findings outlined in this report represent the opinions of 45 females and 26 males. The ability of some countries in the Caribbean to produce timely, high-quality data for decision-making purposes is hindered by staffing challenges, specifically the availability of adequate numbers of trained personnel, limitations in budgetary allocations to departments with mandates specific to data, and in some cases a lack of recognition of the importance of data and its possible uses. ECLAC therefore sought to address some of these concerns through this Development Account project. Generally speaking, the project was responsive to the situation in the countries, but an analysis by component reveals that activities such as training and technical assistance were seen as more useful than activities such as the MDG publication. In addition, while short-term improvements were reported both for individuals and for countries, the majority of participants were unable to provide insights into the multiplier effects of the project. 1 Fifteen countries were mentioned in the project document but the assessment’s findings indicate that an additional six participated in the project. vii FINAL ASSESSMENT REPORT In terms of its efficiency, the project was affected by challenges relating to oversight and staff turnover, which contributed to deviations from the original project design. Some of these include increasing the number of countries from 15 to 21 and emphasizing capacity-building to the detriment of other activities. Owing to the lack of an effective system for results-based monitoring and evaluation, changes were recorded but there was no mechanism for taking corrective action. Consideration to sustainability appeared to be an oversight by ECLAC and the target countries. The Development Account project was time-bound and participants in the assessment could not provide any information on plans that ECLAC or the respective governments had made to continue the activities either in whole or part. Therefore, the assessment highlighted that, while there may have been gains at the individual level, there was not enough information on activity maintenance and the long-term impact of the project in the countries. The recommendations highlight the importance of paying attention to: x Meaningful involvement of national representatives during the design of projects x Results-based monitoring and evaluation x Project-steering committees and consistent project management x Planning for the continuation of activities by ECLAC and Caribbean governments. viii FINAL ASSESSMENT REPORT 1. INTRODUCTION 1.1. Statement of the problem and project objectives 1. In the Caribbean region, the ability of governments to collect data for decision-making purposes is hampered by inadequate staff levels in national statistical offices (NSOs) and related units, a lack of technical capacity, insufficient financial resources and a lack of recognition of the importance of data for policy and programme initiatives.2 2. To this end, the Economic Commission for Latin America and the Caribbean (ECLAC) subregional headquarters for the Caribbean implemented a Development Account project from 1 July 2008 to 31 December 2011 to address the challenges faced by 15 national statistical offices in the subregion.3 The main focus of the project was to strengthen the capacity of national statistical offices by improving the staff’s technical capabilities, carrying out activities to support the collection, analysis and dissemination of data on the Millennium Development Goals (MDGs) and other internationally agreed development goals (IADGs), and fostering opportunities for collaboration between governments and organizations operating at the regional level. 3. Other direct and indirect beneficiaries included the end users of data such as the technical staff, planners and policymakers in government departments and ministries, development agencies with related mandates, academic institutions and the private sector. 4. The following activities were completed as part of the project: (a) Technical assistance to member States with regard to the collection, analysis and use of data (b) Capacity-building through an expert group comprising partner agencies as well as national and regional training workshops (c) The publication of a report on attainment of the MDGs in the Caribbean 5. It was also anticipated that information systems for data management and the sharing of best practices among stakeholders would be part of the project. However, while preparatory activities such as the development of a database and the procurement of a server were undertaken, this component was not completed due to changes in staffing and differing perceptions of the priorities of the project. 6. The following outcomes were anticipated by the end of the project:4 (a) “Adoption by Caribbean countries of common data collection protocols, definitions and classifications for the collection and dissemination of an extended set of MDG indicators that are more suitable to their situation, in order to monitor the accomplishment of the MDGs. (b) Strengthened technical capacity of governmental institutions to produce and analyse MDG indicators derived from harmonized household survey data sets and use them in the design and implementation of public policies that lead to the fulfilment of the goals. 2 Source: Development Account project document, January 2008. 3 The countries included in the initial project design were Antigua and Barbuda, Bahamas, Barbados, Belize, Dominica, Dominican Republic, Grenada, Guyana, Haiti, Jamaica, Saint Kitts and Nevis, Saint Lucia, Saint Vincent and the Grenadines, Suriname and Trinidad and Tobago. However, an additional six countries – Anguilla, Aruba, British Virgin Islands, Cayman Islands, Montserrat and the Turks and Caicos Islands–participated in the project. (Sources: the Development Account project document, 2008 and the Development Account project terminal report). 4 As per the terms of reference. 1 FINAL ASSESSMENT REPORT (c) Improved capacity in the NSOs of participating countries to collect, compile, analyse and disseminate data on social and environmental indicators to measure poverty, social cohesion and environmental sustainability. (d) Establishment of a database which makes it possible to draw up indicators at the subregional level in order to measure the economic, social and environmental costs of maintaining current development patterns; evaluate and diagnose the current state of the environment in the region; gauge the environmental consequences of existing macroeconomic policies; and supply orientation for policy decisions, especially in regard to public and private investment.” (P.2) 1.2. Assessment objectives and scope 1.2.1. Assessment objectives 7. The standard criteria of relevance, effectiveness, efficiency and sustainability were utilized to determine the extent to which the project’s objective of improving the capacity of national statistical offices in the Caribbean was achieved. The assessment also outlines the results and impact of the project on its beneficiaries and other stakeholders, and determines whether the activities were implemented as planned. 8. Furthermore, since gender and human rights are key considerations within United Nations programming, the assessment examines the extent to which the cross-cutting themes of gender as it relates to the rights and needs of women and respect for human rights were included in project design and execution. 1.2.2. Assessment scope 9. To review all aspects of the Development Account project and provide ECLAC with an objective review that documents any benefits for the countries and other stakeholders, changes in the approach of the NSOs, the sustainability of the project activities, the success of the collaboration between the regional commissions and other development partners and any key lessons learned that could be incorporated into the design of similar projects. 10. This report outlines the assessment’s findings, conclusions and recommendations and it is accompanied by a PowerPoint presentation that summarizes the main sections of the report. It is suggested that ECLAC use these outputs to design future Development Account projects and activities specific to data for decision-making purposes in the Caribbean subregion. Please refer to annex 3 for the assessment terms of reference. 2 FINAL ASSESSMENT REPORT 2. ASSESSMENT FRAMEWORK AND METHODOLOGY 2.1. Assessment framework 11. The criteria of relevance, effectiveness, efficiency and sustainability served as the overarching framework for the assessment.5 x Relevance refers to whether the project was in keeping with the priorities and plans of the beneficiaries, countries and ECLAC. x Effectiveness determines the extent to which the project was implemented in accordance with the initial objectives. x Efficiency measures the use of resources and the ways in which these contributed to achieving the project’s results. x Sustainability considers whether any positive results were continued on completion of the project. 12. Therefore, the criteria listed above informed the development of the overall assessment questions, the design of the data collection tools, the analysis and interpretation of the findings and the reporting of the final results. 2.2. Assessment methodology 13. A mixed methods approach encompassing qualitative and quantitative techniques was used to ensure assessment rigour and validity. An analysis of select documentation on the MDGs and the Development Account project was undertaken to obtain information on the activities, beneficiaries, outcomes, outputs and achievements. 14. In the second phase of the assessment, Survey Monkey was used to distribute three electronic surveys targeting (1) representatives from the project countries, (2) ECLAC staff and consultants, and (3) development partners and experts. The evaluation data from workshops and meetings were also included where possible. The quantitative data are complemented by key informant interviews (KIIs) with select ECLAC staff members who were directly involved in project execution and representatives from the project countries. 2.3. Triangulation and data validation 15. The inclusion of multiple strategies such as qualitative and quantitative approaches namely electronic surveys and KIIs, document analysis and an internal review by the Evaluation Reference Group facilitated the comparison of data and cross checks against different sources. Data triangulation reduces the likelihood of bias and the findings are more representative of the project outcomes. Therefore, in order to achieve the objective of documenting the results and impact of the project, the assessment questions and methods outlined below in table 1 were utilized. 5 As per the United Nations Development Programme Handbook on Planning, Monitoring and Evaluating for Development Results and the Development Assistance Committee (DAC) Criteria for Evaluating Development Assistance. 3 FINAL ASSESSMENT REPORT Table 1 Assessment questions by method and data source Assessment questions Method and data source Relevance 1. Did the project’s design reflect the needs and priorities of NSOs and governments? ƒ Survey 1, 2 and 3, KIIs with beneficiaries, project document and reports 2. Similarly, were the project’s activities and outputs in line with the priorities of the target countries? ƒ Survey 1, 2 and 3, KIIs with beneficiaries, project document and reports 3. To what extent are the project’s objectives still applicable to the needs of the target countries and by extension ECLAC? 4. Was the project aligned with the ECLAC work programme in the region? ƒ Survey 1, 2 and 3, KIIs with ECLAC staff and beneficiaries, project document and reports ƒ Survey 2, KIIs with ECLAC staff, project document and reports 5. Were there any complementarities and synergies with other work (that is, development partners and national governments) at the country level? ƒ Survey 1 and 2, KIIs with ECLAC staff, project document and reports Effectiveness 1. To what extent were the project’s objectives achieved? ƒ Survey 1, 2 and 3, KIIs with ECLAC staff and beneficiaries, project document and reports 2. Were the beneficiaries satisfied with the activities? ƒ Survey 1, KIIs with beneficiaries and project reports 3. What are the main results as identified by the beneficiaries? 4. Has the project contributed to any improvements in the technical competencies of the officers of the NSOs/other government departments? 5. Have the participants reported increases in skills and knowledge due to their involvement in the workshops? If yes, did the beneficiaries report the application of the skills and knowledge gained in the workshops? 6. Did the project facilitate improvements in reporting on social, economic and environmental indicators (specifically the MDGs and other IADGs) at the national and or regional levels? 7. Has the project resulted in any national policies geared towards the collection, analysis and use of data derived from social, economic and environmental indicators? ƒ Survey 1, KIIs with beneficiaries and project reports ƒ Survey 1, 2 and 3, KIIs with ECLAC staff and beneficiaries, project document and reports ƒ Survey 1, KIIs with beneficiaries, project document and reports ƒ Survey 1, 2 and 3, KIIs with ECLAC staff and beneficiaries, project document and reports ƒ Survey 1, 2 and 3, project document and reports 4 FINAL ASSESSMENT REPORT Assessment questions Method and data source 8. Did the project result in the development of a database of social, economic and environmental indicators either at the national and or regional level? 9. How did the expert group and partner agencies contribute to the design and implementation of the project? That is, did they significantly contribute to the overall results? 10. Were there any outcomes (either positive or negative) as a result of the project? ƒ Survey 1 and 2, project document and reports ƒ Survey 3, KIIs with ECLAC staff, project document and reports ƒ Survey 1, 2 and 3, KIIs with ECLAC staff and beneficiaries, project document and reports. Efficiency 1. Were the project activities completed within the ƒ KIIs with ECLAC staff, project document and reports specified timelines? 2. Were the available resources (specifically technical ƒ KIIs with ECLAC staff, project document and reports and financial) fully utilized by ECLAC? 3. Were the resources (technical and financial) utilised ƒ KIIs with ECLAC staff, project document and reports as planned? 4. Did the project utilize the technical, human and other ƒ KIIs with ECLAC staff, project document and reports resources that were available in the target countries? 5. How efficient was the collaboration between ECLAC ƒ Survey 2, KIIs with ECLAC staff, project document and Santiago and the subregional headquarters in Port reports of Spain? 6. Did ECLAC (Port of Spain and Santiago) provide the ƒ Survey 1, 2 and 3, KIIs with ECLAC staff and necessary support in a timely and reliable manner, beneficiaries, project document and reports as well as in accordance with the priorities established in the project document? 7. Did ECLAC utilize any internal protocols/guidelines in ƒ KIIs with ECLAC staff, project document and reports the design and execution of the project? 8. Were any good practices/lessons learned ƒ KIIs with ECLAC staff, project document and reports documented for inclusion in similar projects? Sustainability With beneficiaries/national representatives: 1. Did any of the national governments develop or implement a plan to ensure the sustainability of activities on completion of the project? 2. Have the project’s main results and recommendations been used or incorporated into the work programmes of the countries? 3. Were any multiplier effects generated by the project? ƒ Survey 1, KIIs with ECLAC staff and beneficiaries, project document and reports ƒ Survey 1, 2 and 3, KIIs with ECLAC staff and beneficiaries, project document and reports ƒ Survey 1, 2 and 3, KIIs with ECLAC staff and beneficiaries, project document and reports 4. To what extent did the outcomes (that is, improvements in capacity/reporting, networks etc.) continue following the project’s conclusion? ƒ Survey 1, 2 and 3, KIIs with ECLAC staff and beneficiaries, project document and reports 5 FINAL ASSESSMENT REPORT Assessment questions Within the regional commissions: Method and data source 1. Did the project design incorporate a sustainability plan? And, if yes, was this plan implemented? 2. To what extent were national stakeholders involved in the design and implementation of the project? 3. Has the project contributed to shaping and/or enhancing ECLAC priorities, work programmes and activities? 4. Has ECLAC built on any of the findings/lessons learned from the project? ƒ KIIs with ECLAC staff, project document and reports ƒ KIIs with ECLAC staff, project document and reports ƒ KIIs with ECLAC staff, project document and reports ƒ KIIs with ECLAC staff, project document and reports Cross-cutting themes 1. How many people (specifically national officers) benefited from the implementation of the project? ƒ Survey 1, KIIs with beneficiaries, project document and reports including participant lists 2. Did the project activities respect and promote the needs, priorities and rights of all beneficiaries, in particular women? ƒ Survey 1 and 2, KIIs with ECLAC staff, project document and reports including participant lists 2.4. Implementation of the assessment 16. The assessment was conducted in four main stages, namely inception, data collection, data analysis and report writing. 2.4.1. Inception phase 17. This phase was conducted from 3 to 28 August 2015 and involved the following activities: (a) The development of the consultancy workplan (b) The literature and document review of publications on the MDGs and the Development Account project (c) The determination of the sample population, strategy and response rates (d) The development of the data collection tools for the surveys and KIIs 2.4.2. Sample population and selection criteria 18. The attendance records from meetings and workshops were used to identify potential participants. Of 251 possible respondents, 242 were identified for the surveys and 20 for the KIIs. The first survey was sent to 197 representatives of member and associate States, the second survey to 14 ECLAC staff members and consultants, and the third survey to 31 development partners and experts. It should be noted that the development partners and experts completed only the survey, while select members of the other two groups completed both surveys and interviews. 19. Five ECLAC staff members were selected as possible interviewees because of their roles as project coordinators and technical experts. The selection criteria further focused on staff who attended three or more workshops and meetings during the project period, since personnel who were directly involved in the project should be better positioned to provide insight into the challenges with which the countries were faced, actions to address any deficiencies, the outcomes and any related outputs. Similarly, 15 representatives from the target countries were identified by project staff as possible key informants. Table 2 outlines the sample population by data collection method. 6 FINAL ASSESSMENT REPORT Table 2 Proposed sample population by data collection method Respondent category and survey number Total populationa Total sample population Surveys Key informant interviews Representatives of member and associate States (survey 1) ECLAC staff and consultants – headquarters and subregional headquarters (survey 2) Development partners and experts (survey 3) Total: 203 19 31 253 197 Yes (197) Yes (15) 19 Yes (14) Yes (5) 31 Yes (31) 247 242 No 20 a The total population was developed based on the participant lists from workshops and meetings. The total sample population therefore represents contacts with known e-mail addresses. 2.4.3. Data collection tools (see annex 1) A. Surveys 20. The surveys collected data pertaining to the following themes: (a) Background/demographic information (e.g. gender, country, occupation, ECLAC office, etc.) (b) Relevance of the project. (c) Relationship of the project to national and regional priorities (i.e. improvements in data quality and use). (d) Participation in the project and satisfaction with the activities. (e) Perceptions of the benefits of the various activities at the individual level and for countries. (f) Application of knowledge and skills. (g) Continued improvements past December 2011. (h) Perceptions of activity coordination. (i) Collaboration between regional commissions and other agencies. B. Key informant interviews (KIIs) 21. KIls were asked interview questions based on the following themes: (a) Relevance of the project to the needs of the beneficiaries as well as to the priorities and programmes of the countries, ECLAC and other development organizations. (b) Participation in project activities and levels of satisfaction. (c) Perceptions of the benefits for stakeholders. (d) The main strengths and weaknesses of the project. (e) The extent to which the main objective was achieved. (f) Factors that contributed to the achievement or non-achievement of the project’s objectives. (g) Outcomes (positive and negative) that impacted the project’s results. (h) The efficiency with which the project objectives were achieved. (i) Key lessons learned and consideration of sustainability issues. (j) The impact of the project on its beneficiaries and other groups. 7 FINAL ASSESSMENT REPORT 2.4.4. Data collection 22. Data collection commenced on 14 September 2015 with the surveys, which were available for completion until 6 October 2015. The interviews were conducted from 25 September to 12 October 2015. Due to respondents’ lack of availability and a slow response rate, the deadlines for completion of the interviews and surveys were extended by six and twelve days, respectively. 2.4.5. Strategies to increase stakeholder participation 23. In an effort to encourage stakeholders to take part in the assessment, the following strategies were employed prior to and during data collection: x Notification by ECLAC: Potential respondents were notified via letter about the assessment and the importance of their support. x Survey design: The design of a survey can influence its completion. Therefore, consideration was given to the number of questions per survey, the length and content of the questions (ensuring that there was variation in the structure of survey questions, e.g. use of scales, rating, open-ended, partially close-ended and close-ended), the order of the questions and even the amount of space provided for open-ended questions. x Development of the interview guide: The inclusion criteria for KIs were also considered when developing the interview guides. As with the survey instruments, the questions were directly linked to the evaluation criteria. They were also structured to enhance the flow of the interview, to facilitate probing on the part of the interviewer and to clarify previous responses. x The provision of contact information: The e-mail addresses of the ECLAC Programme Planning and Evaluation Unit and the Consultant were provided as part of the official communication and at the start of each tool. Potential participants were able to pose queries and request additional information about the assessment. x Distribution of the surveys: The surveys were distributed by ECLAC in an effort to increase recognition of the work. x Periodic reminders: Three reminders were sent via e-mail to encourage respondents to participate in the data collection exercise. 2.4.6. Response rates 24. The rates of completion varied according to the target group and method. Tables 3 and 4 summarize the response rates. Respondent category Representatives of NSOs and other departments/ministries Development partners and experts ECLAC staff and consultants Table 3 Survey responses by target group No. of surveys sent out 197 Returned undeliverable 2 No. of surveys delivered without issue 195 31 8 14 2 23 12 No. of completed surveys 49 6 7 Response rate 25% 26% 58% 8 FINAL ASSESSMENT REPORT Table 4 Interview response rates by target group Respondent category Representatives of NSOs and other departments/ministries ECLAC staff and consultants No. of people contacted 15 5 No. of interviews completed 6 3 Response rates 40% 60% 2.4.7. Brief profile of respondents 25. The following section provides a synopsis of the characteristics of the assessment participants: x Total population and gender: Seventy one people took part in the assessment (45 females and 26 males). x Country of employment at the time of the project: Out of the 55 representatives from the target countries, respondents from Antigua and Barbuda accounted for the majority of respondents with 11, followed by Saint Vincent and the Grenadines with 9, Grenada with 7 and Saint Lucia with 6. x Place of employment at the time of the project: The majority of participants (33) stated that they were employees of government ministries and departments other than NSOs. Personnel from the statistical offices accounted for 22 of the respondents. This finding suggests that while statisticians and related officers were the main target group for the project, their participation was lower than that of other government employees. ECLAC staff members made up the third-largest group with 10 employees – the majority of whom (six) were from the Port of Spain office, followed by three from headquarters and one individual who indicated that they were based in New York. In terms of the development partners/experts, four out of the six respondents indicated that they were affiliated with an international organization. The other two people stated that they worked in the private sector and consulting. 2.4.8. Data analysis 26. The survey data were analysed using the analytical tools in Survey Monkey. Further analysis was undertaken using Microsoft Excel and presented by way of descriptive statistics using graphs and charts as appropriate. However, the data were presented by number of respondents instead of in percentages because of the relatively small sample size. 27. The qualitative data were not transcribed due to time and budgetary constraints. However, the interviewer notes were used to code and analyse the data by theme. 28. Finally, the data were triangulated through multiple sources including the provision of feedback by the ECLAC task manager and the Evaluation Reference Group. 9 FINAL ASSESSMENT REPORT 2.5. Limitations 29. This assessment was conducted almost four years after completion of the Development Account project. As a result, the following factors may have impacted participation rates and the assessment results: (a) General limitations: x Staff turnover affected the availability and willingness of respondents to participate in the assessment. x Similarly, some of the e-mail addresses on file were no longer in use. x The population could have been affected by shifting interest in the MDGs and other IADGs – and the assessment may not have been a priority based on current needs and plans. x The ability of respondents to recall specifics about the project was also affected. In order to assist with recall (while not influencing the respondents), the questions included the time frame of the project, the workshop topics and other themes. However, participants still experienced difficulties in responding to both the survey and interview questions. (b) Remote data collection is more cost effective than in-country work but it loses the immediacy factor. Respondents did not respond quickly to invitations to complete the surveys or to participate in an interview. This slow rate of response affected the overall timeline for the data collection phase. (c) Similarly, data collection using online surveys such as Survey Monkey can add value to work of this nature, but it is widely accepted that low response rates are one of the main limitations of this approach. (d) The surveys were sent out by ECLAC and this assisted with recognition of the exercise by potential participants. However, this raises the issue of limited anonymity, since ECLAC has access to the completed surveys and as a result to the consolidated responses which can be linked to e-mails via the online system. (e) Finally, in keeping with ethical principles, participation was voluntary and some stakeholders refused outright by either not responding to the e-mails or by providing a reason for their unwillingness to take part. 2.6. Assessment principles and ethical issues 30. The assessment was guided by the following principles: x Gender: As previously stated, gender is a key consideration of Development Account projects and ECLAC work programmes.6 Males and females were therefore both included in the full stakeholder list. Also, a question on gender was included to assist with the analysis and reporting of data by sex. x Ethical issues: The assessment was conducted within the parameters of ethical research and evaluation, namely informed consent, voluntary participation, the right of participants to remain anonymous and respect for respondents’ human rights. ƒ Furthermore, the work was guided by the United Nations Evaluation Group principles for evaluations, which include independence, impartiality, disclosure of conflicts of interest, honesty and integrity, accountability, confidentiality and transparency.7 6 As per the terms of reference for the assessment. 7 United Nations Evaluation Group Principles for Evaluations (June 2008) [online] http://www.unevaluation.org/ document/detail/100. 10 FINAL ASSESSMENT REPORT 3. ASSESSMENT FINDINGS 3.1. Assessment findings by criterion 31. As previously stated, the evaluation criteria of relevance, effectiveness, efficiency and sustainability were used to analyse the assessment findings. 3.1.1. Project beneficiaries 32. The project was designed to collaborate with NSOs in 15 countries in the Caribbean subregion. These were Antigua and Barbuda, Bahamas, Barbados, Belize, Dominica, Dominican Republic, Grenada, Guyana, Haiti, Jamaica, Saint Kitts and Nevis, Saint Lucia, Saint Vincent and the Grenadines, Suriname and Trinidad and Tobago. 33. However, during project execution the decision was taken to broaden its scope to include Anguilla, Aruba, the British Virgin Islands, the Cayman Islands, Montserrat and the Turks and Caicos Islands. The reasons for this change were explored as part of the interviews, and it was suggested that the additional countries were included in an effort to make full use of the available funds. One of the KIs explained that since the funds were time-bound, it was felt that they should be utilized to support improvements in as many countries as possible. 34. Similarly, statisticians and other representatives of NSOs were the original target group for the project. The assessment revealed that staff from key line ministries such as health, social development, finance and economic affairs also benefited from inclusion in the project. Furthermore, other departments such as gender affairs, the police, the environment and agriculture were also included. Therefore, the project consisted of 203 national officials from a diverse range of professional backgrounds. 3.1.2. Relevance of the project 35. The three groups (national representatives, ECLAC staff/consultants and development partners/experts) viewed the Development Account project as pertinent to the needs and priorities of NSOs and governments. Forty five out of the 49 representatives from the target countries reported that the project was either very relevant or relevant (see figure 1). The six interviewees from the countries were also asked about the relevance of the project and they stated that the work was beneficial to countries in areas such as MDG monitoring, the collection and disaggregation of data, and enhancement of staff technical skills. Figure 1 The perceptions of national representatives regarding the relevance of the project In your opinion, how relevant was the project to your country’s national objectives? I do not know Somewhat relevant Relevant Very relevant 05 No. of respondents Very relevant 24 10 15 20 Relevant Somewhat relevant 21 3 25 30 I do not know 1 11 FINAL ASSESSMENT REPORT 36. Four out of the seven ECLAC staff members/consultants who responded to this survey question indicated that the project was very relevant. The following explanation was provided by one of the respondents: “The project aimed to enhance the capacity of data producers and users to develop muchneeded statistics and indicators to be used in the monitoring of the MDGs. [Since] data are quite limited in the Caribbean subregion.” 37. The three ECLAC staff members/consultants who participated in the interviews also shared the same perspective. It was noted that the project was “timely” and aligned to the countries’ needs, since it focused on the MDGs when support was needed in that area. In addition, the majority of development partners/experts who completed the survey – four out of five – also agreed that the project reflected the situation in the countries. 38. The assessment also explored linkages between the project and the work programmes of ECLAC and other development organizations. The staff members/consultants interviewed confirmed that the project was developed on the basis of the ECLAC work programme and was in keeping with requests for technical assistance from the member States. In terms of other agencies, KIs stated that the work reflected the priorities of United Nations agencies such as the United Nations Development Programme (UNDP), United Nations Statistics Division (UNSD), the United Nations Population Fund (UNFPA) and the United Nations Children’s Fund (UNICEF). Regional agencies such as the Organization of Eastern Caribbean States (OECS), Caribbean Community (CARICOM) and the Caribbean Development Bank (CDB) partnered with ECLAC but the extent of the technical support varied across the life span of the project. 39. The survey specific to development partners/experts also explored partnerships between ECLAC and other agencies. All six respondents stated that technical assistance was the main form of support, although one person indicated that their organization had also provided financial assistance to member States (see figure 2). This finding is consistent with the interviews with staff – it was reported that the United Nations Environment Programme (UNEP) provided US$15,000 directly to countries to support the participation of national officers in the training. Figure 2 The contributions of development agencies to the project What was the nature of your organizations partnership with ECLAC on the 2008-2011 project? Please select as many options as appropriate 1 6 Organization Financial assistance to the member States 12 FINAL ASSESSMENT REPORT 40. Partners were also asked to rate their collaboration with ECLAC. The responses showed that partners approved of their work with ECLAC on this project. Figure 3 illustrates that four out of the six respondents described the collaboration as “excellent”, while the other two rated it as “good”. One of the development partners also provided insights into the nature of the partnerships: “UNDP and ECLAC provided inputs according to their respective advantages, with UNDP coordinating in-country officials and stakeholders and ECLAC mobilizing external resources. The agencies worked together to support incountry consensus building.” Figure 3 Partner perspectives of the work with ECLAC How would you rate your organizations collaboration with ECLAC on the 2008-2011 project? 5 4 3 2 1 0 No. of respondents Excellent 4 Good 2 3.1.3. Relevance of the activities and the outputs 41. As stated earlier, it was envisaged that the project would focus on the provision of technical assistance, capacity-building in statistics and the documentation of regional achievements related to the MDGs. This section of the report examines whether these activities and the related outputs were in keeping with beneficiaries’ needs and priorities. A detailed analysis of the effectiveness of the various components is discussed in subsequent sections of the report. x Technical assistance to ECLAC member and associate States 42. Respondents were asked whether they had benefited either personally or through their organization from ECLAC support. Only 13 out of the 48 respondents stated yes, another 13 responded negatively and 22 people said they did not know (see figure 4). 13 FINAL ASSESSMENT REPORT Figure 4 Recipients of ECLAC support Did you or your institution receive technical assistance from ECLAC as part of the 2008-2011 project? 13 22 13 I do not know Yes No 43. During the interviews, KIs were also asked whether they had received technical assistance as part of the project. The data revealed that assistance mainly consisted of national workshops and KIs were unable to say whether any other form of support had been provided. 44. Respondents were also asked about the relevance of the technical assistance provided to the context in their countries. While there was a general lack of knowledge about ECLAC support, 11 out of the 13 respondents were of the view that the organization’s work was applicable to the situation in their countries. x Capacity-building: expert group meetings and training workshops 45. The purpose of the expert group was to outline an approach to collection, analysis and reporting on the MDGs in the subregion. Two meetings were held between February and June 2009 and these assisted with identifying the requirements in each of the countries and the types of support that could be provided as part of the project.8 46. Training workshops were designed to build staff’s technical capacity to collect and report on the MDGs and other IADGs. Statisticians and other personnel with responsibility for data were the main target audience but the scope was widened to include other branches of government. The nine workshops covered topics such as methodological aspects of data collection and analysis, computer applications such as Census Processing Systems (CSPro), specific MDG indicators, and the approach to the measurement of poverty in the subregion (table 5 lists the workshops that were held as part of the project). 8 See the project document and the reports of the expert group meetings. 14 FINAL ASSESSMENT REPORT Table 5 National and regional workshops Workshop topic Training Workshop on the Use of the Statistical Programme Census Survey Processing System (CSPro) Training Workshop on the Construction of Core Environmental Indicators Regional Workshop on REDATAM+SP XPLAN and Webserver Applications Development for Data Dissemination Training Workshop on the Use of the Statistical Programme Census Survey Processing Systems (CSPro) Computer Editing of Population and Housing Census Data National Workshop on the Construction of Environmental Statistics and Indicators National Training Workshop on Capacity Building for the Production of Reliable Disaggregated Data Regional Workshop on the Measurement of Poverty in the Caribbean National Training Workshop on Capacity Building for the Collection of Data Regional Evaluation on Strengthening the Capacity of National Statistical Offices in Caribbean Small Island Developing States to fulfil the Millennium Development Goals and Other Internationally Agreed Development Goals Date 25 October - 2 November 2010 6-10 December 2010 2-12 August 2010 15-25 August 2010 19-21 September 2011 18-20 October 2011 27-28 October 2011 1-3 November 2011 12 December 2011 47. A comparison of the data pertaining to the expert group meetings and the workshops revealed that all three groups participated in events where they contributed to project execution or honed their skills. Thirty-six out of the 49 respondents from the project countries indicated that they participated in either the expert group or in capacity-building sessions. Further analysis highlighted that personnel from the countries were involved in the workshops more so than the expert group, since 33 out of the 35 respondents reported that they participated in either the national or regional workshops. When asked to identify the workshops attended, the mostly commonly selected workshop was the National Workshop on the Construction of Environmental Statistics and Indicators. The interviews with representatives of NSOs and other government agencies confirmed this finding. 48. The assessment also determined the links between the capacity-building sessions and the situation in the respective countries. The responses to this survey question were favourable: 31 of 35 people selected very relevant or somewhat relevant (see figure 5). 15 FINAL ASSESSMENT REPORT Figure 5 Perceptions of the relevance of the training to national priorities To what extent do you consider the expert group meetings and/or training workshops relevant to the situation in your country? 2 2 19 12 Very relevant Relevant Somewhat relevant I do not know 49. The results show that ECLAC staff and consultants had a similar perspective to that of public officials. This group was more involved in the workshops than in the expert group and project steering committee. 50. As expected, development partners and experts accounted for membership of the expert group and also provided technical support for project implementation (see figure 6). The involvement of this group in training was limited and it could be concluded that this was due in part to their contributions to the design and facilitation of the sessions. Figure 6 The role of development partners and experts Which of the following statements best describes your involvement in the ECLAC project? Please select as many options as appropriate. I participated in the regional training 1 Other (Please specify) I participated in expert group meetings I was involved in the provision of technical assistance to Member States Response Count 2 3 3 16 FINAL ASSESSMENT REPORT x Publication of the Caribbean MDG report 51. The development and publication of a report outlining progress towards achievement of the MDGs was another output of the project. Knowledge of the publication among national officials was mixed. Twenty-four of the survey respondents indicated that they were familiar with the report. Conversely, the other 24 respondents did not know about the document. The findings from the interviews were no different – three out of the six interviewees stated that they had never heard of the report. 52. Respondents were asked if they considered the Caribbean MDG report to be relevant to the situation in their country. This question was specific to those respondents who had indicated that they were familiar with the report. As illustrated in figure 7, 21 out of 24 participants described the report as pertinent to national interests. Figure 7 Relevance of the Caribbean MDG report to national interests To what extent do you consider the Caribbean MDG Report 2010 to be relevant to the situation in your country? 3 4 9 8 Very relevant Relevant Somewhat relevant I do not know x Is the project still relevant to the needs of the target countries? 53. The Sustainable Development Goals (SDGs) have provided a new framework to guide the focus of development work over the next 15 years. It could be argued that the SDGs will build on the achievements associated with the MDGs and other IADGs because of similarities in areas such as poverty, education and gender equality. The importance of NSOs and other government entities with SDG-related mandates are just as important as when the project under review was conceptualized. 54. While there have been improvements in NSO capacities and by extension those of the staff, the KIs highlighted that further work is needed. Several of the statisticians interviewed pointed to the need for ongoing training due to the loss of skills and staff changes to other areas and promotions. KIs also reported challenges related to the sharing of data within countries, despite a multisectoral approach that included a wide range of government departments in the project. The need for the greater collaboration among regional NSOs was cited by the interviewees, particularly in areas such as data harmonization and the sharing of best practices. 17 FINAL ASSESSMENT REPORT 3.1.4. Effectiveness of the project 55. This Development Account project contributed to the strengthening of NSOs in the subregion. This is evidenced by the examples provided by the target groups. During the interviews, national focal points stated that the project had supported the development of skills, which ultimately resulted in improvements within the NSOs. Furthermore, an ECLAC staff member/consultant expressed a similar opinion: “NSOs have started to use the CSPro and REDATAM software that they were trained in.” “[In addition], countries also used the indicators that they developed in the workshops on gender and the environment.” One of the development partners/experts noted: “The support was timely, supported critical discussions on key issues related to the [MDGs] and linked well with other activities.” 56. The assessment report also discusses the findings related to the main activities. It will examine the results as identified by the beneficiaries, their level of satisfaction with the activities and any reported improvements in the knowledge and skills of the officers. It will also determine whether there were any changes in the ways in which data are reported at the national and regional levels, whether any policies to support data collection, analysis and use were developed and whether a database was created for use by the countries. x The national and regional workshops 57. The majority of the assessment participants were exposed to training as part of the project. Therefore, capacity building was a regular discussion point during the interviews and rated highest in the surveys. Thirty-five out of 49 national officials rated the content of the meetings or training workshops. The responses selected were positive and suggest that participants were generally satisfied, since 20 respondents selected “good” and 15 opted for “excellent” (see figure 8). Figure 8 The satisfaction of national officials with the meetings and workshops Overall, how would you rate the content of the meetings or training workshops? 15 20 Excellent Good 58. The interviews with representatives from NSOs and other public agencies also examined the perceived benefits of the workshops and meetings. All six interviewees stated that they had benefited from this component of the project. The survey data showed similar results. Thirty-five representatives 18 FINAL ASSESSMENT REPORT responded to this question and the majority either strongly agreed or agreed with the options presented. That is, respondents felt that they had benefited either through increased levels of awareness of the MDGs and other IADGs or had been able to undertake specialised tasks (see figure 9). Only three respondents said that they had not benefited from the project. Figure 9 National officials and the perceived benefits of the expert group meetings and/or training workshops Do you feel that you benefited in any way from participation in the expert group meetings and/or training workshops? 25 21 20 20 18 17 17 15 12 10 9 66 5 0 Strongly agree 9 33 1 5 Agree Neutral 9 11 6 4 232 Disagree Strongly disagree Not applicable (N/A) I am more aware of the importance of reporting on the MDGs and other IADGs I am able to undertake work related to the collection of data I am better positioned to analyse national data I am involved in the dissemination of data through various outputs such as presentations and reports I do not believe that I benefitted from the ECLAC project 59. In order to further assess the usefulness of the capacity-building initiatives, respondents were asked about their application of the knowledge and skills over the six months prior. According to the results, the technical abilities of national representatives improved thanks to their participation in the project. Only five out of the 35 respondents stated that they never used the information provided during the training. Twenty-two respondents applied what they had learned on a regular basis and 8 people reported infrequent use (see figure 10). 19 FINAL ASSESSMENT REPORT Figure 10 Reported use of knowledge and skills in the last six months In the last six months, how often have you used the knowledge and skills gained in the meeting or training workshops? 15 14 Number of respondents 10 88 5 5 0 Sometimes Often Seldom Frequency Never x Technical assistance 60. Levels of satisfaction with the technical support provided were high. Six respondents declared themselves to be very satisfied while the other six said they were somewhat satisfied (see figure 11). Figure 11 Levels of satisfaction with the assistance provided by ECLAC How satisfied are you with the technical assistance that you received from ECLAC? 66 Very satisfied Somewhat satisfied 61. The six KIs from the countries were also asked about their levels of satisfaction with ECLAC support. Not only were high levels of satisfaction reported but the interviewees were able to share examples of work that occurred as a result of the project. A senior statistician indicated that her organization had produced a publication on the environment-related MDGs. One of the officers attached to the Bureau of Gender Affairs stated that the national workshops had provided an opportunity to engage stakeholders from multiple sectors and this had contributed to the development of the country’s gender policy. 62. Furthermore, all 12 participants indicated that ECLAC support was useful. The utility of ECLAC support is illustrated by the following comments: “It made us more aware of the importance of collecting the relevant data in a systematic way”. Another respondent stated that they are “better position[ed] to construct and interpret indicators and to analyse data.” 20 FINAL ASSESSMENT REPORT x Publication of the Caribbean MDG report 63. According to the survey data, 13 out of the 24 national representatives agreed that they had contributed to the development of the publication on the MDGs in the subregion. The respondents were also asked to rate the quality of the publication. Twelve of the 24 respondents selected “good” and four people stated “fair” and “excellent”, respectively. Four of the participants could not provide a response about the quality of the report (see figure 12). When the survey data were compared to the interviews, it was found that two out of the three interviewees had stated that the quality of the report was good. Figure 12 Opinions of national officials regarding the quality of the MDG report How would you rate the quality of the Caribbean MDG Report 2010? 4 4 12 4 Good Excellent Fair I do not know 64. The assessment also looked at levels of satisfaction with ECLAC support in the preparation of the MDG report. Sixteen out of 24 respondents reported that they were either very satisfied (8) or somewhat satisfied (8) with the technical assistance provided. Three respondents were neutral and five selected the option “I do not know” (see figure 13). Figure 13 Satisfaction with ECLAC support in the preparation of the Caribbean MDG report How satisfied are you with ECLAC support in the preparation of the Caribbean MDG Report 2010? 88 5 3 Very satisfied Somewhat satisfied I do not know Neither satisfied nor dissatisfied 21 FINAL ASSESSMENT REPORT x Improvements in the collection, analysis and use of data 65. There was a lack of consensus among national officials when asked to rate statements on improvements in the collection, analysis and use of data (see figure 14). When questioned about improvements in the collection of data within their respective countries, 16 of the 45 respondents agreed that there had been improvements and 15 opted to remain neutral. The second statement asked about improvements in the analysis of data, which 17 respondents agreed with while 16 people remained neutral. Perceptions of the use of data at the country level yielded 21 neutral responses as against 11 who agreed with the statement. Finally, 15 respondents agreed with the statement that participation in the ECLAC project had resulted in improvements in the collection, analysis and use of data, while 16 respondents indicated a neutral opinion. On the basis of the data, it could be concluded that the representatives who completed the survey were ambivalent about improvements in their respective countries. Figure 14 Participation in the ECLAC project and improvements in data collection, analysis and use at the country level 25 20 15 10 8 77 5 5 0 Strongly agree To what extent do you agree or disagree with the following statements? 16 17 15 11 21 15 16 16 Agree Neutral 3433 3333 1 Disagree Strongly disagree Not applicable (N/A) Participation in the ECLAC project has resulted in improvements in my country’s focus on the collection of data on social, economic and environmental indicators Participation in the ECLAC project has resulted in improvements in my country’s focus on the analysis of data on social, economic and environmental indicators Participation in the ECLAC project has resulted in improvements in my country’s focus on the use of data on social, economic and environmental indicators Participation in the ECLAC project has resulted in improvements in my country’s focus on the collection, analysis and use of data x Improvements in reporting on the MDGs and other IADGs 66. The survey for national representatives also asked specifically about improvements in reporting on the MDGs and other IADGs. The responses to this question were also mixed; 23 out of 45 participants selected “Yes”, 18 selected “No” and 4 said they did not know (see figure 15). One of the officials who believed that improvements had occurred stated the following: “Some of the recommendations were already being implemented in my country. However, it [the ECLAC project] has helped to reinforce best practices.” Another representative noted: “There is more focus on reporting on the core indicators.” And a similar perspective was shared by a third person: “Greater efforts have been made by [the] various organizations to collect data and compile reports.” 67. National officials stated during the interviews that improvements had been seen in areas such as MDG monitoring, the calculation and reporting of environmental indicators, the accessibility of data within countries, and collaboration between departments with responsibility for data. 22 FINAL ASSESSMENT REPORT Figure 15 Opinions on improvements in reporting on the MDGs and other IADGs Due to your country’s involvement in the 2008-2011 ECLAC project, do you think that there have been improvements in the reporting on the MDGs and other IADGs? 4 23 18 Yes I do not know No 68. The survey targeting ECLAC staff and consultants also asked if improvements in most countries could be attributed to this project. Three out of the four respondents agreed with the statement. The same group also expressed similar sentiments during the interviews; it was stated that countries are better positioned to report on the MDGs thanks to the Development Account project. Furthermore, another KI stated that capacity had been built and there were greater levels of awareness about reporting on the MDGs. When compared with the data specific to development partners and experts, three of the people who responded agreed with the question and two opted to remain neutral. 69. The country representatives were also asked whether their countries had been able to maintain improvements in reporting. Fourteen of the 23 participants said yes, six opted for “I do not know” and three responded “No” (see figure 16). One of the respondents shared this view: “[Yes], reporting has made gaps visible and we have been working to improve those.” Another participant said: “Not only have we been able to keep in existence long-term reporting, but we have effectively fulfilled demands and requirements internationally.” Figure 16 The continuity of improvements over the long term Do you believe that your country has been able to sustain the improvements in reporting over the long term? 3 6 14 Yes I do not know No 23 FINAL ASSESSMENT REPORT x National policies on the collection, analysis and use of data 70. Respondents from the target countries were generally unaware of any policies that had been developed following completion of the project. Only one participant out of 45 selected yes, 12 responded negatively and most people (32) indicated that they did not know. The respondents explained that the development of a policy was planned for 2016 through the focal point from the NSO and they were not in a position to respond to the question because they were not attached to the statistical office. Furthermore, all five of the development partners/experts who answered the question selected “I do not know”. 71. When ECLAC staff and consultants, however, were asked if any of the member States had developed policies geared towards the collection, analysis and data, the respondents indicated that eight countries had done so. The countries mentioned were Antigua and Barbuda, Barbados, Belize, the Dominican Republic, Grenada, Haiti, Saint Lucia and Saint Vincent and the Grenadines. When asked for details about the policies, only one of the four respondents who completed this question was able to respond. They referred to the “collection, analysis or use of data.” The differences in the responses from the national representatives and ECLAC staff could be attributed to two factors. First, awareness depends on the job function of the respondents from the project countries. Second, ECLAC staff may have heard about a policy but not been directly involved in its development. x Development of a database on social, economic and environmental indicators 72. The assessment revealed that activities related to the development of a database had been conducted (the first draft of the database was produced and the server procured, a website was set up to assist with information-sharing, and the documentation to assist users with the technical components of the database and website was prepared).9 However, this activity was not followed through, which illustrates how changes in personnel affected the continuity of activities during the project. x Positive or negative outcomes of the project 73. National representatives were asked about any outcomes (either positive or negative) that were a direct result of the project. Only two of the six interviewees were able to respond to this question. One of the statisticians stated that there was a shift in her country from focusing purely on the monitoring of indicators to emphasis on the preparation of reports once the data were collected. Another respondent whose job function does not directly relate to statistics was of the view that the multisectoral approach to the national workshops had created greater levels of awareness about the data that were collected and the possible contributions of other stakeholders. 74. Only one of the four ECLAC staff members/consultants who answered the question indicated that the project had had positive outcomes. However, this participant did not provide examples to support their views. The interviews facilitated the collection of more detailed responses. One of the staff members opined that while it was not anticipated that the project would cover such a wide range of areas, this change was beneficial for the countries. Another interviewee said that collaboration with partners had been advantageous because it complemented the technical skills of the staff. 75. Out of the four development partners who answered the question, only one respondent identified a positive but unexpected result of the project. The respondent noted the following: “[The project highlighted] that there were key differences in the measurement of poverty (the basket and method) from one survey year to the next, and that this meant they should not be used to interpret trends without adjustments being made.” 9 As per the Development Account terminal report. 24 FINAL ASSESSMENT REPORT 76. In summary, none of the three groups indicated that they were any negative outcomes associated with the project. 3.1.5. Efficiency of the project 77. This section of the report discusses the factors contributing to the achievement of the project’s overall results. It begins with an examination of the role of the Steering Committee, and continues with an analysis of project management and implementation. The manner in which resources were utilized as well as the cross-cutting themes of gender and human rights are also assessed. x The Steering Committee 78. The role of the Steering Committee was a discussion point during interviews with staff. The Committee was comprised of representatives from the CARICOM Secretariat, the United Nations Development Programme, the International Labour Organization, the United Nations Statistics Division and the Statistics Division at ECLAC headquarters, one independent expert and select personnel from the countries. 79. It was envisaged that the Committee would provide technical oversight for project implementation. In keeping with the project document, one face-to-face meeting was held on 3 February 2009. The purpose of this meeting was to discuss the role of the Committee and to outline the ways in which it would contribute to project implementation. Owing to financial constraints, it was proposed that faceto-face meetings would be limited and use would be made of technology such as electronic mail and video conferencing.10 80. Initially the Committee was involved in the provision of technical input at workshops, such as the Regional Workshop on the Measurement of Poverty and the formation of the Caribbean MDG Report. However, two out of the three KIs stated that the Committee had not functioned as planned. None of the interviewees were able to provide specific reasons for the group’s non-performance but it was suggested that it was due in part to perceived needs at the time and differences in the work programmes of the various organizations. x Project design and management 81. The KIIs with ECLAC staff and national focal points revealed that the project document had been prepared without direct input from the beneficiaries. KIs noted that the project had been designed on the basis of the ECLAC programme of work in the subregion and requests for technical assistance from some of the member States. However, on formulation of the project, there was no forum to obtain feedback on the planned intervention. As one KI stated: “The project was designed quite quickly.” Therefore, national officers were not afforded the opportunity to provide feedback on activities or establish priorities once the initial plans had been determined by ECLAC. 82. The project reports and KIIs revealed that reporting was conducted on an annual basis and that this only served to provide status updates about the project. It was noted during the interviews that this level of monitoring was insufficient because there was no mechanism for project managers to make adjustments as needed. 83. During the project period, three ECLAC staff members were responsible for project management. When asked about any challenges relating to project execution, staff turnover was an area that was repeatedly mentioned by interviewees. Key informants opined that “knowledge transfer” had been limited and that this had contributed to a lack of activity continuity. That is, the focus of the project changed according to the project manager: one KI mentioned the regional database as an example, since work had begun on its establishment but had not been completed. 10 As per the project document and the Report on the First Meeting of the Steering Committee, March 2009. 25 FINAL ASSESSMENT REPORT 84. The interviews provided insights into other human resource challenges. Key informants suggested that delays had occurred due to limited human resource capacity at the subregional office. Another KI held a similar opinion but stated that partnerships had been established with other development partners to complement the skills available at ECLAC. Furthermore, staff also suggested that their workloads and the competing demands may have contributed to delays in the project. x Collaboration between ECLAC offices 85. Interviewees described the working relationship during the project as “efficient” and “positive” and stated that this augurs well for future collaboration. Most of the staff members/consultants (three out of four) who completed the survey indicated that the project was successful thanks to the efforts of personnel from both offices (see figure 17). Figure 17 Collaboration between ECLAC Port of Spain and headquarters Which of the following statements best describes the level of collaboration between ECLAC headquarters and the subregional office during execution of the Development Account project? 3 1 The level of cooperation between both offices contributed to the success of the project The project was successful but this was mainly due to the input of the staff at the subregional office x The use of resources 86. Time: The staff interviewed indicated that adherence to timelines was an issue throughout the lifetime of the project, that is, from the design stage through to implementation of the components. It was reported that the delays were due to human resource constraints at ECLAC, namely the assignment of personnel to multiple projects and staff changes where the continuation of some actions was limited. 87. Technical: The utilization of technical resources external to ECLAC was one of the strengths of the project. Representatives from partner agencies provided technical input on project execution as well as subject matter expertise in areas such as statistics, poverty and the MDGs. 88. Focal points from the target countries provided support on the administrative and logistical functions relating to the national workshops and technical assistance missions. Senior statisticians from select member and associate States participated in the expert group meetings and officials also participated through membership of the Steering Committee. Throughout the interviews, KIs emphasized the need for greater collaboration among countries, which could have been facilitated if national officials had been given a greater role in the project. Furthermore, the involvement of country representatives could have contributed to greater levels of ownership during and after the Development Account project. 26 FINAL ASSESSMENT REPORT 89. Financial: The allotment for the project was US$ 528,000, with an implementation rate of 86% or US$ 455,812.15. These finances covered expenses relating to human resources (temporary positions, consultants and the expert group), staff travel, contracts, operating expenses, supplies/materials and other contributions.11 90. Discussions with ECLAC staff and consultants focused on the use of the available financial resources. Only two out of the three interviewees felt that they were in a position to respond to this question. Their responses highlighted the differences in management of the activities as well as use of the funds. One of the KIs was of the view that the funds had been spent in accordance with the proposed plans for the project. Another interviewee, in contrast, stated that in some instances the funds had not been utilized as anticipated. For example, the meta-database had not been established and the workshops had been expanded to include a wider range of groups and countries. 91. Internal protocols and guidelines: The interview data suggest that the only guidelines used during the design of the project were those specific to the Development Account. When asked if the project had had policies and procedures that contributed to its effectiveness and efficiency, two out of the four ECLAC staff members and consultants who completed the survey stated “yes” but opted not to provide details about the guidelines. x Documentation of good practices/lessons learned 92. None of the KIs interviewed were aware whether the lessons learned had been documented as part of the project. However, this information was captured as part of the progress and terminal reports.12 The lessons learned include the importance of: (a) Advocacy and relationship building, since it was found that countries were initially “reluctant” to take part in the project, particularly in the sharing of data. (b) Scheduling regional meetings and workshops to avoid conflicts with similar activities. (c) Communicating with other development agencies (including United Nations agencies) to coordinate activities in an effort to identify opportunities for collaboration and reduce the duplication of resources and the consequent burden on national authorities. (d) Involving stakeholders in project design and implementation. It was found that this may have contributed to greater levels of ownership and the long-term sustainability of the activities. 3.1.6. Cross-cutting themes 93. Gender: It was found that gender was considered from two perspectives – the inclusion of male and female participants along with the content of the training sessions. One of the respondents to the staff survey stated: “Every training workshop had a gender mainstreaming component. Some countries also took part in workshops related to developing gender indicators specifically.” A similar perspective was shared by the second respondent who noted: “Gender-disaggregated data for analysis and reporting was a main component of the project.” 94. Human rights: The staff and consultants who participated in the interviews and surveys indicated that human rights issues are an integral part of the work programmes of UN agencies. In addition, one of the respondents stated: “The project focused on developing statistics and indicators in areas that needed improvement in the Caribbean subregion. Some of these included the development of gender indicators and poverty indicators, to name a few human rights issues.” 11 As per the Development Account terminal report. 12 Refer to the Annual Development Account Progress Reports (2009 and 2010) and the Development Account terminal report. 27 FINAL ASSESSMENT REPORT 3.1.7. The sustainability of the project and/or its interventions 95. Sustainability was addressed at two levels as part of the review – within the target countries and the regional commissions. It was found that attention was not paid to the continuity of activities on completion of the project. Sustainability issues are mentioned within the project document but this is limited to alliances with agencies such as UNDP, CARICOM and UNSD. It appears that ECLAC did not consider the uptake and ownership of the interventions by the countries. This is evidenced by the lack of a sustainability plan and the exclusion of countries in the design of the project. 96. The ECLAC staff and consultants interviewed were also asked if the long-term viability of the project or its activities was considered during the design phase. All three interviewees reported that this was done but they could not fully articulate in what ways. When compared with the survey results for the same group, it was found that confidence in the long-term viability was relatively low. Two out of the five respondents selected “I do not know”, one person selected “To some extent” and the other respondent indicated “To a small extent”. The reasons for this apparent lack of confidence included high staff turnover within NSOs and governments in general, the economic challenges with which the majority of countries are faced and the unwillingness of governments to take responsibility for donorfunded projects in the long term. The development partners/experts provided similar responses to staff, as three respondents selected “I do not know” and two indicated “To some extent”. 97. None of the interviewees from the countries were able to provide information regarding whether any formal plans were made to continue the interventions either in whole or part. The discussions with KIs revealed that projects of this nature may be seen as finite and not as the basis for ongoing or new programmes. Therefore, interviewees were able to speak to changes at the individual level (for example changes in knowledge) but they were unable to provide details on significant changes within their national programmes. 28 FINAL ASSESSMENT REPORT 4. GENERAL CONCLUSIONS AND RECOMMENDATIONS 4.1. Relevance 98. The main objective of strengthening NSOs to support improvements in the MDGs and other IADGs was in keeping with the needs of the target countries because of the challenges posed by the collection, analysis and reporting of national achievements. Consequently, the activities formed part of the work programmes of ECLAC and by extension other United Nations agencies operating in the Caribbean. Regional agencies such as CARICOM and OECS contributed to project execution but their involvement was limited and did not fully reflect the functions of the two organizations. Recommendation 1: ECLAC should formalize its partnerships with regional development agencies such as CARICOM and OECS. 99. ECLAC should ensure that regional development partners such as CARICOM and OECS play a greater role in the planning and implementation of projects of this nature. Thanks to shared mandates, these organizations are well positioned to provide technical and/or financial inputs to projects such as this. Assessment finding number 38 highlighted that CARICOM and OECS were involved in the initial stages of the project but this was not sustained for its duration. It is therefore recommended that ECLAC formalize its partnerships through agreements such as a Memorandum of Understanding that outline the objectives and scope of the collaboration. 4.2. Effectiveness 100. On the whole, stakeholders viewed the project as being relevant to the needs and priorities of their countries. However, when analysing the effectiveness of the various components, it appears that not all of the activities were in keeping with the priorities of the target countries and by extension the national focal points. Activities such as the training workshops and in-country technical assistance in which participants were directly involved were deemed more beneficial. Improvements in knowledge and skills at a personal level were reported but the training had not been applied as widely as envisaged. The majority of respondents could not state with certainty that the project had resulted in significant improvements within the member and associate States. Either application of the training had been limited or the assessment participants were unfamiliar with the work of the NSOs in their country. 101. The publication of the Caribbean MDG Report and the development of a database to support data harmonization were the regional-level activities conducted under the project. Surprisingly, some respondents were unaware of this document, which suggests that it was not widely distributed within the countries. The database to support data harmonization was not developed under the project and this was due more to the absence of effective monitoring than to a lack of interest in the countries. 102. The success of the project may have also been compromised by its scope. The inclusion of 21 countries as well as national and regional activities meant that focus was lost and activities such as capacity-building workshops received more attention to the detriment of others. 29 FINAL ASSESSMENT REPORT Recommendation 2: ECLAC should involve country focal points in the establishment of priorities for projects prior to their approval. 103. Assessment finding number 81 noted that countries were not directly involved in the development of the project. ECLAC staff should consult with national representatives at the design stage, since it is unlikely that significant changes will be possible once a project is approved internally. This would facilitate the inclusion of firsthand accounts of needs, challenges and opportunities for collaboration. Recommendation 3: ECLAC should ensure that routine monitoring, incorporating a mechanism for adjustment, is part of any future Development Account project. 104. Assessment finding number 82 highlighted that, owing to a lack of effective monitoring, deficiencies were not addressed when necessary. Therefore, ECLAC should make routine monitoring part of all future Development Account projects. However, the proposed time frame for this function should be reduced from annual to every three to six months. Results-based monitoring would not only have flagged changes in the approach to information systems but also provided an opportunity for corrective action. 4.3. Efficiency 105. The Steering Committee was tasked with ensuring that the project was implemented in accordance with the planned approach. The Committee’s involvement in the project was more ad hoc in nature and it was therefore unable to fulfil its role of providing project oversight. 106. The effectiveness of the project was also weakened by the turnover in project managers. The assessment highlighted that while all three staff members were committed to the project, their approach to its execution differed significantly. Recommendation 4: ECLAC should clearly define the roles and functions of project steering committees. 107. ECLAC should provide guidance on the purpose of project steering committees. Documentation of this nature would assist committee members in understanding their roles in relation to project oversight. In addition, the organization should also plan for face to-face-meetings, because in this case these would have provided structure and improved the functioning of the group. Assessment findings numbers 78-80 highlighted that the Steering Committee did not function as planned. The group was involved in the early stages of the project but this did not continue in subsequent years. It should be noted, however, that KIs were unable to provide reasons as to why the Steering Committee had not functioned as expected. Recommendation 5: Where possible, ECLAC should limit staff changes during the implementation period of a project. 108. Where possible, ECLAC should limit staff changes or minimize their impact by providing the relevant documentation to incoming project managers along with briefings on the project status. Assessment finding numbers 83-84 illustrate the impact that staff changes had on the project. It was found that there were differences in the timeliness of the activities, and staff members had prioritized activities based on their perceptions of what was needed in the countries and on the basis of the available funds (that is, from the Development Account project). These changes resulted in delays in the project’s execution, increases in the number of countries from 15 to 21 and the discontinuation of activities. 30 FINAL ASSESSMENT REPORT 4.4. Sustainability 109. The intervention was limited to a set period and consideration was not given to the continuation of activities by ECLAC or the project countries. The sustained involvement of some partners was a challenge and the countries did not make any formal plans to build on existing work. The lack of ownership could be attributed to the exclusion of interest groups during project design, the perceived role of agencies like ECLAC and the failure of countries to commit the required resources. Recommendation 6: Project sustainability should be considered by ECLAC and the target countries. 110. The assessment findings (numbers 95-97) reveal that sustainability issues were not considered by ECLAC or the countries. It is recommended that ECLAC hold consultations with policymakers and senior staff to underscore the value that could be derived from the maintenance of activities. Governments should be encouraged to build on the gains from Development Account projects by maintaining the components in the form most relevant to their national requirements and interests. These consultations should be documented and the ECLAC role (if any) as well as that of national authorities should be outlined. In addition, sustainability plans should also outline the activities to be continued, responsible agencies, expected outcomes and so forth. 4.5. Cross-cutting themes 111. Gender: Assessment finding number 93 noted that, within the context of this project, gender was limited to ensuring that females were included in the activities. Furthermore, the topics for the training sessions focused on gender-specific indicators that form part of the MDGs and other IADGs. It was also noted that gender was considered in relation to the collection, analysis and reporting of data. Therefore, gender was applied in relation to the project’s activities and not in accordance with its theoretical underpinnings. 112. Human rights: Participants provided positive feedback regarding ECLAC and the Development Account project. It could therefore be concluded that the human rights of the participants were respected. However, as is the case with gender, human rights were viewed in relation to the project and possible improvements within the subregion and not in relation to the participants (assessment finding number 94). Recommendation 7: ECLAC should train its staff on the incorporation of gender and human rights into Development Account and other projects. 113. It is recommended that ECLAC train its staff on the inclusion of cross-cutting themes such as human rights and gender in future projects. Training is necessary since the assessment highlighted that application of these concepts is subjective in nature and may not be in keeping with the original basis for their inclusion. 31 FINAL ASSESSMENT REPORT 5. KEY LESSONS LEARNED 114. The most significant lessons learned pertaining to the design, execution and evaluation of the project are outlined for further consideration by ECLAC. The assessment highlighted that: x Lesson 1: Steering committees may not function as expected if the roles and functions of the grouping are not clearly stated and followed. x Lesson 2: Changes in project management may affect the consistency of activities. In other words, if routine monitoring and adjustments are not undertaken, project activities may be conducted in accordance with the skills and interests of staff members more so than the original plans for the intervention. x Lesson 3: The successful conclusion of activities is dependent on the human resources available within ECLAC and its partners. x Lesson 4: Activities should be geared towards participants’ needs and interests. x Lesson 5: The long-term sustainability of a project is influenced by beneficiaries’ level of involvement, particularly during its development. x Lesson 6: Evaluations should be completed immediately after the conclusion of a project. Delays may result in low participation rates or limited responses owing to difficulties in recalling events. x Lesson 7: ECLAC should train its staff on the incorporation of gender and human rights into its programmes. The assessment highlighted that the application of these topics is subjective and dependent on the perspectives of the respective staff members. 32 ANNEXES ANNEX 1 Data Collection Tools (Surveys and KIIs) ANNEX 2 List of Key Informants ANNEX 3 Assessment Terms of Reference ANNEX 4 Evaluator’s revision matrix ANNEX 1 Data Collection Tools Survey One Target Group: Representatives of NSOs and other Departments/Ministries The United Nations Economic Commission for Latin America and the Caribbean (UN ECLAC) has commissioned an assessment of the Development Account Project ROA/122 entitled: Strengthening the Capacity of National Statistical Offices (NSOs) in the Caribbean Small Island Developing States to fulfil the Millennium Development Goals (MDGs) and other Internationally Agreed Development Goals (IADGs). The project was implemented between July 2008 and December 2011, with the main objective of strengthening the capacity of statistical offices to improve the quality of reporting on the MDGs. The activities included expert group meetings, training workshops, a publication on the Caribbean MDGs and the provision of technical assistance to project countries. Our records indicate that you participated in at least one of the project’s activities and we are therefore seeking your support with the completion of this survey. We would like to learn more about your experiences as one of the project’s participants. You may have been involved in in various ECLAC activities over the years but this survey relates to the 2008-2011 project activities only. Should you agree to participate in the assessment, it should take approximately 20-25 minutes to complete the survey. Any information that you provide will be kept in strict confidence, your answers will not be linked to you in any way, and your name will only be listed in the final report as one of the stakeholders who participated in the assessment. Please note that the survey link is unique to you and should not be shared with others. We would appreciate it if you could complete the survey by Midnight (Eastern Caribbean Time) on September 25, 2015.The survey findings will be used by ECLAC to improve its programming in the region. Should you have any queries please feel free to contact either evaluacion@cepal.org or Ms. Nicole Hazel -Independent Monitoring and Evaluation Consultant at nicole.hazel@gmail.com. Thank you for your support! 35 FINAL ASSESSMENT REPORT Section One: Background Information 1. Please select your gender. a) Male b) Female 2. Please select your country of residence/employment at the time of the project. Only one country should be selected. (a) Antigua and Barbuda (b) Bahamas (c) Barbados (d) Belize (e) Dominica (f) Dominican Republic (g) Grenada (h) Guyana (i) Haiti (j) Jamaica (k) St Kitts and Nevis (l) Saint Lucia (m) St Vincent and the Grenadines (n) Suriname (o) Trinidad and Tobago (p) Other,(Please specify)_______________________________________ 3. Please select the option that best describes your organisation at the time of the project? (a) National Statistical Office (b) Other Government Department or Ministry (c) Non-Governmental Organisation (d) University or Research Institute (e) Other, (please specify)____________________________________ 4. What was your occupation at the time of the project? (a) Statistical Assistant/Clerk (b) Statistician (c) Senior Statistician (d) Research Assistant/Field Worker (e) Research Officer (f) Census Manager (g) Data Manager/Data Processing Manager (h) System Administrator/Information Technology (i) Environmentalist (j) Environmental Health Officer (k) Deputy Director (l) Director or Head of Department (m) Other, (Please specify)______________________________________ 36 FINAL ASSESSMENT REPORT Section Two: General Questions 5. The project’s objective was to strengthen the technical capacities of the targeted Caribbean government’s statistical offices, through networking of institutions and experts, to improve the social and environmental indicators to measure poverty, social cohesion, and environmental sustainability with a view to advancing the fulfilment of the MDGs. In your opinion, how relevant was the project to your country’s national objectives? (a) Very relevant (b) Relevant (c) Somewhat relevant (d) Not very relevant (e) I do not know Section Three: Expert group Meetings and Training Workshops The following expert group meetings and training workshops were implemented as part of the project: (a) National Training Workshop on Capacity Building for the Production of Reliable Disaggregated Data (Antigua and Barbuda) 18 – 20 October 2011 (b) Regional Workshop on the Measurement of Poverty in the Caribbean (Port of Spain) 27- 28 October 2011 (c) National Training Workshop on Capacity Building for the Collection of Data (Turks and Caicos Islands) 1 – 3 November 2011 (d) Workshop on the Construction of Environmental Statistics and Indicators 19-21 September 2011 (e) Training Workshop on the Use of the Statistical Programme Census Survey Processing Systems (CSPro) for Computer Editing of Population and Housing Census Data 15 – 25 August 2011 (f) Regional Workshop on REDATAM+SP XPLAN and Webserver Applications Development for Data Dissemination 2 – 12 August 2011 (g) Training Workshop – Construction of Core Environmental Indicators 6 - 10 December 2010 (h) Regional Evaluation on Strengthening the Capacity of National Statistical Offices in Caribbean Small Island Developing States to fulfill the MDGs and other IADGs 12 December 2011 (i) First meeting of the Steering Committee on Strengthening the Capacity of National Statistical Offices in the Caribbean Small Island Developing States to fulfil the MDGs and IADGs 3 February 2009 (j) Expert Group Meeting on Development Account-supported Project on MDG/IADG Monitoring in the Caribbean 2 - 3 February 2009 (k) Second Expert Group Meeting on the Development Account-supported Project on MDG/IADG monitoring in the Caribbean 16 - 17 June 2009 (l) Technical Meeting on the 2010 Caribbean Millennium Development Goals Report 24 – 25 June 2010 (m) Training Workshop in the Use of the Statistical Programme Census Survey Processing System (CSPro) 25 October – 2 November 2010 6. Did you participate in any of the expert group meetings or training workshops listed above? (a) Yes (b) No (c) I don’t know 37 FINAL ASSESSMENT REPORT 7. Please identify which of the following activities that you participated in? Please check all of the options that apply. (a) National Training Workshop on Capacity Building for the Production of Reliable Disaggregated Data (Antigua and Barbuda) 18–20 October 2011 (b) Regional Workshop on the Measurement of Poverty in the Caribbean (Port of Spain) 27- 28 October 2011 (c) National Training Workshop on Capacity Building for the Collection of Data (Turks and Caicos Islands) 1–3 November 2011 (d) Workshop on the Construction of Environmental Statistics and Indicators 19–21 September 2011 (e) Training Workshop on the Use of the Statistical Programme Census Survey Processing Systems (CSPro) for Computer Editing of Population and Housing Census Data 15–25 August 2011 (f) Regional Workshop on REDATAM+SP XPLAN and Webserver Applications Development for Data Dissemination 2–12 August 2011 (g) Training Workshop – Construction of Core Environmental Indicators 6 - 10 December 2010 (h) Regional Evaluation on Strengthening the Capacity of National Statistical Offices in Caribbean Small Island Developing States to fulfil the Millennium Development Goals and other Internationally Agreed Development Goals 12 December 2011 (i) First meeting of the Steering Committee on Strengthening the capacity of National Statistical Offices in the Caribbean Small Island Developing States to fulfil the MDGs and IADGs 3 February 2009 (j) Expert Group Meeting on Development Account-supported project on MDG/IADG monitoring in the Caribbean 2–3 February 2009 (k) Second Expert Group Meeting on the Development Account-supported Project on MDG/IADG monitoring in the Caribbean 16–17 June 2009 (l) Technical Meeting on the 2010 Caribbean Millennium Development Goals Report 24–25 June 2010 (m) Training Workshop in the Use of the Statistical Programme Census Survey Processing System (CSPro) 25 October–2 November 2010 8. To what extent, do you consider the expert group meetings and or training workshops relevant to the situation in your country? (a) Very relevant (b) Relevant (c) Somewhat relevant (d) Not very relevant (e) I do not know 9. Overall, how would you rate the content of the meetings or training workshops? (a) Excellent (b) Good (c) Fair (d) Poor (e) Very Poor (f) I do not know 38 FINAL ASSESSMENT REPORT 10. Do you feel that you benefitted in any way from participation in the expert group meetings and or training workshops? Statements Strongly Agree agree (a) I am more aware of the importance of reporting on the MDGs and other IADGs (b) I am able to undertake work related to the collection of data (c) I am better positioned to analyse national data (d) I am involved in the dissemination of data through various outputs. For example, presentations and the preparation of reports. (e) I do not believe that I benefitted from the ECLAC project Neutral Disagree Strongly Not disagree applicable (N/A) 11. In the last six months, how often have you used the knowledge and skills gained in the meetings or training workshops? (a) Often (b) Sometimes (c) Seldom (d) Never (e) I do not know Section Four: Publication of the Caribbean Millennium Development Goals Report 2010 12. Are you familiar with the Caribbean MDG Report 2010? (a) Yes (b) No (If no, skip to question 19 ) 13. Did you contribute in any way to the Caribbean MDG Report 2010? (a) Yes (b) No 39 FINAL ASSESSMENT REPORT 14. How would you rate the quality of the Caribbean MDG Report 2010? (a) Excellent (b) Good (c) Fair (d) Poor (e) Very Poor (f) I do not know 15. To what extent do you consider the Caribbean MDG Report 2010 to be relevant to the situation in your country? (a) Very relevant (b) Relevant (c) Somewhat relevant (d) Not very relevant (e) Not relevant at all (f) I do not know 16. How satisfied are you with ECLAC’s support with the preparation of the Caribbean MDG Report 2010? (a) Very satisfied (b) Somewhat satisfied (c) Neither satisfied or dissatisfied (d) Somewhat dissatisfied (e) Very dissatisfied (f) I do not know Section Five: Technical Assistance 17. Did you or your institution receive technical assistance from ECLAC as part of the 2008-2011 project? (a) Yes (b) No (If no, skip to question 25) (c) I do not know 18. Do you think that the technical assistance provided was relevant to the situation in your country? (a) Very relevant (b) Relevant (c) Somewhat relevant (d) Not very relevant (e) Not relevant at all (f) I do not know 40 FINAL ASSESSMENT REPORT 19. How satisfied are you with the technical assistance that you received from ECLAC? (a) Very satisfied (b) Somewhat satisfied (c) Neither satisfied or dissatisfied (d) Somewhat dissatisfied (e) Very dissatisfied (f) I do not know 20. How useful was the technical assistance provided by ECLAC? (a) Very useful (b) It was adequate (c) Somewhat useful (d) Not useful (e) I do not know 21. Has the technical assistance provided by ECLAC contributed to any significant results in your country? Please explain your response. ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ Section Six: General Questions 22. To what extent do you agree or disagree with the following statements? Statements (a) Participation in the ECLAC project has resulted in improvements in my country’s focus on the collection of data on social, economic and environmental indicators (b) Participation in the ECLAC project has resulted in improvements in my country’s focus on the analysis of data on social, economic and environmental indicators (c) Participation in the ECLAC project has resulted in improvements in my country’s focus on the use of data on social, economic and environmental indicators (d) Participation in the ECLAC project has resulted in improvements in my country’s focus on the collection, analysis and use of data Strongly agree Agree Neutral Disagree Strongly disagree Not applicable 41 FINAL ASSESSMENT REPORT 23. Has participation in the ECLAC project, resulted in the development of any national policies geared towards the collection, analysis and use of data? (a) Yes (b) No (c) I do not know Please explain your answer ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ __________________________________________________________________________ 24. Due to your country’s involvement in the 2008-2011 ECLAC project, do you think that there have been improvements in the reporting on the MDGs and other IADGs? (a) Yes (b) No (c) I do not know Please explain your answer ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ __________________________________________________________________________ Instruction: If you selected No or I do not know for question 24, please skip to question 26. 25. Do you believe that your country has been able to sustain the improvements in reporting over the long term? (a) Yes (b) No (c) I do not know Please explain your answer ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ __________________________________________________________________________ 26. What more do you think needs to be done to build the capacity of national statistical offices in the Caribbean? Please provide as detailed a response as possible. ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ 42 FINAL ASSESSMENT REPORT 27. Has your country’s statistical office undertaken any of the following activities to promote the harmonization of data collection and analysis at a regional level? Statements (a) Developed a regional policy in support of support the harmonization of data collection on the MDGS and other IADGs (b) Developed protocols, definitions and forms to facilitate data collection at the regional level (c) Created a regional database to upload data collected on social and other indicators (d) Maintained a regional database of data on social and other indicators Yes No I do not know 28. Do you have any additional comments about the 2008-2011 ECLAC project? ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ________________________________________________________________________ Thank you! 43 FINAL ASSESSMENT REPORT Survey Two Target Group: Staff of ECLAC/Regional Commissions The United Nations Economic Commission for Latin America and the Caribbean (UN ECLAC) has commissioned an assessment of the Development Account Project ROA/122 entitled: Strengthening the Capacity of National Statistical Offices (NSOs) in the Caribbean Small Island Developing States to fulfil the Millennium Development Goals (MDGs) and other Internationally Agreed Development Goals (IADGs). The project was implemented between July 2008 and December 2011, with the main objective of strengthening the capacity of statistical offices to improve the quality of reporting on the MDGs. The activities included expert group meetings, training workshops, a publication on the Caribbean MDGs and the provision of technical assistance to project countries. Our records indicate that you participated in or contributed to the project and we are therefore seeking your support with the completion of this survey. We would like to learn more about your experiences during the project. Should you agree to participate in the assessment, it should take approximately 20 minutes to complete the survey. Any information that you provide will be kept in strict confidence, your answers will not be linked to you in any way, and your name will only be listed in the final report as one of the stakeholders who participated in the assessment. Please note that the survey link is unique to you and should not be shared with others. We would appreciate it if you could complete the survey by Midnight (Eastern Caribbean Time) on September 25, 2015.The survey findings will be used by ECLAC to improve its programming in the region. Should you have any queries please feel free to contact either evaluacion@cepal.org or Ms. Nicole Hazel - Independent Monitoring and Evaluation Consultant at nicole.hazel@gmail.com. 44 FINAL ASSESSMENT REPORT Section One: General Information 1. Please select your gender. (a) Male (b) Female 2. Please indicate which ECLAC Office you were attached to at the time of the development account project ROA 122. (a) Sub-regional office (Port of Spain) (b) Headquarters (Santiago) (c) Other (Please specify)__________________________________ 3. From the list below, please select the activities that you participated in: (a) National Training Workshop on Capacity Building for the Production of Reliable Disaggregated Data (Antigua and Barbuda) 18 – 20 October 2011 (b) Regional Workshop on the Measurement of Poverty in the Caribbean (Port of Spain) 27- 28 October 2011 (c) National Training Workshop on Capacity Building for the Collection of Data (Turks and Caicos Islands) 1 – 3 November 2011 (d) Workshop on the Construction of Environmental Statistics and Indicators 19–21 September 2011 (e) Training Workshop on the Use of the Statistical Programme Census Survey Processing Systems (CSPro) for Computer Editing of Population and Housing Census Data 15 – 25 August 2011 (f) Regional Workshop on REDATAM+SP XPLAN and Webserver Applications Development for Data Dissemination 2 – 12 August 2011 (g) Training Workshop – Construction of Core Environmental Indicators 6 - 10 December 2010 (h) Regional Evaluation on Strengthening the Capacity of National Statistical Offices in Caribbean Small Island Developing States to fulfil the MDGs and other IADGs 12 December 2011. (i) First meeting of the Steering Committee on Strengthening the Capacity of National Statistical Offices in the Caribbean Small Island Developing States to fulfil the MDGs and IADGs 3 February 2009 (j) Expert Group Meeting on Development Account-supported project on MDG/IADG monitoring in the Caribbean 2 - 3 February 2009 (k) Second Expert Group Meeting on the Development Account-supported Project on MDG/IADG monitoring in the Caribbean 16 - 17 June 2009 (l) Technical Meeting on the 2010 Caribbean Millennium Development Goals Report 24 – 25 June 2010 (m) Training Workshop in the Use of the Statistical Programme Census Survey Processing System (CSPro) 25 October – 2 November 2010 45 FINAL ASSESSMENT REPORT Section Two: General Questions 4. In your opinion, how relevant was the development account project ROA 122 to the priorities of the Member States? (a) Very relevant (b) Relevant (c) Somewhat relevant (d) Not very relevant (e) I do not know Please explain your answer _________________________________________________________________________________ _________________________________________________________________________________ _________________________________________________________________________________ _________________________________________________________________________________ 5. Did the project have established policies and procedures that contributed to its effectiveness and efficiency? (a) Yes (b) No (c) I do not know Please explain your answer _________________________________________________________________________________ _________________________________________________________________________________ _________________________________________________________________________________ _______________________________________________________________________________ 6. Which of the following statements best describes the level of collaboration between ECLAC headquarters and the sub-regional office during the execution of the development account project? (a) The level of cooperation between both offices contributed to the success of the project (b) The project was successful but this was mainly due to the input of the staff at the sub-regional office (c) The project was successful but this was mainly due to the input of the staff at headquarters (d) The level of cooperation between offices was acceptable (e) The collaboration between offices is an area that needs improvement 7. Did the project promote human rights as per the objective of development account projects? (a) Yes (b) No (c) I do not know Please explain your answer _________________________________________________________________________________ _________________________________________________________________________________ _________________________________________________________________________________ _________________________________________________________________________________ 46 FINAL ASSESSMENT REPORT 8. Was gender a key factor in the design and implementation of the project? (a) Yes (b) No (c) I do not know Please explain your answer _________________________________________________________________________________ _________________________________________________________________________________ _________________________________________________________________________________ _______________________________________________________________________________ 9. Did the project achieve the objective of strengthening the capacity of National Statistical Offices (NSOs) to report on the MDGs and other IADGs? (a) Yes (b) No (c) I do not know Please explain your answer _________________________________________________________________________________ _________________________________________________________________________________ _________________________________________________________________________________ _______________________________________________________________________________ 10. Do you believe that the majority of countries have improved the quality of reporting on the MDGs and other IADGs, because of involvement in the 2008-2011 ECLAC project? (a) Strongly agree (b) Agree (c) Neutral (d) Disagree (e) Strongly Disagree 11. How often do you use data that are collected by the NSOs? (a) Monthly (b) Every three months (c) Every six months (d) Annually (e) Not applicable 12. How often do you use reports that are prepared by the NSOs? (a) Monthly (b) Every three months (c) Every six months (d) Annually (e) Not applicable 47 FINAL ASSESSMENT REPORT 13. To what extent do you think that the following statements are valid? Statements (a) The project has resulted in the majority of countries utilising common data collection protocols, definitions and tools (b) The technical capacity of national governments has been strengthened as a direct result of the project (c) The staff of national statistical offices are better positioned to execute their duties because of the project (d) The ECLAC project has resulted in harmonized data collection and use on social, economic and environmental indicators at the regional level Strongly agree Agree Neutral Disagree Strongly disagree Not applicable 14. Since the execution of the 2008-2011 ECLAC project, have any of the Member States developed any policies geared towards the collection, analysis or use of data. Countries (a) Antigua and Barbuda (b) Bahamas (c) Barbados (d) Belize (e) Dominica (f) Dominican Republic (g) Grenada (h) Guyana (i) Haiti (j) Jamaica (k) St Kitts and Nevis (l) Saint Lucia (m) St Vincent and the Grenadines (n) Suriname (o) Trinidad and Tobago (p) Other, (Please specify) Yes No I do not know 48 FINAL ASSESSMENT REPORT 15. If you answered yes for any of the countries above, please provide details about the policy. _________________________________________________________________________________ _________________________________________________________________________________ ________________________________________________________________________________ 16. Are you aware of any positive outcomes of the project that were unexpected? (a) Yes (b) No If yes, please provide specific examples: _________________________________________________________________________________ _________________________________________________________________________________ _________________________________________________________________________________ 17. Similarly, are you aware of any negative outcomes of the project that were unexpected? (a) Yes (b) No If yes, please provide specific examples: _________________________________________________________________________________ _________________________________________________________________________________ _________________________________________________________________________________ 18. To what extent, do you believe that the project’s activities and results have been sustained by the beneficiaries? (a) To a large extent (b) To some extent (c) To a small extent (d) To a very small extent (e) Not at all (f) I do not know Please explain your answer _________________________________________________________________________________ _________________________________________________________________________________ _________________________________________________________________________________ 19. What more do you think needs to be done to build the capacity of NSOs in the Caribbean? Please provide as detailed a response as possible. _________________________________________________________________________________ _________________________________________________________________________________ _________________________________________________________________________________ 20. Do you have any additional comments about the 2008-2011 ECLAC project? _________________________________________________________________________________ _________________________________________________________________________________ _________________________________________________________________________________ Thank you! 49 FINAL ASSESSMENT REPORT Survey Three Target Group: Development Partners and Experts The United Nations Economic Commission for Latin America and the Caribbean (UN ECLAC) has commissioned an assessment of the Development Account Project ROA/122 entitled: Strengthening the Capacity of National Statistical Offices (NSOs) in the Caribbean Small Island Developing States to fulfil the Millennium Development Goals (MDGs) and other Internationally Agreed Development Goals (IADGs). The project was implemented between July 2008 and December 2011, with the main objective of strengthening the capacity of statistical offices to improve the quality of reporting on the MDGs. The activities included expert group meetings, training workshops, a publication on the Caribbean MDGs and the provision of technical assistance to project countries. Our records indicate that you participated in or contributed to the project and we are therefore seeking your support with the completion of this survey. We would like to learn more about your experiences during the project. Should you agree to participate in the assessment, it should take approximately 20 minutes to complete the survey. Any information that you provide will be kept in strict confidence, your answers will not be linked to you in any way, and your name will only be listed in the final report as one of the stakeholders who participated in the assessment. Please note that the survey link is unique to you and should not be shared with others. We would appreciate it if you could complete the survey by Midnight (Eastern Caribbean Time) on September 25, 2015.The survey findings will be used by ECLAC to improve its programming in the region. Should you have any queries please feel free to contact either evaluacion@cepal.org or Ms. Nicole Hazel - Independent Monitoring and Evaluation Consultant at nicole.hazel@gmail.com. 50 FINAL ASSESSMENT REPORT Section One: General Information and Collaboration with ECLAC 1. Please select your gender. (a) Male (b) Female 2. Please select the type of organisation that best describes your employment at the time of the ECLAC project. (a) Non-Governmental Organisation (b) Private sector (c) Research/academic institution (d) Regional organisation (e) International organisation (f) Other, please specify_____________________________________ 3. Which of the following statements best describes your involvement in the ECLAC project? Please select as many options as appropriate. (a) I participated in the in-country training (b) I participated in the regional training (c) I was involved in the provision of technical assistance to Member States (d) I participated in expert group meetings (e) I was a member of the project steering committee (f) Other, please specify________________________________________________________________________ ____________________________________________________________________________ 4. What was the nature of your organisation’s partnership with ECLAC on the 2008-2011 project? Please select as many options as appropriate. (a) Technical assistance (b) The support staff provided assistance on administrative matters (c) Financial assistance to the Member States (d) Other please specify_____________________________________ 5. How would you rate your organisation’s collaboration with ECLAC on the 2008-2011 project? (a) Excellent (b) Good (c) Fair (d) Poor (e) Very poor Please explain your answer _________________________________________________________________________________ _________________________________________________________________________________ _________________________________________________________________________________ _________________________________________________________________________________ 51 FINAL ASSESSMENT REPORT 6. In your opinion, was the project implemented in an effective manner? (a) Yes (b) No (c) I do not know Please explain your answer _________________________________________________________________________________ _________________________________________________________________________________ _________________________________________________________________________________ _________________________________________________________________________________ 7. Similarly, in your opinion was the project efficiently implemented? (a) Yes (b) No (c) I do not know Please explain your answer _________________________________________________________________________________ _________________________________________________________________________________ _________________________________________________________________________________ _________________________________________________________________________________ Section Two: General Questions 8. The project’s objective was to strengthen the technical capacities of the targeted Caribbean governments statistical offices, through networking of institutions and experts, to improve the social and environmental indicators to measure poverty, social cohesion, and environmental sustainability with a view to advancing the fulfilment of the MDGs. In your opinion, how relevant was the 2008-2011 ECLAC project to the needs of National Statistical Offices (NSOs) in the Caribbean? (a) Very relevant (b) Relevant (c) Somewhat relevant (d) Not very relevant (e) I do not know 9. In your opinion, did the 2008-2011 ECLAC project achieve the objective of improving the capacity of Caribbean NSOs? (a) Yes (b) No (c) I do not know Please explain your answer _________________________________________________________________________________ _________________________________________________________________________________ _________________________________________________________________________________ _________________________________________________________________________________ 52 FINAL ASSESSMENT REPORT 10. In your opinion, have the majority of countries improved the quality of reporting on the MDGs and other IADGs, because of involvement in the 2008-2011 ECLAC project? (a) Strongly agree (b) Agree (c) Neutral (d) Disagree (e) Strongly Disagree 11. How often do you use data that are collected by the NSOs? (a) Monthly (b) Every three months (c) Every six months (d) Annually (e) Not applicable 12. How often do you use reports that are prepared by the NSOs? (a) Monthly (b) Every three months (c) Every six months (d) Annually (e) Not applicable 13. Since the execution of the 2008-2011 ECLAC project, have any of the Member States developed any policies geared towards the collection, analysis or use of data. Countries (a) Antigua and Barbuda (b) Bahamas (c) Barbados (d) Belize (e) Dominica (f) Dominican Republic (g) Grenada (h) Guyana (i) Haiti (j) Jamaica (k) St Kitts and Nevis (l) Saint Lucia (m) St Vincent and the Grenadines (n) Suriname (o) Trinidad and Tobago (p) Other, (Please specify) Yes No I do not know 53 FINAL ASSESSMENT REPORT 14. If you answered yes for any of the countries above, please provide details about the policy. _________________________________________________________________________________ _________________________________________________________________________________ _______________________________________________________________________________ 15. Are you aware of any positive but unexpected results in the target countries that can be associated with the project? (a) Yes (b) No Please provide specific examples _________________________________________________________________________________ _________________________________________________________________________________ _________________________________________________________________________________ _______________________________________________________________________________ 16. Similarly, are you aware of any negative results in the target countries that can be associated with the project? (a) Yes (b) No Please provide specific examples _________________________________________________________________________________ _________________________________________________________________________________ _________________________________________________________________________________ _____________________________________________________________________ 17. To what extent, do you believe that the project’s activities and results have been sustained by the beneficiaries? (a) To a large extent (b) To some extent (c) To a small extent (d) To a very small extent (e) Not at all (f) I do not know Please explain your answer _________________________________________________________________________________ _________________________________________________________________________________ _________________________________________________________________________________ _______________________________________________________________________________ 54 FINAL ASSESSMENT REPORT 18. What more do you think needs to be done to build the capacity of Caribbean governments to collect, analyse and use social and economic data? Please provide as detailed a response as possible. _________________________________________________________________________________ _________________________________________________________________________________ _________________________________________________________________________________ _________________________________________________________________________________ 19. Do you have any additional comments about the 2008-2011 ECLAC project? _________________________________________________________________________________ _________________________________________________________________________________ _________________________________________________________________________________ _________________________________________________________________________________ Thank you! 55 FINAL ASSESSMENT REPORT Guide for Interviews with Staff of ECLAC/Regional Commissions Interviewee Information Sheet: The United Nations Economic Council for Latin America and the Caribbean (UN ECLAC) has commissioned an assessment of the Development Account Project ROA 122. The purpose of this project was to strengthen the capacity of National Statistical Offices (NSOs) in the Caribbean to report on the Millennium Development Goals (MDGs) and other Internationally Agreed Development Goals (IADGs). The project was conducted from July 1, 2008 to December 31, 2011 and you were identified as a potential interviewee because of your role with ECLAC and involvement with the execution of the project. We would like to obtain your opinions about the reporting on the MDGS and other IADGS in the Caribbean and the development account project ROA 122 on a whole. Should you agree to participate in an interview, it will take approximately one hour. Any information that you provide will be kept in strict confidence, your answers will not be linked to you in any way, and your name will only be listed in the final report as one of the staff members who participated in the assessment. We do hope that you will support the assessment, since the findings will used by ECLAC to improve its programming in the region. Should you have any queries please feel free to contact either Ms. Irene Barquero Tercero - Programme Officer, ECLAC Headquarters via email at Irene.BARQUERO@cepal.org or Ms. Nicole Hazel - Independent Monitoring and Evaluation Consultant at nicole.hazel@gmail.com. Project Design: Relevance 1. Do you think that the project was aligned with the priorities of governments in the respective countries? 2. Was the development account project in keeping with ECLAC’s work programme for the region? Please explain your response. 3. Did the project compliment the work programmes of other development agencies? This refers to current or past initiatives. If yes, in what ways? Project Management and Implementation: Effectiveness 4. What would you say was the main strength of the project? Similarly, what was the main weakness? 5. The main objective of the development account project ROA 122, was to strengthen the capacity of National Statistical Offices (NSOs) to improve the quality of reporting on the Millennium Development Goals (MDGs) and other Internationally Agreed Development Goals (IADGs). Do you believe that this objective has been achieved? 6. Since the completion of the project in 2011, would you say that there have been improvements in reporting on the MDGs and other IADGs? 7. Similarly, were there any specific factors that contributed to the achievement or non-achievement of the project’s objectives? Please explain your response. Note to interviewer: (Refer to page 11 of the project document). 56 FINAL ASSESSMENT REPORT 8. Did the project steering committee function as expected? Note to interviewer: (Refer to page 14 of the project document). 9. Key experts and development partners were integral to the project. Can you provide insights into their roles and contributions? 10. Were there any intended or unintended outcomes (either positive or negative) as a result of the project? Efficiency: 11. Were the project activities completed within the specified timelines? 12. Do you know if any internal protocols or guidelines were used in the design of the project? If yes, please describe the manner in which the protocols or guidelines were used. Also would I be able to obtain copies of the documentation? 13. Do you think that the available resources (specifically technical and financial) were fully utilised by ECLAC? 14. Did the project in anyway utilise (technical, financial and other resources) that were available in the Member States? 15. How efficient was the collaboration between the staff of ECLAC headquarters and the subregional office? Were any particular strategies used? What do you think could have been done differently? 16. Were any lessons learned from the development account project documented? If yes, would you be able to tell me about the main lessons learned? Also would I be able to obtain copies of the documentation? If no, can you tell me why lessons learned were not documented as part of the project? Sustainability: 17. Were sustainability issues considered as part of the project design? That is, sustainability at the level of governments and ECLAC. Please explain your response. Cross-cutting Themes 18. Did the project promote human rights as per the objective of development account projects? 19. Was gender a key factor in the design and implementation of the project? If yes, in what ways? 57 FINAL ASSESSMENT REPORT Guide for Interviews with Representatives of NSOs and Other Departments/Ministries Interviewee Information Sheet: The United Nations Economic Commission for Latin America and the Caribbean (UN ECLAC) has commissioned an assessment of the Development Account Project ROA/122 entitled: Strengthening the Capacity of National Statistical Offices (NSOs) in the Caribbean Small Island Developing States to fulfil the Millennium Development Goals (MDGs) and other Internationally Agreed Development Goals (IADGs). The project was implemented between July 2008 and December 2011, with the main objective of strengthening the capacity of statistical offices to improve the quality of reporting on the MDGs. The activities included expert group meetings, training workshops, a publication on the Caribbean MDGs and the provision of technical assistance to project countries. Our records indicate that you participated in the project, therefore we would like to interview you to learn more about your experiences. Should you agree to participate in the assessment, the interview will take approximately one hour. Any information that you provide will be kept in strict confidence, your answers will not be linked to you in any way, and your name will only be listed in the final report as one of the people who participated in the assessment. We do hope that you will support the assessment, since the findings will used by ECLAC to improve its programming in the region. Should you have any queries please feel free to contact either Ms. Irene Barquero Tercero - Programme Officer, ECLAC Headquarters via email at Irene.BARQUERO@cepal.org or Ms. Nicole Hazel - Independent Monitoring and Evaluation Consultant at nicole.hazel@gmail.com. Background Information: 1. What was your occupation at the time of the project? 2. Similarly, what type of organisation were you working with during the project? 3. Did you attend any of the project’s expert group meeting or workshops? If yes, I am going to list a series of meetings or workshops, please indicate if you attended the particular session. 58 FINAL ASSESSMENT REPORT Meeting or Workshop (a) National Training Workshop on Capacity Building for the Production of Reliable Disaggregated Data (Antigua and Barbuda) 18 – 20 October 2011 (b) Regional Workshop on the Measurement of Poverty in the Caribbean (Port of Spain) 27-28 October 2011 (c) National Training Workshop on Capacity Building for the Collection of Data (Turks and Caicos Islands) 1- 3 November 2011 (d) Workshop on the Construction of Environmental Statistics and Indicators 19 – 21 September 2011 (e) Training Workshop on the Use of the Statistical Programme Census Survey Processing Systems (CSPro) for Computer Editing of Population and Housing Census Data 15 – 25 August 2011 (f) Regional Workshop on REDATAM+SP XPLAN and Webserver Applications Development for Data Dissemination 2 – 12 August 2011 (g) Training Workshop – Construction of Core Environmental Indicators 6 - 10 December 2010 (h) Regional Evaluation on Strengthening the Capacity of National Statistical Offices in Caribbean Small Island Developing States to fulfil the MDGs and other IADGs 12 December 2011 (i) First meeting of the Steering Committee on Strengthening the Capacity of National Statistical Offices in the Caribbean Small Island Developing States to fulfil the MDGs and IADGs 3 February 2009 (j) Expert Group Meeting on Development Accountsupported Project on MDG/IADG Monitoring in the Caribbean 2 - 3 February 2009 (k) Second Expert Group Meeting on the Development Account- supported Project on MDG/IADG monitoring in the Caribbean 16 - 17 June 2009 (l) Technical Meeting on the 2010 Caribbean Millennium Development Goals Report 24 – 25 June 2010 (m) Training Workshop in the use of the Statistical Programme Census Survey Processing System (CSPro) 25 October – 2 November 2010 Yes No I do not know 4. The publication of the Caribbean Millennium Goal (MDG) Report 2010 was also completed as part of the project. Are you familiar with this publication? If yes, did you contribute in any way to the report? If no, skip to question six. 5. What do you think about the quality of the Caribbean MDG Report 2010? 6. Did you benefit in any way from the technical assistance provided by ECLAC as part of the project? If yes, could you please explain your response. 7. Do you know if your country also benefitted from technical assistance as part of the project? If yes, what was the nature of the support? 8. Are you satisfied with the level of support provided by ECLAC? 59 FINAL ASSESSMENT REPORT Relevance: 9. Do you believe that the project was in keeping with your country’s priorities? 10. Similarly, was the project in any way linked to your organisation’s work programme? 11. Earlier in the interview, you indicated that you participated in some of the project’s activities. Were these relevant to your work and professional development goals? Effectiveness: 12. Do you feel that your knowledge and skill improved as a result of the project? If yes, please explain. 13. The main objective of the project was to strengthen the capacity of NSOs to improve the quality of reporting on the MDGs and other IADGs. Do you believe that this objective was achieved in your country? 14. Are you familiar with your country’s reporting on the MDGs and other IADGs? If yes, do you think that this has improved? If there have been improvements, please explain. Also, would you attribute the improvements to the 2008-2011 ECLAC project? 15. Do you know about any outcomes (either positive or negative) that occurred as a direct result of the project? If yes, could we look at the outcomes that you are most familiar with in detail? Efficiency: 16. Think for a moment about the activities that you participated in. Were you satisfied with the ways in which these were organised? If no, is there anything that you would change about the project? 17. Also, how would you rate the overall content of the meetings or training workshops? Sustainability: 18. Do you know if your government made any formal plans to continue the activities now that the project has ended? 19. Think about the improvements that you mentioned earlier, were any of these sustained over the long term? If yes, in what ways? And if no, can you tell me why not? 20. What more do you think needs to be done to build the capacity of NSOs in the Caribbean? 21. Do you have any additional comments about the project? 60 ANNEX 2 L i s t o f K e y I n f o r m a n t s 13 Representatives of the Target Countries: 1. Ms. Sheila Roseau 2. Mr. Ramon Roach 3. Ms. Jacqueline Tull 4. Ms. Janet Geoghagen-Martin 5. Ms. Jacinta Francis 6. Ms. Carolyn Dickenson UN ECLAC Staff and Consultants: 7. Ms. Sheila Stuart 8. Ms. Karoline Schmid 9. Ms. Patricia Mendoza FINAL ASSESSMENT REPORT Antigua and Barbuda Barbados Guyana Jamaica St. Lucia Turks and Caicos Islands Sub-regional Office - Port of Spain Sub-regional Office - Port of Spain Sub-regional Office - Port of Spain 13 This list refers to the country and or organisation at the time of the ECLAC project. 61 ANNEX 3 Assessment Terms of Reference FINAL ASSESSMENT REPORT TERMS OF REFERENCE Assessment of the Development Account Project ROA 122 Strengthening the capacity of National Statistical Offices (NSOs) in the Caribbean Small Island Developing States to fulfil the Millennium Development Goals (MDGs) and other Internationally Agreed Development Goals (IADGs)” I. Introduction 1. This assessment is in accordance with the General Assembly resolutions 54/236 of December 1999 and 54/474 of April 2000, which endorsed the Regulations and Rules Governing Programme Planning, Aspects of the Budget, the Monitoring of Implementation and the Methods of Evaluation (PPBME). In this context, the General Assembly requested that programmes be evaluated on a regular, periodic basis, covering all areas of work under their purview. As part of the general strengthening of the evaluation function to support and inform the decision-making cycle in the UN Secretariat in general and ECLAC in particular and within the normative recommendations made by different oversight bodies endorsed by the General Assembly, ECLAC’s Executive Secretary is implementing an evaluation strategy that includes periodic evaluations of different areas of ECLAC’s work. This is therefore a discretionary internal evaluation managed by the Programme Planning and Evaluation Unit (PPEU) of ECLAC’s Programme Planning and Operations division (PPOD). II. Assessment Topic 2. This assessment is an end-of-cycle review of a sub-regional project on Strengthening the capacity of National Statistical Offices (NSOs) in the Caribbean Small Island Developing States to fulfill the Millennium Development Goals (MDGs) and other Internationally Agreed Development Goals (IADGs)” that was implemented in the Caribbean region between 2008 and 2011. III. Objective of the Assessment 3. The objective of this assessment is to review the efficiency, effectiveness, relevance, and sustainability of the project implementation and, more particularly, document the results and impact of the project attained in relation to its overall objectives and expected results, as defined in the project document. 4. The assessment will place an important emphasis in identifying lessons learned and good practices that were derived from the implementation of the project, its sustainability and the potential of replicating the approaches within the region and across different regions, i.e. transferability to Latin and Central America. The lessons learned and good practices in actual project implementation will in turn be used as tools for the future planning and implementation of ECLAC projects. 62 FINAL ASSESSMENT REPORT 5. The overall objective of the project is included in its title: Strengthening the technical capacities of the targeted Caribbean governments statistical offices, through networking of institutions and experts, to improve the social and environmental indicators to measure poverty, social cohesion, and environmental sustainability with a view to advancing the fulfilment of MDGs. 6. The expected accomplishments of the project were: 1. Adoption by Caribbean countries of common data collection protocols, definitions and classifications for the collection and dissemination of an extended set of MDG indicators that are more suitable to their situation, in order to monitor the accomplishment of the MDGs. 2. Strengthened technical capacity of governmental institutions to produce and analyse MDG indicators derived from harmonized household survey data sets and use them in the design and implementation of public policies that lead to the fulfilment of the goals. 3. Improved capacity in the NSOs of participating countries to collect, compile, analyse and disseminate data on social and environmental indicators to measure poverty, social cohesion and environmental sustainability. 4. Establishment of a database which makes it possible to draw up indicators at the sub-regional level in order to measure the economic, social and environmental costs of maintaining current development patterns; evaluate and diagnose the current state of the environment in the region; gauge the environmental consequences of existing macroeconomic policies; and supply orientation for policy decisions, especially in regard to public and private investment. 7. To achieve the objective and accomplishments above, the key project activities included expert group meetings, four regional training workshops, six national training workshops and a publication on the status of MDG achievements in the Caribbean. 8. Annex 1 to these Terms of Reference includes the full project outline. IV. Background The Development Account 9. The Development Account (DA) was established by the General Assembly in 1997, as a mechanism to fund capacity development projects of the economic and social entities of the United Nations (UN). By building capacity on three levels, namely: (i) the individual; (ii) the organizational; and (iii) the enabling environment, the DA becomes a supportive vehicle for advancing the implementation of internationally agreed development goals (IADGs) and the outcomes of the UN conferences and summits. The DA adopts a medium to long-term approach in helping countries to better integrate social, economic and environmental policies and strategies in order to achieve inclusive and sustained economic growth, poverty eradication, and sustainable development. 10. Projects financed from the DA aim at achieving development impact by building the socio-economic capacity of developing countries through collaboration at the national, sub-regional, regional and interregional levels. The DA provides a mechanism for promoting the exchange and transfer of skills, knowledge and good practices among target countries within and between different geographic regions, and through the cooperation with a wide range of partners in the broader development assistance community. It provides a bridge between in-country capacity development actors, on the one hand, and UN Secretariat entities, on the other. The latter offer distinctive skills and competencies in a broad range of economic, environmental and social issues that are often only marginally dealt with by other development partners at country level. For target countries, the DA provides a vehicle to tap into the 63 FINAL ASSESSMENT REPORT normative and analytical expertise of the UN Secretariat and receive on-going policy support on economic, environmental and social issues, particularly in areas where such expertise does not reside in the capacities of the UN country teams. 11. The DAs operational profile is further reinforced by the adoption of pilot approaches that test new ideas and eventually scale them up through supplementary funding, and the emphasis on integration of national expertise in the projects to ensure national ownership and sustainability of project outcomes. 12. DA projects are being implemented by global and regional entities, cover all regions of the globe and focus on five thematic clusters14. Projects are programmed in tranches, which represent the Accounts programming cycle. The DA is funded from the Secretariats regular budget and the Economic Commission for Latin America and the Caribbean (ECLAC) is one of its ten implementing entities. The UN Department of Economic and Social Affairs (DESA) provides overall management of the DA portfolio. 13. ECLAC undertakes internal assessments of each of its DA projects in accordance with DA requirements. Assessments are defined by ECLAC as brief end-of-project evaluation exercises aimed at assessing the relevance, efficiency, effectiveness and sustainability of project activities. They are undertaken as desk studies and consist of a document review, stakeholder survey, and a limited number of telephone-based interviews. The project 14. The project under evaluation is part of the projects approved under the DA for the 2008-2009 tranche, under the coordination of the Economic Commission for Latin America and the Caribbean (ECLAC), Sub-regional Headquarters in the Caribbean. 15. The original duration of this project was of two years (2008-2010). Having started activities in March 2009, it was extended until December 2011. In 2009 and 2010 interim reports for the project were produced and in January 2012 a final report was presented. It is important to understand that both the final report and the interim reports are to be taken into account in the evaluation since they will outline some of the difficulties faced in the project implementation. 16. Substantively, the project outline states that: “In spite of various initiatives, Caribbean countries continue to have difficulties in addressing additional demands of monitoring and measuring progress created by the MDGs and other global commitments. Therefore, it is imperative to carry out activities to ensure the further building/strengthening of institutional capabilities for generating reliable social and economic statistics among the various Caribbean States. This project seeks to build on past and current initiatives directed towards broadening and improving statistics and other indicators through the better use of experience and accumulated documentation, technical assistance, and through other projects and programmes. The project will be executed by ECLAC, through its Sub-regional Headquarters for the Caribbean and in close collaboration with its Statistics and Economic Projections Division in Santiago, Chile. Other cooperating agencies in the project are the Statistics Division of the Department of Economic and Social Affairs (DESA/UNSD); the United Nations Development Programme (UNDP) and the Caribbean Community (CARICOM) Secretariat. Over a two-year period, the outputs of the project will benefit the NSOs and agencies from 15 member countries. ECLAC will be the lead agency managing the project with a Steering Committee overseeing overall project implementation. The total budget of the project amounts to US$528,000 and the project activities will be executed over the biennium 2008-2009.” 14 Development Account projects are implemented in the following thematic areas: advancement of women; population/ countries in special needs; drug and crime prevention; environment and natural resources; governance and institution building; macroeconomic analysis, finance and external debt; science and technology for development; social development and social integration; statistics; sustainable development and human settlement; and trade. See also UN Development Account website: http://www.un.org/esa/devaccount/projects/active/theme.html 64 FINAL ASSESSMENT REPORT 17. The project objective, accomplishments and indicators were derived to address the demands of MDG and other global commitments. Hence, one key part of this evaluation will be to assess the degree to which these difficulties were overcome by implementing the project. 18. The overall logic of the project against which results and impact will be assessed contains an overall objective and a set of expected accomplishments and indicators of achievement that will be used as signposts to assess its effectiveness and relevance. Stakeholder Analysis 19. The outputs delivered benefitted the participating countries listed below. The meeting reports name the participants individually; contact details will be provided to the consultant by the ECLAC Sub-regional Headquarters for the Caribbean: • Anguilla: Construction of Core Environmental Indicators, Second Expert Group Meeting on MDG Monitoring • Antigua and Barbuda: National Workshop on Disaggregated Data, Regional Workshop on the Measurement of Poverty, Construction of Core Environmental Indicators, First Expert Group Meeting on MDG Monitoring, Second Expert Group Meeting on MDG Monitoring • Aruba: REDATAM Workshop, Second Expert Group Meeting on MDG Monitoring • Bahamas: Regional Workshop on the Measurement of Poverty, Construction of Core Environmental Indicators, First Expert Group Meeting on MDG Monitoring, Second Expert Group Meeting on MDG Monitoring • Barbados: Regional Workshop on the Measurement of Poverty, Construction of Core Environmental Indicators, Second Expert Group Meeting on MDG Monitoring • Belize: Regional Workshop on the Measurement of Poverty, REDATAM Workshop, Construction of Core Environmental Indicators, Second Expert Group Meeting on MDG Monitoring, Technical Meeting on the 2010 Caribbean Millennium Development Goals Report • British Virgin Islands: REDATAM Workshop, Second Expert Group Meeting on MDG Monitoring • Cayman Islands: REDATAM Workshop, Construction of Core Environmental Indicators • Curacao: First Expert Group Meeting on MDG Monitoring, Second Expert Group Meeting on MDG Monitoring • Dominica: Regional Workshop on the Measurement of Poverty, Construction of Core Environmental Indicators, First Expert Group Meeting on MDG Monitoring • Grenada: Regional Workshop on the Measurement of Poverty, REDATAM Workshop, Construction of Core Environmental Indicators, First Expert Group Meeting on MDG Monitoring, Second Expert Group Meeting on MDG Monitoring • Guyana: Regional Workshop on the Measurement of Poverty, REDATAM Workshop, Construction of Core Environmental Indicators, Second Expert Group Meeting on MDG Monitoring, Training Workshop in the use of the Statistical Programme Census Survey Processing System (CSPro) 65 FINAL ASSESSMENT REPORT • Jamaica: Regional Workshop on the Measurement of Poverty, REDATAM Workshop, Construction of Core Environmental Indicators, First meeting of the Steering Committee, First Expert Group Meeting on MDG Monitoring, Second Expert Group Meeting on MDG Monitoring, Technical Meeting on the 2010 Caribbean Millennium Development Goals Report • Montserrat: Construction of Core Environmental Indicators • Saint Kitts and Nevis: Regional Workshop on the Measurement of Poverty, REDATAM Workshop, Construction of Core Environmental Indicators, First Expert Group Meeting on MDG Monitoring, Second Expert Group Meeting on MDG Monitoring • Saint Lucia: Regional Workshop on the Measurement of Poverty, REDATAM Workshop, Construction of Core Environmental Indicators, First Expert Group Meeting on MDG Monitoring, Second Expert Group Meeting on MDG Monitoring • Saint Vincent and the Grenadines: REDATAM Workshop, Construction of Core Environmental Indicators, First Expert Group Meeting on MDG Monitoring, Second Expert Group Meeting on MDG Monitoring • Suriname: Regional Workshop on the Measurement of Poverty, REDATAM Workshop, Construction of Core Environmental Indicators, First Expert Group Meeting on MDG Monitoring, Second Expert Group Meeting on MDG Monitoring, Technical Meeting on the 2010 Caribbean Millennium Development Goals Report • Trinidad and Tobago: Regional Workshop on the Measurement of Poverty, REDATAM Workshop, Construction of Core Environmental Indicators, First meeting of the Steering Committee, First Expert Group Meeting on MDG Monitoring, Second Expert Group Meeting on MDG Monitoring • Turks and Caicos Islands: National Workshop on Collection of Data • United States Virgin Islands: First Expert Group Meeting on MDG Monitoring, Second Expert Group Meeting on MDG Monitoring V. Guiding Principles 20. The assessment will seek to be independent, credible and useful and adhere to the highest possible professional standards. It will be consultative and engage the participation of a broad range of stakeholders. The unit of analysis is the project itself, including its design, implementation and effects. The assessment will be undertaken in accordance with the provisions contained in the Project Document. The assessment will be conducted in line with the norms, standards and ethical principles of the United Nations Evaluation Group (UNEG)15. 21. Although this exercise should not be considered a fully-fledged evaluation (e.g. less extensive data collection and analysis involved, less evaluation criteria considered, etc.), it is expected that the evaluator will apply ECLAC’s guiding principles to the evaluation process.16 In particular, special consideration will be taken to assess the extent to which ECLAC’s activities and outputs respected and promoted human rights17. 15 Standards for Evaluation in the UN System, UNEG, April 2005, http://www.unevaluation.org/ document/detail/22; Norms for Evaluation in the UN System, UNEG, April 2005, http://www.uneval.org/document/detail/21; UNEG Ethical Guidelines for Evaluation, UNEG, March 2008, http://www.unevaluation.org/document/detail/102. 16 See ECLAC, “Preparing and Conducting Evaluations: ECLAC Guidelines” (2009) and ECLAC, “Evaluation Policy and Strategy”(2014) for a full description of its guiding principles. 17 For further reference see UNEG “Integrating Human Rights and Gender Equality in Evaluations” (2014). 66 FINAL ASSESSMENT REPORT This includes a consideration of whether ECLAC interventions treated beneficiaries as equals, safeguarded and promoted the rights of minorities, and helped to empower civil society. 22. The assessment will also examine the extent to which gender concerns were incorporated into the project – whether project design and implementation incorporated the needs and priorities of women, whether women were treated as equal players, and whether it served to promote women’s empowerment. 23. Moreover, the evaluation process itself, including the design, data collection, and dissemination of the evaluation report, will be carried out in alignment with these principles. 24. Evaluators are also expected to respect UNEG’s ethical principles as per its “Ethical Guidelines for Evaluation”18: • Independence: Evaluators shall ensure that independence of judgment is maintained and that evaluation findings and recommendations are independently presented. • Impartiality: Evaluators shall operate in an impartial and unbiased manner and give a balanced presentation of strengths and weaknesses of the policy, program, project or organizational unit being evaluated. • Conflict of Interest: Evaluators are required to disclose in writing any past experience, which may give rise to a potential conflict of interest, and to deal honestly in resolving any conflict of interest which may arise. • Honesty and Integrity: Evaluators shall show honesty and integrity in their own behavior, negotiating honestly the evaluation costs, tasks, limitations, scope of results likely to be obtained, while accurately presenting their procedures, data and findings and highlighting any limitations or uncertainties of interpretation within the evaluation. • Competence: Evaluators shall accurately represent their level of skills and knowledge and work only within the limits of their professional training and abilities in evaluation, declining assignments for which they do not have the skills and experience to complete successfully. • Accountability: Evaluators are accountable for the completion of the agreed evaluation deliverables within the timeframe and budget agreed, while operating in a cost effective manner. • Obligations to Participants: Evaluators shall respect and protect the rights and welfare of human subjects and communities, in accordance with the UN Universal Declaration of Human Rights and other human rights conventions. Evaluators shall respect differences in culture, local customs, religious beliefs and practices, personal interaction, gender roles, disability, age and ethnicity, while using evaluation instruments appropriate to the cultural setting. Evaluators shall ensure prospective participants are treated as autonomous agents, free to choose whether to participate in the evaluation, while ensuring that the relatively powerless are represented. • Confidentiality: Evaluators shall respect people’s right to provide information in confidence and make participants aware of the scope and limits of confidentiality, while ensuring that sensitive information cannot be traced to its source. • Avoidance of Harm: Evaluators shall act to minimize risks and harms to, and burdens on, those participating in the evaluation, without compromising the integrity of the evaluation findings. 18 UNEG Ethical Guidelines for Evaluation, UNEG, March 2008 (http://www.unevaluation.org/ethicalguidelines). 67 FINAL ASSESSMENT REPORT • Accuracy, Completeness and Reliability: Evaluators have an obligation to ensure that evaluation reports and presentations are accurate, complete and reliable. Evaluators shall explicitly justify judgments, findings and conclusions and show their underlying rationale, so that stakeholders are in a position to assess them. • Transparency: Evaluators shall clearly communicate to stakeholders the purpose of the evaluation, the criteria applied and the intended use of findings. Evaluators shall ensure that stakeholders have a say in shaping the evaluation and shall ensure that all documentation is readily available to and understood by stakeholders. • Omissions and wrongdoing: Where evaluators find evidence of wrong-doing or unethical conduct, they are obliged to report it to the proper oversight authority. VI. Scope of the assessment 25. In line with the assessment objective, the scope of the assessment will more specifically cover all the activities implemented by the project. The assessment will review the benefits that the various stakeholders in participating countries have had, how their approach towards measuring MDG’s was changed and intensified by participating in the activities and how sustainable the project interventions were. The assessment will also assess and review the interaction and coordination modalities used in its implementation within ECLAC and other implementing partners, participating in the implementation of the project. 26. In summary, the elements to be covered in the assessment include: • Actual progress made towards project objectives • The extent to which the project has contributed to outcomes in the identified countries whether intended or unintended. • The efficiency with which outputs were delivered. • The strengths and weaknesses of project implementation on the basis of the available elements of the logical framework (objectives, results, etc.) contained in the project document • The validity of the strategy and partnership arrangements. Coordination among the different Regional Commissions. • The extent to which the project was designed and implemented to facilitate the attainment of the goals. • Relevance of the project’s activities and outputs towards the needs of Member States. 27. It will also assess various aspects related to the way the project met the following Development Account criteria: • Result in durable, self-sustaining initiatives to develop national capacities, with measurable impact at field level, ideally having multiplier effects; • Be innovative and take advantage of information and communication technology, knowledge management and networking of expertise at the sub regional, regional and global levels; • Utilize the technical, human and other resources available in developing countries and effectively draw on the existing knowledge/skills/capacity within the UN Secretariat; • Create synergies with other development interventions and benefit from partnerships with non-UN stakeholders. 68 FINAL ASSESSMENT REPORT VII. Methodology 28. The assessments will use the following data collection methods to assess the impact of the work of the project: (a) Desk review and secondary data collection analysis, of DA project criteria, the project document, annual reports of advance, workshops and meetings reports and evaluation surveys, other project documentation such as project methodology, country reports, consolidated report, webpage, etc. The available documents include the Project Document itself, Progress Reports for 2008-2010 and the individual meeting reports as outlined in the table above. Especially, the report of the project evaluation workshop of December 2011 should be taken into account as well. The desk review should also include, to the extent possible, a review of beneficiaries and Member States MDG reports from various years and analysis of improvements thereof in case of project outputs being implemented. The following reports on the project meetings and workshops are available: Title of activity Date of activity Link to report Evaluation Report of the National Training Workshop on Capacity-Building for the Production of Reliable Disaggregated Data (Antigua and Barbuda) 18 – 20 October 2011 http://www.eclac.org/portofspain/ publicaciones/xml/6/45076/lcarl3 61.pdf Evaluation Report of the Regional Workshop on the Measurement of Poverty in the Caribbean (Port of Spain) 27- 28 October 2011 http://www.eclac.org/portofspain/ publicaciones/xml/8/45138/lcarl3 63.pdf Evaluation Report of the National Training Workshop 1 –3 November on Capacity-building for the Collection of Data (Turks 2011 and Caicos Islands) http://www.eclac.org/portofspain/ publicaciones/xml/1/45151/lcarl3 64.pdf Report of National Workshop on the Construction of 19 – 21 September Environmental Statistics and Indicators 2011 http://www.eclac.org/portofspain/ publicaciones/xml/8/45048/lcarl3 35.pdf Evaluation report of the Training Workshop on the Use of the Statistical Programme Census Survey Processing Systems (CSPro) Computer editing of Population and Housing Census Data 15 – 25 August 2011 http://www.eclac.org/publicaciones /xml/0/44760/lcarl333.pdf Report of the regional Workshop on REDATAM+SP XPLAN and Webserver Applications Development for data dissemination 2 – 12 August 2011 http://www.eclac.org/publicaciones /xml/8/44598/LCARL.305.pdf Evaluation Report of the Training Workshop – Construction of Core Environmental Indicators 6 - 10 December 2010 http://www.eclac.org/publicaciones /xml/0/42120/LCARL.276.pdf 69 FINAL ASSESSMENT REPORT Report of the Regional Evaluation on Strengthening the Capacity of National Statistical Offices in Caribbean Small Island Developing States to fulfil the Millennium Development Goals and other Internationally-agreed Development Goals 12 December 2011 Report of the First meeting of the Steering Committee on Strengthening the capacity of National Statistical Offices in the Caribbean Small Island Developing States to fulfil the MDGs and IADGs 3 February 2009 Report of the Expert Group Meeting on development account supported project on MDG/IADG monitoring in the Caribbean 2-3 February 2009 Evaluation report on Expert Group Meeting on Development Account-supported project on MDG/IADG monitoring in the Caribbean 2-3 February 2009 Evaluation report on Expert Group Meeting on Development Account-supported project on MDG/IADG monitoring in the Caribbean 16-17 June 2009 Report of the Second Expert Group Meeting on the Development Account Supported Project on MDG/IADG monitoring in the Caribbean 16-17 June 2009 Report of the Technical Meeting on the 2010 Caribbean Millennium Development Goals Report 24 – 25 June 2010 Evaluation of the Technical Meeting on the 2010 Caribbean Millennium Development Goals Report 24 – 25 June 2010 Evaluation report of the Training Workshop in the use of the Statistical Programme Census Survey Processing System (CSPro) 25 October–2 November 2010 Document will be LC/CAR/L.373, to be published on eclacpos.org shortly. Please check http://www.eclac.org/cgibin/getProd.asp?xml=/portofspain /agrupadores_xml/aes825.xmlxsl =/portofspain/agrupadores_xml/ agrupa_listado.xslbase=/portofs pain/tpl/top-bottom.xsl http://www.eclac.org/publicaciones /xml/2/35612/LCARL.192.pdf http://www.eclac.org/publicaciones /xml/1/35611/LCARL.191.pdf http://www.eclac.org/publicaciones /xml/7/36737/LCARL.202.pdf http://www.eclac.org/publicaciones /xml/1/36741/LCARL.203.pdf http://www.eclac.org/publicaciones /xml/4/37084/LCARL.212.pdf http://www.eclac.org/publicaciones /xml/1/40731/LCARL.269.pdf http://www.eclac.org/publicaciones /xml/2/40732/LCARL.270.pdf http://www.eclac.org/publicaciones /xml/9/41829/LCARL.273.pdf The following publication was published under the project: Caribbean Millennium Development Goals Report 2010 (http://www.eclac.org/publicaciones/xml/2/45442/LCARL.371.pdf) (b) Self-administered surveys: At least three types of surveys will be used: a) Surveys to beneficiaries and Member States in the five regions; b) Surveys to Regional Commission’s staff involved in the project, and c) Survey to partners and stakeholders within the United Nations and the countries from the five regions participating in the project. PPEU will provide support to manage the online surveys through Survey Monkey. PPEU will distribute the surveys among project beneficiaries to the revised lists facilitated by the consultant. PPEU will finally provide the evaluator with the consolidated responses. 70 FINAL ASSESSMENT REPORT (c) Semi-structured interviews and focus groups to validate and triangulate information and findings from the surveys and the document reviews, a limited number of interviews (structured, semi-structured, in-depth, key informant, focus group, etc.) may be carried out via tele- or video-conference with project partners to capture the perspectives of managers, beneficiaries, participating ministries, departments and agencies, etc. PPEU will provide assistance to coordinate the interviews, including initial contact with beneficiaries to present the assessment and the evaluator. Following this presentation, the evaluator will directly arrange the interviews with available beneficiaries and project managers. 29. Methodological triangulation is an underlying principle of the approach chosen. Suitable frameworks for analysis and evaluation are to be elaborated – based on the questions to be answered. The experts will identify and set out the methods and frameworks as part of the inception report. VIII. Evaluation Issues/ Questions 30. This assessment encompasses the different stages of the given project, including its design, process, results, and impact, and is structured around four main criteria: relevance, efficiency, effectiveness, and sustainability. Within each of these criteria, a set of evaluation questions will be applied to guide the analysis19. The responses to these questions are intended to explain “the extent to which,” “why,” and “how” specific outcomes were attained. 31. The questions included hereafter are intended to serve as a basis for the final set of evaluation questions, to be adapted by the evaluator and presented in the inception report. Efficiency (a) Collaboration and coordination mechanisms between and within ECLAC and the Sub-regional Headquarters in the Caribbean that ensure efficiencies and coherence of response; (b) Provision of services and support in a timely and reliable manner, according to the priorities established by the project document; (c) Presence of protocols and practices to ascertain that good practices and lessons learned are recognized and integrated into work practices. Effectiveness (a) How satisfied are the project’s main clients with the services they received? (b) What are the results identified by the beneficiaries? (c) Has the project made any difference in the behavior/attitude/skills/ performance of the clients? (d) How effective were the project activities in enabling capacities and influencing policy making? (e) Are there any tangible policies that have considered the contributions provided by the ECLAC in relation to the project under evaluation? (f) How much more knowledgeable are the participants in workshops and seminars? Relevance: (a) How in line were the activities and outputs delivered with the priorities of the targeted countries? (b) How aligned was the proposed programme of work with the sub-programme activities? (c) Were there any complementarities and synergies with the other work being developed? 19 The questions included here will serve as a basis for the final set of evaluation questions, to be adapted by the evaluator and presented in the inception report. 71 FINAL ASSESSMENT REPORT Sustainability: With beneficiaries: (a) How did the project utilise the technical, human and other resources available in developing countries? (b) How have the programme’s main results and recommendations been used or incorporated in the work and practices of beneficiary institutions after completion of the project’s activities? What were the multiplier effects generated by the programme? (c) What mechanisms were set up to ensure the follow-up of networks created under the project? Within the Regional Commissions: (a) How has the programme contributed to shaping / enhancing the ECLAC programmes of work/priorities and activities? The work modalities and the type of activities carried out? How has ECLAC built on the findings of the project? IX. Deliverables 32. The assessment will include the following outputs: Work Plan. No later than five days after the signature of the contract, the consultant must deliver to PPOD a detailed Work Plan of all the activities to be carried out related to the assessment of project ROA/122, schedule of activities and outputs detailing the methodology to be used, etc. Inception Report. No later than four weeks after the signature of the contract, the consultant should deliver the inception report, which should include the background of the project, an analysis of the Project profile and implementation and a full review of all related documentation as well as project implementation reports. Additionally, the inception report should include a detailed evaluation methodology including the description of the types of data collection instruments that will be used and a full analysis of the stakeholders and partners that will be contacted to obtain the evaluation information. First drafts of the instruments to be used for the survey, focus groups and interviews should also be included in this first report. Draft final evaluation Report. No later than 12 weeks after the signature of the contract, the consultant should deliver the preliminary report for revision and comments by PPOD which should include the main draft results and findings of the evaluation, lessons learned and recommendations derived from it, including its sustainability, and potential improvements in project management and coordination of similar DA projects. Final Evaluation Report. No later than 16 weeks after the signature of the contract, the consultant should deliver the final evaluation report which should include the revised version of the preliminary version after making sure all the comments and observations from PPOD and the Evaluation Reference Group have been included. Before submitting the final report, the consultant must have received the clearance on this final version from PPOD, assuring the satisfaction of ECLAC with the final evaluation report. Presentation of the results of the evaluation. A final presentation of the main results of the evaluation to ECLAC will be delivered at the same time of the delivery of the final evaluation report. 72 FINAL ASSESSMENT REPORT X. Payment schedule and conditions 33. The duration of the consultancy will be initially for 16 weeks during the months of June-September 2015. The consultant will be reporting to and be managed by the Programme Planning and Evaluation Unit (PPEU) of the Programme Planning and Operations Division (PPOD) of ECLAC. Coordination and support to the evaluation activities will be provided by the Sub-regional Headquarters in the Caribbean. 34. The contract will include the payment for the services of the consultant as well as all the related expenses of the evaluation. Payments will be done according to the following schedule and conditions: 30% of the total value of the contract will be paid against the satisfactory delivery of the inception report which should be delivered as per the above deadlines. 30% of the total value of the contract will be paid against the satisfactory delivery of the draft final evaluation report which should be delivered as per the above deadlines. 40% of the total value of the contract will be paid against the satisfactory delivery and presentation of the Final Evaluation Report which should be delivered as per the above deadlines. 35. All payments will be done only after the approval of each progress report and the final report from the Programme Planning and Evaluation Unit (PPEU) of the Programme Planning and Operations Division (PPOD) of ECLAC. XI. Profile of the Evaluator 36. The evaluator will have the following characteristics: Education x MA in political science, public policy, development studies, sociology economics, business administration, or a related social science. Experience x At least seven years of progressively responsible relevant experience in programme/project evaluation are required. x Proven competency in quantitative and qualitative research methods, particularly selfadministered surveys, document analysis, and informal and semi-structured interviews is required. x At least two years of experience in areas related to the measurement and evaluation of the Millennium Development Goals or other Internationally Agreed Development Goals is required. x Experience in at least three evaluations with international (development) organizations is required. Experience in Regional Commissions and United Nations projects, especially Development Account projects is highly desirable. x Working experience in the Caribbean is desirable. Language Requirements x Proficiency in English is required. 73 FINAL ASSESSMENT REPORT XII. Roles and responsibilities in the evaluation process 37. Commissioner of the evaluation Î (ECLAC Executive Secretary and PPOD Director) x Mandates the evaluation x Provides the funds to undertake the evaluation x Safeguards the independence of the evaluation process 38. Task manager Î (PPEU Evaluation Team) x Drafts evaluation TORs x Recruits the evaluator/evaluation team x Shares relevant information and documentation and provides strategic guidance to the evaluator/evaluation team x Provides overall management of the evaluation and its budget, including administrative and logistical support in the methodological process and organization of evaluation missions x Coordinates communication between the evaluator/evaluation team, implementing partners and the ERG, and convenes meetings x Supports the evaluator/evaluation team in the data collection process x Reviews key evaluation deliverables for quality and robustness and facilitates the overall quality assurance process for the evaluation x Manages the editing, dissemination and communication of the evaluation report x Implements the evaluation follow-up process 39. Evaluator/Evaluation team Î (External consultant) x Undertakes the desk review, designs the evaluation methodology and prepares the inception report x Conducts the data collection process, including the design of the electronic survey and semi-structured interviews x Carries out the data analysis x Drafts the evaluation report and undertakes revisions 40. Evaluation Reference Group (ERG) Î (Composed of representatives of each of the implementing partners) x Provides feedback to the evaluator/evaluation team on preliminary evaluation findings and final conclusions and recommendations x Reviews draft evaluation report for robustness of evidence and factual accuracy XIII. Other Issues 41. Intellectual property rights. The consultant is obliged to cede to ECLAC all authors rights, patents and any other intellectual property rights for all the work, reports, final products and materials resulting from the design and implementation of this consultancy, in the cases where these rights are applicable. The consultant will not be allowed to use, nor provide or disseminate part of these products and reports or its total to third parties without previously obtaining written permission from ECLAC 74 FINAL ASSESSMENT REPORT 42. Coordination arrangements. The evaluation team comprised of the consultant and the staff of the Programme Planning and Evaluation Unit of ECLAC will confer and coordinate activities on an on-going basis, ensuring a bi-monthly coordination meeting/teleconference to ensure the project is on track and that immediate urgencies and problems are dealt with in a timely manner. If any difficulty or problem develops in the interim the evaluation team member will raise it immediately with the rest of the team so that immediate solutions can be explored and decisions taken. XIV. Evaluation use and dissemination 43. This evaluation seeks to identify best practices and lessons learned in the implementation of development account projects and specifically the capacity of the countries to regularly and appropriately measure violence against women. The evaluation findings will be presented and discussed to ECLAC. An Action Plan will be developed to implement recommendations when appropriate in future development account projects. The evaluation report will also be circulated through regional commissions’ intranet (and other knowledge management tools), including circulating a final copy to DESA, as the programme manager for the Development Account, so as to constitute a learning tool in the organisation. 75 FINAL ASSESSMENT REPORT ANNEX 4 Evaluator’s revision matrix A. COMMENTS ERG SPECIFIC COMMENTS PARAGRAPH NUMBER Page 10 Page 14 Page 14 COMMENTS ERG EVALUATOR’S RESPONSE “Five ECLCAC staff members were selected as possible interviewees because of their roles as coordinators of the project and technical experts….” This was a typing error and the change was made to ECLAC. ECLAC is misspelled “Finally, the data was triangulated through multiple sources including the provision of feedback by the ELCAC Task Manager and the Evaluation Reference Group.” This was a typing error and the change was made to ECLAC. ECLAC is misspelled “The surveys were sent out by ELCAC and as noted earlier this could assist with the recognition of the exercise by potential participants….” This was a typing error and the change was made to ECLAC. ECLAC is misspelled B. COMMENTS PPOD GENERAL COMMENTS PARAGRAPH NUMBER General Project background, page 5 Evaluation objectives and scope Page 5 Assessment Methodology Page 6 PARAGRAPH NUMBER PARAGRAPH NUMBER Please number each paragraph, finding, recommendation and lesson learned. Please include information on the activities actually implemented by the project, if different from those originally planned. Please also highlight in this section (even if it is mentioned afterwards) that the evaluation also included an evaluation of cross-cutting issues such as the incorporation of human rights and gender perspectives in the project Please mention in this section that the evaluation methodology was developed and implemented taking into consideration the incorporation of human rights and gender perspectives. The change was made as requested. The text was revised to reflect the activities that were completed as planned and those that were not as successful. Apart from the non-completion of the information system all other activities were conducted as outlined on page 12 and 13 of the project document. The change was made as requested. Please refer to the third paragraph under the heading assessment objectives. This was already included as part of the assessment methodology. Please refer to the last page (16 of the draft final report/please note that page numbers may change in the final version) for the section that has the heading assessment principles and ethical issues. 76 FINAL ASSESSMENT REPORT Conclusions Lessons learnt and recommendations Make sure to include reference to the incorporation of Gender perspective by the project in the conclusions section, and if consider relevant in the recommendations or lesson learned. We would appreciate if you could further develop and deepen the analysis in both the recommendations and lessons learned sections to increase its usefulness for our continuous learning and improvements processes. The cross cutting themes are discussed in the findings (page 37 of the draft final report/please note that page numbers may change in the final version). The conclusions, recommendations and lessons learned were updated to reflect the finding on the cross cutting themes of gender and human rights. I reviewed both sections and made the necessary adjustments. Please number each lesson learnt in the report The lessons learned were numbered (lesson learned 1, lesson learned 2...). as requested. Each recommendation should also be numbered and should include a title, summarizing the recommendation, information on the findings and conclusions that support it and to whom is the recommendation addressed, providing some clear examples on how the recommendation could be implemented (more specific actions). SPECIFIC COMMENTS The section was reviewed, numbered, titles included and information on the findings/conclusions supporting the recommendations were included. Also, the recommendations were addressed to specific entities and where possible examples on how they could be implemented were included. PARAGRAPH NUMBER Page 21 Paragraph 2 PARAGRAPH NUMBER PARAGRAPH NUMBER In the following paragraph, there seems to be a mix between relevance (data from surveys) and effectiveness (reported improvements in knowledge and skills from the interviews). The assessment also determined the linkage between the capacity building sessions and the situation in the respective countries. The responses to this survey question were favourable - 31 out of 35 people selected very relevant or somewhat relevant (Refer to figure five). The interviewees also provided insights into this finding – they reported improvements in knowledge and skills and the ability to apply what they learnt to support national programmes and international reporting requirements. Please revise the text. I agree that there is a mix between relevance and effectiveness in the paragraph. This was done to substantiate the survey findings. But at your request I have deleted the highlighted text because of a lack of additional data with which to make the revision. 77 FINAL ASSESSMENT REPORT Page 39 Effectiveness First paragraph, first line and its related lesson learned (4) In this section, it is stated as a conclusion that “It appears that all of the activities were not in keeping with the priorities of the target countries and by extension the national focal points”, deriving in what we understand is a related lesson learned stating that Activities should be geared towards the needs and interests of participants”. We have, however, not found the data from where this conclusion and lesson learned are derived from and furthermore think that it somehow contradicts the following information presented in the relevance section of the findings: “The three groups (national representatives, ECLAC staff/consultants and development partners/ experts) viewed the development account project as pertinent to the needs and priorities of NSOs and governments. Forty five out of the 49 representatives from the target countries reported that the project was either very relevant or relevant (Refer to figure one below). The six interviewees from the countries were also asked about the relevance of the project and they stated that the work was beneficial to countries in areas such as the monitoring of the MDGs, the collection and disaggregation of data and through the enhancement of the technical skills of staff.” Could you please revise the text and incorporate the evidence for this conclusion? The conclusion was reviewed and a minor revision to the text was made in order to make the point clearer. It should be noted however, that this conclusion was based on the overall findings of the assessment. When looking at the project as a whole participants stated that it was relevant to the countries needs and priorities. But once the various activities were reviewed, all of the activities were not seen as being relevant and consequently effective especially since long terms gains could not be reported. This conclusion is based on the part of the findings that examine the effectiveness of the project’s activities (page 19-33 of the draft final report). Also I arrived at the lessons learned that the activities should be geared towards the needs and interests of participants i.e. lessons learned # 4 based on the conclusion above. I do not think that there is a contradiction in the data – since the conclusion refers to effectiveness of specific components while the data to the left speaks to relevance of the project as a whole. In other words the project was relevant but its effectiveness (i.e. of all of the activities) did not fully reflect the needs/interests of the countries and focal points. 78