1995 es Corden, W. Max Corden, W. Max Una zona de libre comercio en el Hemisferio Occidental: posibles implicancias para América Latina En: La liberalización del comercio en el Hemisferio Occidental - Washington, DC : BID/CEPAL, 1995 - p. 13-40 2014-01-02T14:51:16Z hdl:11362/40641 FINAL ASSESSMENT REPORT ASSESSMENT OF DEVELOPMENT ACCOUNT PROJECT 12/13 AX Social inclusion of youth within a context of increasing violence and insecurity through innovative programmes and evidence-based policies May 2016 FINAL ASSESSMENT REPORT This report was prepared by Raúl Guerrero, an external consultant, who led the evaluation. Mr. Guerrero worked under the overall guidance of Raúl García-Buchaca, Chief of the Programme Planning and Operations Division of the Economic Commission for Latin America and the Caribbean (ECLAC), and Sandra Manuelito, Officer-in-Charge of the Programme Planning and Evaluation Unit, and under the direct supervision of Irene Barquero, Programme Officer in the same unit, who provided strategic and technical guidance, coordination, and methodological and logistical support. Assistance was also provided by María Victoria Labra, Programme Assistant, also of the Programme Planning and Evaluation Unit. The evaluation team is grateful for the support provided by its project partners at ECLAC, all of whom were represented in the Evaluation Reference Group (ERG). Warm thanks go to the programme managers of the Social Development Division for their cooperation throughout the evaluation process and their assistance in the review of the report, in particular Daniela Trucco, Social Affairs Officer, and Heidi Ullmann, Associate Economic Affairs Officer. The team also wishes to thank the programme managers of the ECLAC subregional headquarters in Mexico, in particular Humberto Soto, Associate Social Affairs Officer. All comments on the evaluation report by the Evaluation Reference Group and the evaluation team of the Programme Planning and Evaluation Unit were considered by the evaluator and duly addressed, as appropriate, in the final text of the report. The views expressed in this report are those of the author and do not necessarily reflect the views of the Commission. The annexes to this evaluation report have been reproduced without formal editing. Copyright © United Nations, May 2016. All rights reserved Printed at United Nations, Santiago S.16-00657 FINAL ASSESSMENT REPORT iii CONTENTS 1. INTRODUCTION ..................................................................................................................................................... 1 1.1 DEVELOPMENT ACCOUNT .................................................................................................................................................... 1 1.2 DEVELOPMENT CONTEXT ..................................................................................................................................................... 2 1.3 PROJECT DESCRIPTION ......................................................................................................................................................... 3 2. EVALUATION METHODOLOGY .............................................................................................................................. 4 2.1 PRINCIPLES ............................................................................................................................................................................... 4 2.2 OBJECTIVES AND SCOPE ...................................................................................................................................................... 4 2.3 APPROACH .............................................................................................................................................................................. 5 2.3.1 INCEPTION ...................................................................................................................................................................... 6 2.3.2 DATA COLLECTION ........................................................................................................................................................ 6 2.3.3 DATA ANALYSIS AND REPORTING ............................................................................................................................. 7 2.4 LIMITATIONS ............................................................................................................................................................................ 7 2.4.1 INTRINSIC TO THE EVALUATION ................................................................................................................................. 7 2.4.2 INTRINSIC TO THE PROJECT ......................................................................................................................................... 8 3. MAIN FINDINGS .................................................................................................................................................... 9 3.1 RELEVANCE .............................................................................................................................................................................. 9 3.1.1 COUNTRY AND REGIONAL NEEDS ............................................................................................................................. 9 3.1.2 THE ECLAC MANDATE .................................................................................................................................................. 10 3.1.3 PROJECT DESIGN ......................................................................................................................................................... 13 3.2 EFFICIENCY ............................................................................................................................................................................ 20 3.2.1 ORGANIZATIONAL ARRANGEMENTS AND ADDED VALUE .................................................................................. 20 3.2.2 RESOURCE MANAGEMENT ........................................................................................................................................ 22 3.2.3 ACTIVITY/OUTPUT REALIZATION .............................................................................................................................. 22 3.3 EFFECTIVENESS ..................................................................................................................................................................... 25 3.3.1 STRENGTHENED CAPACITY (EXPECTED ACCOMPLISHMENT 1) .......................................................................... 25 3.3.2 ENHANCED SHARING OF DATA AND INFORMATION (EXPECTED ACCOMPLISHMENT 2) ............................. 27 3.4 SUSTAINABILITY .................................................................................................................................................................... 29 3.4.1 IMPACT AND MULTIPLIER EFFECTS ............................................................................................................................ 29 3.4.2 DISSEMINATION ........................................................................................................................................................... 31 4. CONCLUSIONS ................................................................................................................................................... 32 5. LESSONS LEARNED .............................................................................................................................................. 34 6. RECOMMENDATIONS .......................................................................................................................................... 35 ANNEXES .......................................................................................................................................................................................... 39 ANNEX 1: TERMS OF REFERENCE ................................................................................................................................................. 40 ANNEX 2: DOCUMENTS REVIEWED ............................................................................................................................................ 51 ANNEX 3: PARTICIPANTS ANALYSIS ............................................................................................................................................ 52 ANNEX 4: SIMPLIFIED LOGICAL FRAMEWORK .......................................................................................................................... 55 ANNEX 5: EVALUATION MATRIX ................................................................................................................................................... 57 ANNEX 6: INTERVIEW GUIDELINES .............................................................................................................................................. 62 ANNEX 7: SURVEY QUESTIONNAIRE ........................................................................................................................................... 64 ANNEX 8: LIST OF INTERVIEWEES................................................................................................................................................. 75 ANNEX 9: EVALUATOR’S REVISION MATRIX .............................................................................................................................. 76 FINAL ASSESSMENT REPORT 1 1. INTRODUCTION 1. This report presents the final evaluation of the Development Account project, “Social inclusion of youth in the context of increasing violence and insecurity with a focus on Central America” (Development Account project ROA 254-8), which was commissioned by the Economic Commission for Latin America and the Caribbean (ECLAC). For further details, please see the terms of reference (ToR) included in annex 1. This report was prepared by Raúl Guerrero. 1.1 DEVELOPMENT ACCOUNT 2. The Development Account was established by the General Assembly in 1997, as a mechanism to fund the capacity development projects of the economic and social entities of the United Nations. It is intended to be a supportive vehicle for advancing the implementation of internationally agreed development goals and the outcomes of United Nations conferences and summits by building capacity at three levels: individual, organizational and enabling environment. The Development Account adopts a medium- to long-term approach in helping countries to better integrate social, economic and environmental policies and strategies in order to achieve inclusive and sustained economic growth, poverty eradication, and sustainable development. 3. Development Account projects are implemented by global and regional entities, cover all regions of the globe and focus on five thematic clusters. Projects are programmed in tranches, which represent the Account’s programming cycle. The Development Account is funded from the Secretariat’s regular budget and ECLAC is one of its 10 implementing entities. The United Nations Department of Economic and Social Affairs provides overall management of the Development Account portfolio. 4. Development Account projects aim at achieving development impact by building the socioeconomic capacity of developing countries through collaboration at the national, subregional, regional and interregional levels. The Development Account provides a mechanism for promoting the exchange and transfer of skills, knowledge and good practices between target countries in the same region or in different geographical regions, and through cooperation with a wide range of partners in the broader development assistance community. It acts as a bridge between in-country capacity development actors, on the one hand, and United Nations Secretariat entities, on the other. The latter offer distinctive skills and competencies in a broad range of economic and social areas that are often dealt with only marginally by other development partners at the country level. 5. For target countries, the Development Account provides a vehicle for tapping into the normative and analytical expertise of the United Nations Secretariat and receiving ongoing policy support in the economic and social area, particularly in areas where United Nations country teams lack the relevant expertise. The Development Accounts operational profile is reinforced by the adoption of pilot approaches that test new ideas and eventually scale them up through supplementary funding, and by an emphasis on the integration of national expertise in the projects to ensure national ownership and the sustainability of project outcomes. 6. ECLAC undertakes internal assessments of each of its Development Account projects in accordance with requirements under this programme. These brief end-of-project evaluation exercises examine the relevance, efficiency, effectiveness and sustainability of the project activities. Undertaken as desk studies, they consist of a document review, a stakeholder survey, and a limited number of telephone-based interviews. FINAL ASSESSMENT REPORT 2 1.2 DEVELOPMENT CONTEXT 7. The Project Document stated that violence associated with crime was a widespread problem in Latin American and Caribbean societies. Indeed, the “Global status report on violence prevention” (World Health Organization, 2014) affirms that the Americas have the highest estimated rates of homicide for low- and middle-income countries with an annual rate of 28.5 deaths per 100,000 inhabitants (followed by the African region with a rate of 10.7 deaths per 100,000 inhabitants). In addition, whatever the income level of the country, the homicide rate is disproportionately high among young people. It is, however, much more pronounced in low- and upper-middle-income countries than in lower-middle- and high-income countries. 8. During the design of the project, it was acknowledged that addressing violence was an increased priority of Central American governments.1 The high rates of violence result in high economic costs and a negative impact on well-being, particularly for young people. Addressing social exclusion and violence as they affect youth was considered as crucial to preserving the rights of this population and to enabling them to lead socially and economically productive lives in society. 9. The exclusion of youth and the violence that is rife among persons of this age group 2 are inextricably linked, highly complex issues. 3 Several initiatives launched in the past decade to promote youth inclusion in the region have been stymied by the changing socioeconomic and political context, marked by rising youth unemployment and uncontrolled violence. 10. Clearly, policy responses have not succeeded in supporting and promoting youth inclusion against the background of violence in the region. This is due partly to the lack of reliable data not only on the nature and extent of violence but also on the populations at risk and on the causes and consequences of violence. These data are essential for the development of well-informed national plans of action and policies, programmes and services to prevent and respond to violence. This lack of context-specific, up-to-date information was exacerbated by scant information on policy options and strategies that have been successfully implemented in the region. 11. The following assumptions were made to underpin the logic of the intervention: (a) All young people aspire to participate fully in social life. Young people are agents, beneficiaries and victims of major societal changes and are generally confronted by a dilemma: they must either seek integration into an existing order or serve as a driving force to transform it. (b) Youth inclusion is one of the most urgent challenges that societies in Latin America and the Caribbean face, and one of the key objectives that must be met in order to accelerate the achievement of the Millennium Development Goals and other internationally agreed development goals.4 (c) Effective youth inclusion must combine recognition of human rights with the assertion and full exercise of such rights by young people. To promote the social inclusion of youth, the countries of 1 This subregion has one of the highest homicide rates in the world. 2 Including school-based violence, street crime, gang membership, drug consumption and trafficking, and possession of weapons. 3 Separate and apart from the impact of individual biological, cognitive, and behavioural characteristics, societal factors also create conditions that contribute to youth violence. Economic crises, a decline in the standard of living, and in the case of Central America, post-conflict environments have destabilized control mechanisms (both formal and informal), exacerbated income inequality, housing shortages and overcrowding, and limited opportunities, causing frustration and an increase in alcohol and drug consumption. Across Latin America and the Caribbean, and in particular in Central America, transnational criminal networks and easy access to small arms have led to an escalation in violence among youth. Cultural factors may also come into play insofar as cultures that are more tolerant of violence and that propagate these attitudes through rituals, customs, and the media tend to have higher rates of youth violence. In some settings, violence may be seen as a normative and appropriate means for both personal gain and conflict resolution. 4 Social inclusion of youth means reducing poverty levels in this age group through productive employment and decent work (Goal1 of the Millennium Development Goals), ensuring their access to education (Goal 2), as well as to the skills they will need for a better integration in the labour market; promoting gender equality and the empowerment of young women (Goal 3); fostering access to reproductive health and improving maternal health (Goal 5), and reversing the spread of HIV/AIDS among this age group (Goal 6). FINAL ASSESSMENT REPORT 3 Latin America and the Caribbean must develop comprehensive approaches that combine several dimensions, including human security.5 (d) Social exclusion among youth can lead to social fragmentation and polarization, generate widening inequalities and create strains on individuals, families, communities, and institutions. Moreover, social exclusion and invisibility are contributing factors to violence among youth. At the same time, violence further exacerbates social exclusion among youth.6 1.3 PROJECT DESCRIPTION 12. The project was financed under the Development Account’s 8th Tranche (2012-2015) and implemented under the coordination of the ECLAC Social Development Division and the subregional headquarters in Mexico. The project document provided for implementation over a two-and-a-half- year period (from July 2013 to December 2015) for a total budget of US$ 500,000, targeting the following countries: Costa Rica, El Salvador, Guatemala, Honduras, Nicaragua and Panama. 13. The objective was to strengthen the capacity of Latin American and Caribbean countries, particularly those of Central America, to promote effective social inclusion of youth in the context of violence and insecurity. This objective was to be achieved through two intermediate “expected accomplishments”: (a) Strengthened capacity of Central American governments to assess and design more effective policies to confront youth exclusion and violence. (b) Enhanced sharing of data and information on critical economic and social challenges for youth in Latin America and the Caribbean to support national and regional decision-making. 14. The table below summarizes the intervention logic in relation to its expected accomplishments, main activities and indicators as described in the documents. The complete simplified logical framework is included in annex 4. Table 1 Summary of the intervention logic Expected accomplishment Main activities Indicators Expected accomplishment 1 Strengthened capacity of Central American governments to design and assess more effective policies to face youth exclusion and violence Consultative meetings (activity 1); technical cooperation to national youth institutes (activity 2); a regional comparative study and two national case studies (activity 3); six national and two sub-regional workshops (activity 4) y Increased number of policies or policy initiatives on youth exclusion and violence in the target region y Number of programmes at the national, subregional, and regional level which support specific action for the social inclusion of youth Expected accomplishment 2. Enhanced sharing of data and information on critical economic and social challenges for youth in Latin America and the Caribbean to support national and regional decision-making Two regional online databases relating to youth development were to be created: policies and programmes (activity 5) and socioeconomic data (activity 6). A network was to be established (activity 7) and a regional seminar organized (activity 8) y Enhanced availability to stakeholders of timely data on relevant aspects of social and economic challenges for youth y Increased number of Latin American and Caribbean countries for which information on innovative programmes addressing youth policy priorities relevant to meet MDGs/IADG –including violence– is analysed Source: Prepared by the author on the basis of the project document. 5 Comprehensive approaches that combine the dimensions of risk, capabilities, opportunities, belonging and value systems, and social participation. Human security –understood as the context in which people can make their choices safely and freely, while being relatively confident that the opportunities they have today will not be lost tomorrow– is clearly paramount, as there are few aspects more important to development than security from violence. 6 Those who resort to violence frequently lack access to economic opportunities and the social capital necessary to obtain services and resources available to mainstream society. When conventional methods of obtaining and working for increased social status, higher income, and wider influence are limited, as they often are in marginalized areas, some feel compelled to resort to violence. FINAL ASSESSMENT REPORT 4 2. EVALUATION METHODOLOGY 15. This final evaluation is in accordance with General Assembly resolutions 54/236 of December 1999 and 54/474 of April 2000, which endorsed the Regulations and Rules Governing Programme Planning, Programme Aspects of the Budget, the Monitoring of Implementation and the Methods of Evaluation (PPBME). In this context, the programmes are to be evaluated on a regular, periodic basis, covering all areas of work within their purview. 16. In this framework, the Executive Secretary of ECLAC is implementing an evaluation strategy that includes periodic evaluations of different areas of the Commission’s work. The present evaluation is therefore a discretionary internal evaluation managed by the Programme Planning and Evaluation Unit of the ECLAC Programme Planning and Operations Division. 2.1 PRINCIPLES 17. Despite the limited scope of this evaluation,7 it was conducted in line with the norms, standards and ethical principles of the United Nations Evaluation Group.8 As far as possible, the assessment was carried out in accordance with the guiding principles established by ECLAC. The evaluator adhered fully to the principle that “...evaluations should be carried out in a participatory and ethical manner...” During the evaluation process, efforts were made (subject to resources and time constraints) to involve as many of the key stakeholders as possible. 18. The information was triangulated at different levels (including sources and methods). To the extent possible, the evaluator ensured a cross-checking of the findings obtained through the different lines of inquiry (for example, desk research, interviews, surveys, beneficiaries or project managers) in order obtain credible and comprehensive answers to the evaluation questions. 19. Lastly, the evaluator sought to ensure that all beneficiaries, irrespective of their sex or ethnic group, were able to participate under the right conditions. Furthermore, ECLAC activities and products were assessed to determine whether they respected and promoted human rights. This included a consideration of whether ECLAC interventions treated beneficiaries as equals, safeguarded and promoted the rights of minorities, and helped to empower civil society. 2.2 OBJECTIVES AND SCOPE 20. In accordance with Development Account requirements, this internal assessment9 was undertaken between February and May 2016. The evaluation is retrospective and summative in nature and takes into account expected as well as unexpected results. It looks at all project activities and, to the extent possible, at non-project activities. In particular, it sought to assess and analyse: (a) The actual progress made towards project objectives. (b) The extent to which the project had contributed to outcomes, whether intended or unintended, in the relevant countries. (c) The efficiency with which outputs were delivered. 7 According to the terms of reference, “this exercise should not be considered a fully-fledged evaluation (i.e. less extensive data collection and analysis are involved…)”. 8 Standards for Evaluation in the United Nations System, UNEG, April 2005: http://www.unevaluation.org/document/detail/22, Norms for Evaluation in the United Nations System, UNEG, April 2005: http://www.uneval.org/document/detail/21. UNEG Ethical Guidelines for Evaluation, UNEG, March 2008: http://www.unevaluation.org/document/detail/102. 9 Self-evaluation undertaken under the respective programme managers may give rise to some ambiguity in view of the complementary nature and role of the self-evaluator as opposed to independent evaluation (i.e. that undertaken by oversight bodies not directed by the programme managers). FINAL ASSESSMENT REPORT 5 (d) The strengths and weaknesses of project implementation on the basis of the available elements of the logical framework (objectives, results, etc.) contained in the project document. (e) The validity of the strategy and partnership arrangements; the coordination between the two implementing divisions/offices and other implementing partners. (f) The extent to which the project was designed and implemented to facilitate the attainment of the goals. (g) The relevance of the project activities and outputs to the needs of member States, the needs of the region/subregion and the ECLAC mandates and programme of work. 21. In terms of the time frame, the evaluation covered the period beginning with the initial project design up to the completion of its final activities, taking into account also any results or repercussions generated since completion. As regards the geographical coverage, the evaluation covered six project countries: Costa Rica, El Salvador, Guatemala, Honduras, Nicaragua and Panama. The target audience and principal users of the evaluation were all of the project implementing partners (ECLAC divisions and subregional headquarters, as well as associated donors), the Development Account Programme Manager (the United Nations Department of Economic and Social Affairs) and other entities of the Executive Committee on Economic and Social Affairs. 22. The evaluation also sought to ascertain whether gender concerns were incorporated into the project, whether project design and implementation took into account the needs and priorities of women, whether women were treated as equal players, and whether it served to promote women’s empowerment. Lastly, the evaluation has placed special emphasis on measuring the extent to which the project adhered to the following key Development Account criteria:10 (a) Result in durable, self-sustaining initiatives to develop national capacities, with measurable impact at field level, ideally having multiplier effects. (b) Be innovative and take advantage of information and communications technology, knowledge management and networking of expertise at the subregional, regional and global levels. (c) Utilize the technical, human and other resources available in developing countries and effectively draw on the existing knowledge/skills/capacity within the United Nations Secretariat. (d) Create synergies with other development interventions and benefit from partnerships with non- United Nations stakeholders. 2.3 APPROACH 23. The evaluation focused on addressing the evaluation questions presented in the terms of reference in a timely manner (see annex 1). The unit of analysis was the project itself, including the design and implementation of planned activities as well as the results achieved. 24. The evaluation was structured around four standard evaluation criteria established by the United Nations Evaluation Group: relevance, efficiency, effectiveness and sustainability (the impact of the project was addressed only briefly as a proxy for sustainability): (a) Relevance: the extent to which the project and its activities were suited to the priorities and policies of the region and countries at the time of formulation and were linked or related to the ECLAC mandate and programme of work. 10 Guidelines for the Preparation of Concept Notes for the 8th Tranche of the Development Account (2012-2013), United Nations General Assembly. FINAL ASSESSMENT REPORT 6 (b) Efficiency: measurement of the outputs (qualitative and quantitative) in relation to the inputs, including complementarity (the extent to which the activities and the outcomes of the project were able to establish and/or exploit synergies with other actions implemented by ECLAC, other United Nations bodies or local organizations) and value added (the extent to which the project activities and outcomes confirmed the advantages of the Commission’s involvement, especially by promoting human rights and gender equality). (c) Effectiveness: the extent to which the objectives were attained and expected accomplishments fulfilled. (d) Sustainability: the extent to which the benefits and impacts of the project are likely to continue after funding has been withdrawn and the likelihood of the project’s being disseminated and replicated in the long term. 25. The evaluator worked independently but did receive organizational support from ECLAC in the setting up of interviews and management of the online survey (see paragraph 2.3.2). The evaluation took the form of a desk study and comprised three phases: (a) Inception; (b) Data collection; and (c) Data analysis and reporting. The approach and these phases are outlined below. 2.3.1 INCEPTION 26. During this phase, which started with the document review, the evaluation considered the project, context, main stakeholders (partners, beneficiaries, etc.) and the intended and achieved results. Relevant sources of information and conceptual frameworks that fall within the framework of the project were identified and reviewed, including: allotment advice, redeployments, project document, progress reports, final report, meeting reports, workshop-related documents, studies and the terms of reference for consultants (for the full list of documents, see annex 2). In addition, a break-down of the participants and their attendance at the various events was provided (for further details, see annex 3). 27. This phase culminated with the preparation of the inception report, which describes the overall evaluation approach and contains an evaluation matrix and a detailed work plan. The evaluation matrix served as a comprehensive guide for the preparation and application of data collection tools (see annex 5). It also shows how the evaluation criteria and key questions were organized (for example, by using encapsulating questions to avoid repetition and redundancy). 2.3.2 DATA COLLECTION 28. To the extent possible, the data collection and analysis were based on a mixed-methods approach. On the basis of the evaluation matrix, several tools were developed to gather primary data, including specific interview guides (see annex 6) and survey questionnaires (see annex 7). The evaluator carried out eight interviews (semi-structured, telephone-based, individual, key informant) with project managers, implementing partners and beneficiaries (see the full list of interviewees in annex 8). 29. In order to probe different hypotheses, both quantitative and qualitative data were collected from key project stakeholders (a sample of implementing partners and project beneficiaries) through two electronic surveys. The Project Planning and Evaluation Unit administered the surveys and compiled preliminary information. The table below summarizes the number of stakeholders that were contacted and the different response rates. 30. As the above figures demonstrate, every effort was made to ensure that sufficient responses were received from beneficiaries. Partner institutions were differentiated depending on the role they played in the implementation but they also happen to be project beneficiaries. FINAL ASSESSMENT REPORT 7 Table 2 Response rate Implementing partners (including United Nations agencies and intergovernmental organizations) Project beneficiaries (including government institutions, civil society and academia) ECLAC staff (project managers and others participating in the events) Total Interviews: Number of stakeholders contacted 4 16 2 22 Number of stakeholders interviewed 2 3 2 7 Surveys: Number of stakeholders contacted 14 346 9 369 Number of survey responses 12 (86%) 65 (19%) 5 (55%) 82 (22%) Source: Prepared by the author. 2.3.3 DATA ANALYSIS AND REPORTING 31. Recalling that this was not a fully-fledged evaluation but a brief end-of-project assessment, the evaluator used the data collected (a) to make judgments on whether meanings and assertions from the different data sources were reliable; and (b) to identify patterns (whether consistencies or covariations)11 in the data. 32. The evaluation included a content analysis of findings in the document review to the extent that they provided answers to the evaluation questions. In particular, the evaluator analysed both the problem and objective trees included in the project document by logically reconstructing the theory of change (ToC), identifying original weaknesses, gaps and/or any unintended effects (both positive and negative). 33. In addition, the interview responses were analysed12 to tease out any details, gaps and uncertainties in questions that were not clarified by the documentary evidence. The responses to those questions that were answered through the documents were cross-checked with the responses from interviewees. 34. Lastly, the Consultant reviewed the results of the surveys provided by the Project Planning and Evaluation Unit to check (a) internal consistency between the different respondents; and (b) external consistency between the survey results and the findings from the two other sources of evidence (document review and stakeholder interviews). 2.4 LIMITATIONS 2.4.1 INTRINSIC TO THE EVALUATION 35. This end-of-project evaluation should be seen as a quick review through an expedited process. The available resources were limited and therefore the assessment’s depth and scope were also somewhat limited (for example, the evaluation did not involve in-country field work or any face-to- face interviews with project stakeholders or target groups). Caution should be exercised in interpreting the findings, in particular those relating to the project impacts at the policy level. As 11 An effect is attributed to the one of its possible causes with which, over time, it covaries (Kelley, 1973). 12 The 55-minute interviews were conducted in Spanish. They were later transcribed and translated into English in order to identify themes based on categories of codes that consistently appeared in the transcribed data using the grounded theory method developed by Glaser and Strauss (1967). FINAL ASSESSMENT REPORT 8 discussed earlier, this evaluation addressed the impact of the project but only in a cursory manner (see paragraph 2.3). A future evaluation (ideally at a more strategic level) based on a more comprehensive methodology would enable the evaluators to conduct a more thorough examination of the contribution and/or attribution. 36. Despite the triangulation foreseen by the methodology, the evaluation may contain biases of various kinds. In this regard, it should be noted that (a) the reformulation of hypotheses has been very limited; (b) the limited number of actors consulted may suggest that the findings are inconclusive; (c) the methodology deliberately did not provide for the study of power relationships, possible conflicts and the boundaries of the system13 (this means that the evaluation did not seek to answer why some aspects were prioritized over others); and some stakeholders found that the specific activities of the project were not easy to identify. 37. Although 22 persons were invited to take part in interviews and great efforts were made to schedule as many as possible, only seven were successfully implemented. The survey also yielded a low response and many beneficiaries did not answer all the questions. This diminished the comparability of the surveys and a more careful interpretation of the survey results was needed. 2.4.2 INTRINSIC TO THE PROJECT 38. The evaluability14 of the project is somehow limited by the absence of baseline and monitoring data. Furthermore, the documentary information available for the project was often descriptive rather than analytical. The final report is an exception as it is quite comprehensive but it was available only at the very end of the evaluation process. 13 The boundaries of the system define what is inside and what is outside. 14 The extent to which an activity or project can be evaluated in a reliable and credible fashion (Organization for Economic Cooperation and Development (OECD)-Development Assistance Committee (DAC), 2010). FINAL ASSESSMENT REPORT 9 3. MAIN FINDINGS 39. This section outlines the main findings and analysis relating to each of the evaluation criteria (relevance, effectiveness, efficiency and sustainability), including the design and theory of change. 3.1 RELEVANCE 3.1.1 COUNTRY AND REGIONAL NEEDS 40. As explained in section 1.2, achieving the social inclusion of youth and reducing violence continue to be high priorities for Central American governments. The project is therefore consistent with regional priorities and the main bottlenecks were identified at the design phase. This was confirmed by all stakeholders during both the interviews and the survey. In particular, all five ECLAC project managers replied in the affirmative to this question in the survey (unfortunately, the question was skipped by the other recipients). Both the interviews and the comments offered in the survey confirm that addressing the factors that link violence and youth is of crucial importance. Social inclusion of young people is recognized as a priority for the countries of the region not only from the life-cycle (intergenerational) perspective but also in terms of the demographic dividend15 to be derived from a growing youth population. 41. The beneficiaries recognized the relevance of the project both technically and politically. Indeed, intersectoral coordination is weak and the approach to the problem is fragmentary (each ministry has a partial vision rather than a comprehensive perspective of the situation). The institutions responsible for improving this coordination (for example, youth institutes) have numerous technical, operational and political weaknesses. All the interviews identified the policymakers’ lack of knowledge, skills and information as one of the main limitations to promoting effective initiatives for the social inclusion of youth. Although the project did not target the “social” ministries with the biggest budgets and responsibilities, they were encouraged to participate in some of the activities, including main events in order to enhance the institutional dialogue and contribute to a joint understanding of the problem. 15 The demographic dividend is defined by the United Nations Population Fund (UNFPA) as “the economic growth potential that can result from shifts in a population’s age structure, mainly when the share of the working-age population (15 to 64) is larger than the non-working-age share of the population (14 and younger, and 65 and older).” In other words, it is “a boost in economic productivity that occurs when there are growing numbers of people in the workforce relative to the number of dependents.” Both project managers and beneficiaries considered that the project was aligned with regional priorities. All sources of information (documents, surveys and interviews) confirm that the different problems arising from the linkages between violence and youth must be addressed as a matter of crucial importance (finding 1). Project managers identified the main bottlenecks at the project design stage. One of them being the fact that decision makers’ efforts to design effective and targeted policies and programmes were hindered by a lack of skills, information and practical tools (finding 2). The specificities of each country were analysed at the consultative meetings. All stakeholders considered that the project responded to the different needs by adapting to the specific context. The majority of beneficiaries agreed that both the methodology and the implemented activities were relevant to their work and to the national context (finding 3). FINAL ASSESSMENT REPORT 10 3.1.2 THE ECLAC MANDATE 42. The project is related to the outcomes of several major United Nations conferences, summits and human rights conventions, such as the Convention on the Rights of the Child, the Convention on the Elimination of all Forms of Discrimination against Women, the World Programme of Action for Youth to the Year 2000 and Beyond, and the United Nations High-level Meeting on Youth (July 2011).16 The project also contributed to the achievement of the Millennium Development Goals, in particular Goal1 (Eradicate extreme poverty and hunger); Goal 2 (Achieve universal primary education); Goal 3 (Promote gender equality and empower women); Goal 5 (Improve maternal health) and Goal 6 (Combat HIV/AIDS, malaria and other diseases). 43. The project was well aligned with the General Assembly’s Strategic Framework and Programme of Work insofar as it contributed to and coordinated actions towards economic development and reinforcing economic relationships in Latin America and the Caribbean. Furthermore, the project contributed directly to the ECLAC Strategic Framework. During the period 2012-2015, one of the main priorities of the Commission’s work on economic and social development was to promote a social covenant by improving social equality, reducing social risks, and reinforcing gender mainstreaming in public policies.17 The aims of subprogrammes 5 (Social Development and Equality) and 12 (Subregional Activities in Central America, Cuba, the Dominican Republic, Haiti and Mexico) were as follow: • Subprogramme 5 includes two expected accomplishments: (a) Increased capacity of Latin American and Caribbean governments to formulate policies and programmes that address the structural and emerging social risks affecting various socioeconomic groups, with attention mainly to the poor, women, youth and children; and (b) Strengthened technical capacities of social policy institutions to improve the social impact of public action and to enhance dialogue with other government entities and stakeholders on poverty alleviation and reducing inequality. • Subprogramme 12 provides for subregional activities geared to achieving dynamic growth and sustainable, inclusive and equitable development within a robust and democratic institutional framework, to enable the countries in the subregion to fulfil the internationally agreed development goals, including the Millennium Development Goals. 44. Both the interviews and the survey confirmed the relevance of the project, which contributed to the ECLAC programme of work in at least three different ways: by promoting crucial research, by strengthening technical capacities and by encouraging collaboration between different ECLAC 16 This meeting emphasized the need to advance towards comprehensive and multisectoral programmes that address the different factors that generate scenarios of vulnerability and exclusion among young people. 17 For further details, see the Programmes of Work of the ECLAC System for 2012-2013 and 2014-2015. All sources of information indicate that the project was fully in line with major United Nations conferences and summits (finding 4). Project managers were confident that the project contributed to the ECLAC mandate by coordinating actions towards economic development and reinforcing economic relationships within the region. This was fully demonstrated by the documentation at their disposal (finding 5). Gender-related issues were not taken into account and the design was not gender-responsive. Nevertheless, most stakeholders were satisfied that human rights and gender equality had been taken into consideration during implementation (finding 6). FINAL ASSESSMENT REPORT 11 offices. The project also contributed to the strategic aim of generating, disseminating and applying “innovative and sound approaches to tackling development challenges in the subregion” while strengthening “multisectoral and interdisciplinary analysis … and the development of analytical models with quantitative and qualitative tools”. 45. Although the project document fails to address gender-related issues (see paragraph 3.1.3) and the design is not gender-responsive, both the interviews and the survey confirm that the project was implemented from a human rights and gender perspective. All five project managers and almost 67% of the beneficiaries (48 out of 72 respondents) confirmed that there had been equal participation of men and women at the events. Only eight (11%) beneficiaries considered that women were underrepresented. Table 3 Beneficiaries’ perception of gender representation at events Do you think that there was equal participation of women and men at the events? Yes: 48 respondents (67%) No: 8 respondents (11%) Without sufficient knowledge to answer: 16 respondents (22%) Question skipped by 10 respondents Source: Prepared by the author. 46. Along the same lines, 75% of the beneficiaries (54 out of 72) felt that the themes treated at the events incorporated a human rights and gender perspective and 66% (46 out of 72) considered that this perspective was adopted in the publications. It should be noted that almost all respondents acknowledged that this perspective was relevant for both the events and the publications. Table 4 Beneficiaries’ perception of the inclusion of gender perspective at events and publications Do you think that the themes treated at the events incorporated a human rights and gender perspective? Yes, they were adequately treated: 54 respondents (78%) No, they were not adequately treated: 11 respondents (15%) No, they were not relevant: 2 respondents (3%) Without sufficient knowledge to answer: 5 respondents (7%) Question skipped by 10 respondents Do you think that the publications incorporated a human rights and gender perspective? Yes, they were adequately treated: 46 respondents (66%) No, they were not adequately treated: 5 respondents (7%) No, they were not relevant: 0 respondents (0%) Without sufficient knowledge to answer: 19 respondents (27%) Question skipped by 10 respondents Source: Prepared by the author. 47. It could be argued that gender could be mainstreamed more visibly and that most of the project documents do not address gender issues or analyse the particular situation of women in relation to youth violence. Gender equality is mentioned only in the acknowledgement that most countries incorporated it into their national youth policies. For example, the study “Políticas y prácticas de FINAL ASSESSMENT REPORT 12 prevención de la violencia que afecta a la juventud en Centroamérica - Análisis comparativo”18 contains 17 recommendations none of which concern gender equality. Only one of those recommendations targets the specific situation of women (a recommendation for promoting initiatives to address teenage pregnancy). 48. Nevertheless, one of the most important outputs of the project, the book Youth: realities and challenges for development with equality19 explores the links between gender and youth violence. Chapter III presents a very interesting and innovative analysis of gender as a risk factor for youth violence, youth as a factor for gender violence and gender violence within gangs. The document “Hacia la inclusión social juvenil: herramientas para el análisis y el diseño de políticas”20 also analyses these links in chapter 3. During the interviews, it was also mentioned that the project addressed the high rate of school dropouts among young males and that the events addressed human rights and gender issues. Diagram 1 Snapshots from two presentations made at the subregional workshop in the Dominican Republic 49. In addition to gender equality, the project also addressed other human rights. For example, stereotypes such as those associated with the ‘NEET category (not in employment, education or training) were to be avoided. Both of the studies referred to above point out that youths who are neither working nor studying are often stigmatized as persons at risk, prone to laziness, crime or alcohol and drug abuse. The documents demonstrate the diversity of this group and the need to recognize the complex factors that result in their exclusion. Lastly, this is shown to be tied in with gender equality as many of these young persons are women. 50. Although the gender perspective was not thoroughly mainstreamed (for example, in the design), there is no doubt that women and men, including panellists at the events, participated equally in the design and the implementation of activities. The lists of participants confirm that female participation in the events ranged between 40% and 52%. The main concern expressed by several stakeholders about human rights and gender issues was that a greater effort could be made to translate into practice the very important concepts introduced by the project. 18 http://www.cepal.org/es/publicaciones/39820-politicas-practicas-prevencion-la-violencia-que-afecta-la-juventud-centroamerica. 19 http://repositorio.cepal.org/bitstream/handle/11362/38978/S1500718_es.pdf?sequence=4. 20 http://repositorio.cepal.org/bitstream/handle/11362/39001/S1501236_es.pdf?sequence=1. En foqu e de Equidad: Los derechos deben ser cumpl idos para todas las personas sin discrim inación alguna . Reconoce, como una regla fundamenta l, el derecho de todas las personas a las libertades fundamentales, s in distinción alguna, y reconoce el derecho de agrupaciones sociocu lturales d iversas a ejercer plena y eficazmente todos los derechos en condiciones de igua ldad. En foqu e de Derechos Humanos: atraviesa su di seño y desarrollo, promueve el ejercicio de la ciudadanía plena y establece la participación de las juventudes como un eje fundamenta l. Se d irige a la cons trucción de una sociedad i ntegrada, solidaria, equi tativa y jus ta; por lo tanto reconoce a las personas jóvenes como sujetos/as de derechos, sin d isti ngo de condición socioeconómica , étn ica, de relig ión , sexo, opción sexua l, idioma , opin ión política o de otra índole, or igen naciona l o socia l, nacimiento, edad o cualqu ier otra condición social; permite una concepción de i ntegridad, no- discrim inación, equidad e igualdad social, y favorece a la totalidad de las personas jóvenes. Políticas de juventud: contenidos Salud Elementos en común • Promoción de salud sexual y reproductiva • Provenir tabaquismo, uso de alcohol y sustancias Especific idades •Salud integral •Calidad •Violencia de género •Dist ribución de métodos anticonceptivos FINAL ASSESSMENT REPORT 13 3.1.3 PROJECT DESIGN 51. The design of the project entailed several steps: stakeholder analysis, problem analysis and objectives analysis. A. STAKEHOLDER ANALYSIS 52. Regarding stakeholders, the project document was descriptive and rather succinct. It stated that all Latin American and Caribbean countries have national youth institutes or similar governmental agencies whose role it is to develop public policies for youth, and, in particular, to tackle the problem of youth violence. As the principal promoters of the social inclusion of youth, such agencies were the primary stakeholders and main beneficiaries of the project. During the design phase, potential partnerships were sought for effectively addressing the problem. The project sought to collaborate with key regional organizations such as the Ibero-American Youth Organization (OIJ) and the Central American Social Integration Secretariat. 53. The most recent guidelines drawn up by the Department of Economic and Social Affairs for the preparation of project documents 21 recommend that all non-United Nations stakeholders in the project be identified, including those affected by the problem(s) in question. These guidelines were not approved until after the project design was completed but can be used as a benchmark and best practice. They suggest that the implementing entities provide the following information for each relevant stakeholder: Table 5 Stakeholder analysis Non United Nations stakeholders Type and level of involvement in the project Capacity assets Capacity gaps Desired future outcomes Incentives All direct and indirect non- United Nations stakeholders should be listed here each on a separate row How does each of the stakeholders relate to the project/problem outlined in the previous section? What are the stakeholder’s resources and strengths that can help address the problem that the project strives to solve? What are the stakeholder’s needs and vulnerabilities that the project attempts to bridge? What are the desired outcomes for the stakeholder as a result of project implementa-tion? What is the stakeholder’s incentive to be involved in the project? How can buy-in be ensured? Source: Guidelines for the preparation of Project Documents for the 10th tranche of the Development Account. 21 Guidelines for the preparation of Project Documents for the 10th tranche of the Development Account (July 2015), http:// www.un.org/esa/devaccount/projects/guidelines.html. Important cause-effect assumptions and potential risks were made explicit in the design of the project and seem plausible (finding 7). The analysis of both the problem and the objectives highlighted credible cause-effect relationships and confirm the adequacy of the project to address the challenges (finding 8). The roles that different stakeholders needed to play in solving the problem was to some extent assessed during the design phase. However, there is no evidence of their engagement in the design (finding 9). The simplified logical framework was useful at the project proposal stage but was underemployed as a management tool (finding 10). FINAL ASSESSMENT REPORT 14 54. The roles that different stakeholders needed to play in solving the problem was to some extent assessed during the design phase. The aim of the project was to enhance the awareness, skills, and capacity of the staff of the relevant agencies and provide them with more diverse approaches and instruments for improving the impact of initiatives for the social inclusion of youth. It was recognized that such institutions often lacked appropriate funding or the necessary human resources to sustain effective youth development policies and programmes and to accomplish overall policy coordination. 55. The events organized as part of the project were attended by over 700 participants. As set out in the project document, the participants consisted of high-level public sector decision makers and senior advisers in national youth institutes and similar governmental agencies (41%); experts, practitioners, representatives of civil society organizations and community members that deal with youth and violence among youth, including young people themselves (36%); and representatives of United Nations entities, academia and regional organizations (for example, the Ibero-American Youth Organization, the Central American Integration System, the Caribbean Community (CARICOM) and Common Market and the Organization of Ibero-American States). Approximately 95% of the participants in these events worked in eight countries (Chile, Costa Rica, Dominican Republic, El Salvador, Guatemala, Honduras, Nicaragua and Panama, (for further details, see annex 3). Figure 1 Participants at the events Source: Prepared by author. B. PROBLEM ANALYSIS 56. The project document provides an analysis of the main problems faced by the youth in the region in relation to social exclusion and violence. Addressing social exclusion of and violence among youth was fundamental for preserving the rights of young people and ensuring that they can enjoy socially and economically productive lives. The document also acknowledged that devising coherent public policies to tackle these issues was challenging due to the complex and multifaceted nature of the problems. 57. The existing policy responses were deemed insufficient for supporting and promoting the social inclusion of youth in contexts of violence in the region owing to a number of shortcomings, notably the lack of relevant, context-specific and up-to-date information on the socioeconomic situation of the youth, which hindered decision makers from designing effective, targeted policies and programmes. Failure to share information on successful policy options in the region was another stumbling block. Lastly, although every 29 209 238 20 29 47 14 61 Academia Civil society (and local administration) Public administration Regional organization Unknown United Nations ECLAC FINAL ASSESSMENT REPORT 15 country in the region had a national youth institute or similar governmental agencies, they had insufficient funding or scant human resources to sustain effective youth development policies and programmes or to accomplish overall policy coordination with other entities. 58. In general, all the assumed causal relationships seemed plausible. Nevertheless, the credibility of the assumption could have been increased by including references to relevant documents, research, policies and statistics. Diagram 2 depicts the main assumptions and causal links. Diagram 2 Problem tree Source: Project document. 59. The text of the project document presents a clearer analysis than the diagram above. The hierarchy of the different levels and their causal relationships are not always evident. For example, the “lack of awareness among policy-makers…” seems rather a cause than a consequence of the “economic, social and environmental shocks” that threaten youth; the same applies to the “limited linkages between social inclusion and other policies and programmes directed at youths” and the statement that “progress with regard to equity and the Millennium Development Goals is stifled”. 60. Nevertheless, it is fair to say that the analysis detected crucial underlying causes of the problems and supported the premise that the project would be able to address the challenges. Over 63% of the respondents to the survey (47 out of 74) felt that lack of capacity and/or information was the main constraint faced by policymakers in their efforts to formulate effective policies for the social inclusion of youth. Youth remain disengaged from society due to limited opportunities, lack of access to institutions, and low levels of solidarity Excluded youth are at risk for engaging in violence, as they seek to assert themselves and gain access Youth are marginalized and are denied social status and rights Policies and programmes fail to address key issues such as violence and the social inclusion of youth Lack of awareness among policymakers of new challenges, policy options, strategies, and good practices to promote youth social inclusion and prevent violence among youth Economic, social, and environmental shocks threaten youth Limited linkages between social inclusion and other policies and programmes directed at youth Progress with regard to equity and the MDGs is stifled Lack of data to identify, monitor, and evaluate the specific situation of youth Good practices and experiences relating to youth social inclusion and violence in the region are not readily exchanged Funding for youth social inclusion programmes is weak FINAL ASSESSMENT REPORT 16 Figure 2 Stakeholders perception of the underlying problems (Percentages) Source: Prepared by the author based on the survey responses. 61. Admittedly, the analysis could be considered a bit simplistic as it did not identify the relationships (or risks) with other identified problems (such as lack of resources of the institutions) and it did not address specific country-level problems, needs or constraints. Stakeholders have mentioned other bottlenecks such as lack of interest or lack of political commitment, lack of resources, weak institutions or absence of a coordinated approach. The above-mentioned guidelines for the preparation of Development Account project documents recommend undertaking a country-by-country analysis in order to provide a clearer picture of the state of affairs in each target country and the realistic outcome sought.22 Although recognizing that it was included as part of the implementation (initial consultative meetings), the project design probably would have benefited from additional analysis with specific stakeholders at the country level. This would have provided insight into the actual scale and complexity of the problem and the links between the different contributing factors (more targeted technical assistance). Table 6 Country analysis Country State of affairs Realistic outcomes Country name How does the identified problem identified play out in the selected country? What progress has already been made or what steps have been taken to address the issues? What are the principle assets the country has in addressing the issue? What are the principle gaps to be addressed? What will this project be able to achieve in the country within the time frame available? What tangible outcomes/outputs are foreseen? Source: Guidelines for the preparation of Project Documents for the 10th tranche of the Development Account. 22 As the guidelines were not available at the time of the design, the project could not have adhered to them. Any reference to them in the present evaluation is based on their value as a relevant benchmark and as a source of best practices. 64 23 13 Do you think that the policymakers’ lack of capacity and/or information is the main limitation in the formulation of effective policies for the social inclusion of youth? Yes No Without sufficient knowledge to answer FINAL ASSESSMENT REPORT 17 62. The analysis does not take into account gender-related issues. The guidelines are also clear in this respect as they recommend devoting attention to gender considerations, identifying dimensions of gender inequality and determining the extent to which women and men may be affected differently by the problem and require differentiated capacity development support. The evaluation acknowledges that there has been little research on and less evidence of the relationships between gender and youth violence. However, it has been reported that, almost everywhere, youth homicide rates are substantially lower among females than among males, suggesting that being a male is a strong demographic risk factor.23 C. OBJECTIVES ANALYSIS 63. Apparently, the analysis of objectives was limited to modifying the problems into positive statements of what is to be achieved. The objective tree corresponds directly to the problem tree discussed above. Diagram 3 Objective tree Source: Project Document. 64. Diagram 3 above sets out some of the short-, medium- and long-term goals to be achieved in order to obtain a sustainable solution. It reflects the issues identified above for the problem tree, for example, lack of detail, vague causality and doubtful hierarchy. A more robust problem tree could have translated into a more robust objective tree. To this end, it would have been advisable to 23 “World report on violence and health”, World Health Organization, 2002. http://apps.who.int/iris/bitstream/10665/42495/ 1/9241545615_eng.pdf. Youth are more engaged in society enjoying increased opportunities, greater access to institutions, and higher levels of solidarity Youth are less at risk for engaging in violence because of greater social inclusion The social status and rights of youth are improved Policies and programmes address key issues such as violence and the social inclusion of youth Enhanced awareness of policymakers of new social challenges, policy options, strategies, and good practices in social inclusion of youth, including violence Greater attention is directed to economic, social and environmental shocks that threaten youth Enhanced linkages between social inclusion and other policies and programmes directed at youth Disparity is reduced in the Millennium Development Goals process Adequate data on identifying, monitoring, and evaluating youth and their specific conditions Good practices and experiences within the region are widely exchanged Funding for youth social inclusion programmes is increased Approaches and instruments for youth social inclusion and related capacity-building are developed An interregional network is developed for knowledge management and partnership development on youth social inclusion and violence prevention FINAL ASSESSMENT REPORT 18 verify the hierarchy and causality of the objectives explicitly. By doing this, stakeholders could have engaged in a process to visualize what the future would look like if the problems were resolved before proceeding to reformulate them. 65. Most of the objectives included in the four lower levels of the tree are related to the second expected accomplishment of the project (“enhanced sharing of data and information…”). It is surprising that the only objective clearly linked with the first expected accomplishment ( ‘strengthened capacity…’) is at the very bottom of the tree and listed as an activity/output (i.e. approaches and instruments for capacity-building are developed). In the text though, it is recognized that “…those involved in designing policies and programmes need to be equipped with the skills to analyse these data as well as the capacity to work coherently…”. A broader analysis would have shed light on both intended and unintended effects and would have allowed project managers to respond to changes during implementation. D. PROJECT STRATEGY 66. The project strategy consists of the project objective, expected accomplishments, indicators of achievement (including means of verification), main activities and explicit assumptions and hypotheses. It is set out in a “simplified logical framework” (see annex 4). Although the project could be considered small in scope and budget, the importance of a robust theory of change (ToC) and/or logical framework should not be understated. These are essential for demonstrating what has been achieved, facilitating monitoring and sharing information, thus, ensuring that the results are realistic, transparent and reliable. 67. Important cause-effect assumptions and potential risks were made explicit. Nevertheless, the project design would have benefited from a more thorough description of its logic, for example, an explicit theory of change. Clearly, a single project cannot address all possible problems. However, a systemic approach to the problems would have made it possible to investigate unintended effects (either positive or negative); power relationships; and possible conflicts at the boundaries of the system. For example, the design did not consider the possible effects on the project of lack of resources, possible institutional weaknesses or staff turnover. 68. As explained above, the analysis (problem and objective) is insufficient to justify expected accomplishment 1, which, furthermore, is too similar to the objective of the project. The strategy (including the expected accomplishments) fails to address gender equality and does not explain how the participation of civil society (and youth) was to be ensured. Most stakeholders believe that the causal relationships established by the project (activities, results and objectives) are logical and credible. Others have pointed out that the objectives were too ambitious. Table 7 Project managers’ perception of the project logic Are the causal relationships established by the project (activities, results and objectives) logical and credible? Yes: 4 respondents (80%) No: 1 respondent (20%) Source: Prepared by the author. 69. The simplified logical framework was useful at the project proposal stage and to some extent during implementation. Although project managers confirmed that it was a good “guide”, it is doubtful that it was an effective management tool. It would have been more useful to expand it further by adding details to facilitate monitoring and reporting. In this sense, the indicators could have been refined (for example, by providing baselines). Although not specifically mentioned in the FINAL ASSESSMENT REPORT 19 Development Account Project Document template, 24 the most recent guidelines recommend strengthening the indicators by ensuring that all of them include clear targets. In this sense, the entities involved would be expected to include benchmarks for all indicators and ensure that there is a baseline for measuring or assessing change quantitatively and/or qualitatively. Table 8 Project managers’ perception of the simplified logical framework Was the simplified logical framework useful as an effective management tool? Yes: 3 respondents (60%) No: 2 respondents (40%) Source: Prepared by the author. 70. According to a report prepared for the Quadrennial Comprehensive Policy Review (2012) of the Department of Economic and Social Affairs, results-based management (RBM) is a broader management strategy and is not synonymous with performance monitoring and evaluation. RBM is conceptualized as a results chain of inputs-activities-outputs-outcomes-impact. The assumption is that actions taken at one level will lead to a result at the next level, and the results chain determines the sequence of actions taken to achieve a particular result.25 71. Results-based management is used to define and measure outcomes (particularly challenging for development interventions such as advocacy, capacity development and advisory services). Measurement at the output level is also important for monitoring the use of resources, implementing related activities and measuring the specific outputs delivered through these activities. However, the project did not develop indicators that comprehensively captured its performance. Table 9 Project results framework Expected accomplishments Indicators of achievement Expected accomplishment 1 Strengthened capacity of Central American governments to assess and design more effective policies to confront youth exclusion and violence. Indicator of achievement 1.1 Increased number of policies or policy initiatives on youth exclusion and violence in the target region. Indicator of achievement 1.2 Number of programmes at the national, subregional and regional level which support specific action for social inclusion of youth. Expected accomplishment 2 Enhanced sharing of data and information on critical economic and social challenges for youth in Latin America and the Caribbean to support national and regional decision-making. Indicator of achievement 2.1 Enhanced availability to stakeholders of timely data on relevant aspects of youth- related social and economic challenges for youth. Indicator of achievement 2.2 Increased number of Latin American and Caribbean countries for which information on innovative programmes addressing youth policy priorities relevant to meet MDGs/IADG – including violence– is analysed. Source: Project document. 72. As shown in the table above, two indicators are used to assess the achievement of each expected accomplishment. For example, the strengthened capacity of the governments is evidenced by the 24 http://www.un.org/esa/devaccount/projects/guidelines.html. 25 Results-Based Management in the United Nations Development System: Progress and Challenges – A report prepared for the United Nations Department of Economic and Social Affairs for the Quadrennial Comprehensive Policy Review (July 2012). See [online] http://www.un.org/esa/coordination/pdf/rbm_report_10_july.pdf. FINAL ASSESSMENT REPORT 20 increased number of policies and programmes. Although these indicators provide valid information about the project’s contribution to major long-term initiatives, the causality is weak. It seems difficult to attribute to a project (especially one of this size and scope) the approval of new policies and programmes at the national, subregional and regional level. In this sense, it would be advisable to include indicators at a lower level also in order to measure the more direct effects of the project and provide evidence to demonstrate the logic of the intervention (reinforcing attribution at higher levels). 3.2 EFFICIENCY 3.2.1 ORGANIZATIONAL ARRANGEMENTS AND ADDED VALUE 73. Project implementation started immediately after the signature of the allotment advice in July 2013. The activities were carried out as planned. During the first months, consultative meetings were organized in Central America with United Nations country teams and national counterparts to discuss and gather information on policies relating to the social inclusion of youth, human security and the prevention of violence. Based on a review of the existing literature, an analytical framework was developed in order to ensure that the different stakeholders involved in the project had a consistent approach to the issues of violence among youth and social inclusion. This framework served not only as a useful guide but also as a basis for the comparative studies and databases. 74. Despite difficulties arising from factors beyond its control, the project seemed able to respond to the changing needs of the beneficiaries and the management structures to contribute to effective implementation. For example, it was not easy to make contact with government stakeholders for the consultative meetings and the presidential elections scheduled in several countries were an added difficulty. In order to resolve this impasse, ECLAC took account of the electoral process when arranging the consultative meetings and sought the collaboration of the Secretariat for Central American Social Integration (SISCA), a body with close ties to regional governmental institutions. In addition, the consultative meetings were able to gauge the particularities and needs of each country. The project team could then identify opportunities for technical cooperation in the area of the social inclusion of youth. 75. The interviews confirmed that the tasks were efficiently divided up between the ECLAC bodies. The Social Development Division at ECLAC headquarters in Santiago led the implementation in close collaboration with the ECLAC subregional headquarters in Mexico. The interviewees affirmed that this collaboration yielded excellent results and that, in particular, the joint design and reporting were an example of the efficient use of resources. Project managers agreed that the division of tasks was complementary and that the activities were well coordinated. In preparing the book, the Social Development Division exploited synergies and maximized collaboration within ECLAC by All sources of information confirmed that the activities were implemented as planned without any significant delays (finding 11). The interviews confirmed the efficient division of tasks within ECLAC (finding 12). Project managers and beneficiaries confirmed that the collaboration between ECLAC and the different counterparts had been outstanding (finding 13). Both project managers and beneficiaries affirmed that the project responded to the changing needs of the beneficiaries and contributed to the development of synergies within ECLAC and with other organizations (finding14). All project managers considered that the project used regionally-generated knowledge (finding 15). FINAL ASSESSMENT REPORT 21 allowing staff from other divisions to conduct research within this framework (the staff of the Social Development Unit in Mexico collaborated on the book). The project benefited greatly from the regional experience and networking of the Mexico office (for example, using SISCA assistance to identify the right stakeholders in each country). This was also confirmed by the survey results, which showed that all project managers were satisfied that the governance and management structures of the project facilitated its implementation. This cooperation is viewed as one of the main successes of the project. Table 10 Project managers’ perception of the governance and management structures Do you think that the governance and management structures of the project facilitated its implementation? Yes: 5 respondents (100%) Source: Prepared by the author. 76. All five project managers surveyed and interviewed considered that the project helped to develop synergies within ECLAC and with other organizations (such as the United Nations Population Fund (UNFPA), the Department of Economic and Social Affairs of the United Nations, SISCA, the Ibero- American Youth Organization (OIJ) and the Latin American Council of Social Sciences (CLACSO) Likewise, all of them felt that the project used knowledge generated regionally. Although not directly targeted by the project, civil society and organized youth participated to some extent in the activities. For example, over 51% of the beneficiaries (36 out of 70 respondents) agreed that the publications incorporated the views of civil society (less than 13% thought otherwise). The interviews pointed to the same conclusions. Clearly, the project must have enhanced the dialogue between governments and civil society. Figure 3 Beneficiaries’ perception of the incorporation of the views of civil society into the publications (Percentages) Source: Prepared by the author. 51 13 2 34 Do you think that the civil societys point of view was sufficiently incorporated into the publications? Yes No No, it was not necessary / relevant Without sufficient knowledge to answer FINAL ASSESSMENT REPORT 22 3.2.2 RESOURCE MANAGEMENT 77. The project did not suffer any significant delays and the activities were implemented mostly as planned (with some slight delays occurring at the beginning, mainly due to changes in government). Most stakeholders and project managers confirmed that effective coordination was crucial for success. 78. Over 97% of the budget was committed by the end of 2015, with real expenditures accounting for over 93%. The expenditure indicated that the funds were properly allocated to the expected item. The remaining balance was partly explained by the funds reserved for this final evaluation. The interviewees acknowledged that collaboration with other organizations (such as SISCA and OIJ had yielded significant efficiency gains. Table 11 Budget implementation Total Allotment Expenditure Commitment Total committed Balance General temporary assistance 39,095 39,031 ... 39,031 64 Consultants and expert groups 162,235 151,735 10,500 162,235 ... Staff travel 69,590 68,057 ... 68,057 1,534 Contractual services 30,953 28,318 ... 28,318 2,635 Operating expenses 11,000 8,841 ... 8,841 2,159 Fellowships, grants and contributions 187,127 169,095 11,268 180,363 6,763 Total 500,000 465,077 21,768 486,845 13,155 Source: Financial Report, March 2016. 79. Following the consultative meetings, the budget was adjusted in November 2014 to meet the needs identified through the consultations (i.e. to compile and share tools and best practices for a more effective policy on the social inclusion of youth —a better toolkit). The redistribution of funds also reflected the national counterparts’ commitment in support of the organization of the meetings. 3.2.3 ACTIVITY/OUTPUT REALIZATION 80. For the most part, the activities were implemented as planned. As mentioned above, they were modified slightly to respond better to the needs identified. The consultative meetings highlighted the importance of providing the beneficiaries with practical tools and facilitating their use as reference material and guidelines. All sources of information confirmed that the project was implemented as planned with some minor modifications in line with beneficiaries’ needs (finding 18). Most stakeholders considered that the different activities and outputs were of high quality and that ECLAC support had been satisfactory (finding 19). Some beneficiaries considered that a broader dissemination of the activities would have increased participation and would probably have increased the benefits of the project (finding 20). All sources of information confirmed that the resources were used as planned (finding16). All project managers believed that the management structure allowed implementation to be smoother and more efficient (finding 17). FINAL ASSESSMENT REPORT 23 81. According to the project’s terminal report, six consultative meetings were held (activity 1.1) with 170 participants from government entities (including national youth authorities), the United Nations system and civil society.26 The level of satisfaction was very high, as 75% of the participants indicated that the meetings were “excellent” or “very good” in contributing to their ability to analyse the social inclusion of youth, while 93% indicated that the meetings provided new knowledge for the design of participatory public policies. These meetings were seen as a first step towards building a network of policymakers, experts, and representatives of national youth and civil society organizations. The activities may therefore be recognized as complementary and as reinforcing the internal coherence of the project. 82. Although not all the participating countries were reached, the project did provide technical assistance (activity 1.2). Two missions were held in Costa Rica (May 2014): the first to assist the Vice-Ministry of Youth in developing indicators to measure the progress of the recently adopted National Youth Policy (Política Pública de la Persona Joven 2014-2019)27 and the second to advise the Council of the Young Person (Consejo de la Persona Joven, CPJ) on ways of improving their online platform28 . A technical assistance meeting was held in April 2015 with the Salvadoran National Institute of Youth to discuss their online youth platform and exploit potential synergies. 83. The regional study and national case studies were completed (activity 1.3). The regional study was published online as an ECLAC book in October 2015.29 It has been downloaded over 2,500 times and a short video clip has been produced and published online to present the main findings (2,086 complete views). The research and investigation that were conducted for the national case studies on Nicaragua and El Salvador culminated in two publications30 and two additional publications on the situation of youth in Central America31 (resulting in a total of 856 downloads). Lastly, a toolkit for youth policy design, based, in part, on these publications was prepared32 (1,018 downloads). 84. In 2015, six national workshops 33 (193 participants) and two subregional workshops 34 (100 participants) were organized as part of the project (activity 1.4). According to the final report, most of the participants (78%-95%) rated the different events as “excellent” or “very good” insofar as they strengthened their capacity to analyse issues relating to the social inclusion of youth. 26 Two consultative meetings were held in 2013 (El Salvador, Panama) and four in 2014 (Honduras, Nicaragua, Guatemala and Costa Rica). 27 Press release of the Ministry of Culture and Youth on the technical cooperation visit by the United Nations ECLAC team: https://www.facebook.com/notes/prensa-ministerio-de-cultura/mcj-inicia-cambios-en-procura-de-modelo-de-desarrollo-para- juventud-del-pa%C3%ADs/840844232595824; Report with recommendations for the Council of the Young Person (Costa Rica), following technical cooperation mission. https://drive.google.com/file/d/0ByegqWWz9yjhRFptM2k1WkRQRzg/view. 28 Council of the Young Person (Consejo de la Persona Joven) of Costa Rica, Social media announcement of technical cooperation visit by the United Nations ECLAC team: http://mcj.go.cr/actualidad/noticias/2014/abril/noticias/consecutivo196.aspx. 29 “Youth: realities and challenges for development with equality” http://repositorio.cepal.org/bitstream/handle/11362/38978/ S1500718_es.pdf?sequence=4. 30 “Las pandillas en El Salvador: propuestas y desafíos para la inclusión social juvenil en contextos de violencia urbana” http://www.cepal.org/es/publicaciones/39362-pandillas-salvador-propuestas-desafios-la-inclusion-social-juvenil-contextos. “Un extremo de nosotros: Lo público y la paz en El Salvador y Nicaragua” http://www.cepal.org/es/publicaciones/39828-un- extremo-nosotros-lo-publico-la-paz-salvador-nicaragua. 31 “Las juventudes centroamericanas en contextos de inseguridad y violencia: realidades y retos para su inclusión social” http://www.cepal.org/es/publicaciones/39229-juventudes-centroamericanas-contextos-inseguridad-violencia-realidades-retos-su. “Políticas y prácticas de prevención de la violencia que afecta a la juventud en Centroamérica - Análisis comparativo”, http://www.cepal.org/es/publicaciones/39820-politicas-practicas-prevencion-la-violencia-que-afecta-la-juventud-centroamerica. 32 “Hacia la inclusión social juvenil: Herramientas para el análisis y el diseño de políticas”, http://repositorio.cepal.org/bitstream/ handle/11362/39001/S1501236_es.pdf?sequence=1. 33 Costa Rica and Honduras (April); Guatemala and Nicaragua (May) and El Salvador and Panama (June). 34 Chile (August) with one or two participants from national youth authorities in 11 countries (Argentina, Brazil, Chile, Colombia, Cuba, Ecuador, Mexico, Paraguay, Peru, Plurinational State of Bolivia and Uruguay) together with representatives of SISCA and OIJ; and Dominican Republic (September) with one or two participants from national youth authorities in six countries (Costa Rica, Dominican Republic, El Salvador, Guatemala, Honduras and Panama ), again, along with representatives of SISCA and OIJ. FINAL ASSESSMENT REPORT 24 85. October 2015 saw the launch of an online youth platform (JUVeLAC),35 a one-stop shop for information on the social inclusion of youth in the region (for example, quantitative indicators, public policies and programmes in relevant areas and a research guide) (activities 2.1 and 2.2). Users register on the JUVeLAC website to share information about their policies and programmes, to participate in online webinars on various aspects of youth development in the region, and to join a regional network of stakeholders interested in youth. According to the final report, the JUVeLAC portal had had over 12,390 visitors up to 1 March 2016 and almost 200 people had registered with the network. 86. However, it is less clear if the project managed to establish a network of policymakers, experts, and representatives of national youth and civil society organizations and a forum for exchanging experiences, policies, and programmes geared to addressing violence among youth in Central America (activity 2.3). For example, none of the interviewees were registered. Nevertheless, it is clear that the different activities have encouraged the exchange of information, raised awareness and fostered communication. The project was completed only a few months ago and building a sustainable network is a long-term undertaking. The follow-up activities to be implemented will be crucial to consolidating this effort. JUVeLAC, in particular, has great potential. 87. In October 2015, a final seminar was organized in El Salvador to discuss the social inclusion of youth from a rights perspective, encompassing the fields of education, employment, health, violence, political participation and culture, with gender and racial or ethnic origin as cross-cutting themes. Over 150 persons attended on each of the two days (activity 2.4). Of the participants responding to the survey, 81% rated the seminar as “excellent” or “very good” insofar as it stregthened their capacity to analyse issues relating to the social inclusion of youth. 88. This positive picture was confirmed in both the surveys and the interviews. It was noted that the publications were based on thorough research and updated information. Of the 72 beneficiaries who responded, 82% thought that the events were “very relevant” (50%) or “relevant” (32%) to the situation in their country. Only one respondent thought that they were “not relevant”. The events were said to be “consistent with the priorities on the national policy”. As is often the case with ECLAC projects, several beneficiaries valued the opportunity to learn from the experiences of other countries. It was also mentioned that these events needed to be timed better and should be disseminated more widely in order to increase participation. Table 12 Beneficiaries’ perceptions of the activities Are you satisfied with the subject matter covered in the events? Very satisfied: 31 respondents (43%) Satisfied: 29 respondents (40%) Somewhat satisfied: 10 respondents (14%) Dissatisfied: 2 (3%) Question skipped by 10 respondents Do you consider that the publications are of high quality? Yes: 52 respondents (74%) No: 1 respondent (1%) Without sufficient knowledge to answer: 17 respondents (24%) Question skipped by 12 respondents Do you consider that the JUVeLAC contains relevant and updated information? Yes: 29 respondents (83%) No: 6 respondents (17%) Question skipped by 47 respondents Are you satisfied with the quality of the technical assistance? Yes: 1 respondent (100%) Question skipped by 81 respondents Source: Prepared by the author. 35 http://dds.cepal.org/juvelac. FINAL ASSESSMENT REPORT 25 3.3 EFFECTIVENESS 3.3.1 STRENGTHENED CAPACITY (EXPECTED ACCOMPLISHMENT 1) 89. The first expected accomplishment of the project was to strengthen the capacity of Central American governments to assess and design more effective policies to address youth exclusion and violence. As mentioned in section 3.2, direct beneficiaries (such as participants in the events) considered that the project strengthened their ability to analyse the social inclusion of youth (see the responses to the questionnaires circulated to the participants at the events). 90. The above-mentioned accomplishment was confirmed by both the interviews and the survey. Most of the beneficiaries (82% or 59 out of 72 respondents) now use the knowledge obtained at the events in their daily work. Only four respondents do not use it while nine did not have sufficient knowledge to answer. In line with the project focus, several respondents noted that it was useful for purposes of comparison and for inter-institutional coordination. Some stakeholders obtained information that served to enhance contact with youth networks and to conduct needs assessments. Similar answers were provided during the interviews, for example, the events were said to have provided valuable knowledge on experiences in other countries, which were replicated to some extent in Honduras. The toolkit was also mentioned as a very useful instrument. 91. Similar responses were obtained when the beneficiaries were asked to what extent the activities (events, publications and technical assistance) had enhanced their capacity: Figure 4 Beneficiaries’ capacities enhanced Source: Prepared by the author. Most beneficiaries considered that the various activities increased their knowledge and skills (finding 21). Most stakeholders felt that the project strengthened the capacity of Central American governments to assess and design more effective policies to address the social exclusion of youth and violence within this age group (finding 22). All sources of information confirmed that at least two new policies were implemented to further the inclusion of youth in Central America (finding 23). Most stakeholders pointed out that these two initiatives were among several others in at least three countries to have benefited from the project activities and ECLAC support (finding 24). Project managers confirmed that all of the countries that participated in the project had policies in place to deal with youth exclusion and/or violence (finding 25). 17 16 11 14 30 25 25 30 8 13 16 9 2 2 3 2 12 13 14 14 0 10 20 30 40 50 60 70 80 To process and / or analyse youth-related information To establish links between social inclusion and youth- related policies To design and implement effective programmes and policies to promote social inclusion of youth To devise and interpret successful youth and social inclusion related policies N um be r of r es po nd en ts A lot Quite Little Not at all Without sufficient knowledge to answer FINAL ASSESSMENT REPORT 26 92. Over 43% of the beneficiaries (32 out of 74 respondents) considered that the activities had contributed (“a lot” or “significantly”) to enhancing the capacities of governments of the region to design and promote more effective youth policies. Some 27% (20 respondents) thought that the contribution was less important and less than 3% (only 2 respondents) believed that there was no contribution. The activities were said to have increased awareness of certain issues and to have delivered useful tools for improving policy design and strengthening policymaker networks. Figure 5 Governments’ capacities enhanced (Percentages) Source: Prepared by the author. INCREASED NUMBER OF INITIATIVES, POLICIES OR PROGRAMMES 93. Almost 18% of the beneficiaries (13 out of 74 respondents) considered that there were new policy initiatives or programmes for dealing with youth exclusion and violence that had benefited from the results of the project. Almost 38% (28 respondents) felt that there were none that benefited and the rest did not have sufficient knowledge to answer. The final reports revealed that only four countries had policies in place on the social exclusion of youth and/or violence at the outset. During the project execution, two new policies to further youth inclusion were implemented in Central America: a National Youth Employment plan in El Salvador and a National Youth Policy in Costa Rica. Both the interviewees and the respondents to the survey felt that these two policies benefited from the project activities. For example, one beneficiary mentioned that the project results were used to develop the universal assistance perspective adopted by the National Youth Employment plan in El Salvador. There is also evidence of ECLAC support for the National Youth Policy in Costa Rica (see page four of the “Política Pública de la Persona Joven 2014-2019 y su Plan de Acción”).36 During the interviews, the project results were said to have been used in the document “Política Nacional de Prevención” (Honduras) and in a new programme to restructure pedagogical centres. Currently, all the countries that participated in the project have policies in place on the social exclusion of youth and/or youth violence. 36 http://www.cpj.go.cr/images/POLITICA_PUBLICA_Y_PLAN_DE_ACCION_2014- 2019_WEB.pdf. 13 30 27 3 27 To what extent have the project activities contributed to enhance goverments capacity to promote and design more effective youth policies? A lot Significantly Little Not at all Without sufficient knowledge to answer FINAL ASSESSMENT REPORT 27 Figure 6 Beneficiaries’ information/opinion on new policies and programmes (Percentages) Source: Prepared by the author. 94. In addition, the interviewees confirmed that, as a direct consequence of the project, the six Central American countries targeted by the project have embarked on initial discussions to establish a subregional agenda for youth in Central America. The ECLAC subregional headquarters in Mexico and SISCA will follow up and support the negotiations. 3.3.2 ENHANCED SHARING OF DATA AND INFORMATION (EXPECTED ACCOMPLISHMENT 2) 95. The second expected accomplishment of the project was enhanced sharing of data and information on critical economic and social challenges for youth in Latin America and the Caribbean to support national and regional decision-making. A. ENHANCED AVAILABILITY OF TIMELY DATA 96. Almost 65% of the beneficiaries (48 out of 74 respondents) considered that the activities contributed either “a lot” or “significantly” to increasing and/or improving the exchange of information about youth exclusion and violence in the region. Only 16% (12 respondents) thought that they contributed “little” and 1% (one respondent) considered that they contributed “not at all”. In this sense, as the final report indicates, the availability to stakeholders of timely data and information on relevant social and economic challenges facing youth has improved as a result of the project. To date, close to 100 indicators on youth development (drawing from nine data sources) are available on the JUVeLAC portal.37 Most of these 37 Sixteen on education, 16 on health, 20 on political participation, 12 on employment, 6 on violence, 5 on culture, 4 on drug prevention, 2 on social protection, and 15 on information and communications technologies. Most stakeholders agreed that the project had enhanced data and information sharing on critical economic and social challenges facing youth in the region (finding 26). The publications are widely known among beneficiaries and often used by them (finding 27). The JUVeLAC portal is online and available to the public. Most stakeholders believe that it will gain momentum and that it has the potential to further enhance collaboration on information sharing within the region (finding 28). 18 38 44 Do you know any new policy initiative or programme related to youth exclusion and violence that has benefited from the project results? Yes No Without sufficient knowledge to answer FINAL ASSESSMENT REPORT 28 indicators can be disaggregated by sex, rural/urban residence, age group and socioeconomic level, to allow for a more nuanced analysis. Figure 7 Beneficiaries’ perception of information sharing (Percentages) Source: Prepared by the author. B. INCREASED NUMBER OF COUNTRIES FOR WHICH INFORMATION IS ANALYSED 97. Over 85% of the beneficiaries (70 out of 82 respondents to the survey) were familiar with at least one of the publications prepared as part of the project. Some 40% of them used these publications “often” or “very often” (28 respondents). Over 31% (22 respondents) used them “rarely” and 14% (10 respondents) did not use them at all. These publications have been used, for example, to support youth networks, to influence policy, as reference in other research or at conferences. Figure 8 Use of the publications by beneficiaries (Percentages) Source: Prepared by the author. 11 29 32 14 14 Have you used the publications in your work? Often Very often Rarely Never Without sufficient knowledge to answer 24 41 16 1 18 To what extent has the project contributed to increase and/or enhance data and information sharing on critical economic and social challenges for youth? A lot Significantly Little Not at all Without sufficient knowledge to answer FINAL ASSESSMENT REPORT 29 98. Over 51% of the beneficiaries (36 out of 70 respondents) were familiar with the JUVeLAC portal. The same percentage had used the portal’s information “often or very often” in their work (18 out of 35 respondents) while 40% (14 respondents) have used it rarely and less than 6% (2) have never used it. Although less than 9% (only 3 out of 35 respondents) participated actively on the portal, it probably should not be considered as a very low rate as the portal was launched only recently. According to the final report, close to 200 stakeholders interested in youth development were participating in JUVeLAC, including government authorities, United Nations staff, academics and members of civil society. Information is available to users on over 560 programmes and the two most visited areas are those that relate to statistical queries and queries on policies and programmes. According to the final report, this suggests that information on innovative programmes addressing youth was being analysed. During the interviews, it was confirmed that most interviewees were familiar with JUVeLAC, some had downloaded and used information but none was registered. Most of them said that it provided timely data on social and economic challenges facing youth. Figure 9 Use of JUVeLAC by beneficiaries (Percentages) Source: Prepared by the author. 3.4 SUSTAINABILITY 3.4.1 IMPACT AND MULTIPLIER EFFECTS All sources of information indicated that the project had contributed to long-term processes such as the design of new policies and programmes in several countries and that the activities were expected to generate multiplier effects (finding 29). There is evidence that ECLAC has implemented adequate mechanisms to ensure the sustainability of the project by, for example, collaborating with strategic partners, promoting a network of practitioners and enhancing local capacities (finding 30). The evidence points to the need for more assistance, for stronger ownership by beneficiaries and for increased political support (finding 31). The interviews have confirmed that ECLAC continues its support in line with the results of the project (finding 32). 5 4640 6 3 Have you used information from JUVeLAC in your work? Very often Often Rarely Never Without sufficient knowledge to answer FINAL ASSESSMENT REPORT 30 99. As described above, most stakeholders think that the project has contributed to long-term processes such as the design of new policies and programmes in several countries (see section 3.3). This seems particularly encouraging given the size of the project (in terms of the resources utilized) and its recent completion. The project has also enhanced the capacities of the beneficiaries (see section 3.3) and the final report takes note that the exchange of experiences between countries supported by SISCA has created a network of practitioners in the field of the social inclusion of youth that will continue beyond the formal conclusion of the activities. It was indeed confirmed during the interviews that the leaders of the national youth entities were in preliminary discussions to establish a subregional youth agenda, similar to the Youth Development Action Plan of the Caribbean Community (CARICOM). 100. Similarly, almost 72% (28 out of 39) of the responses to the survey confirmed that ECLAC had “implemented adequate mechanisms to ensure the sustainability of the project”. The majority also considered that the project or its activities would have multiplier effects: over 67% (31 out of 46 respondents) when asked about JUVeLAC, over 62% (31 out of 50) when asked about technical assistance, over 70% (43 out of 61) when asked about the publications and 74% (40 out of 54) when asked about the events. Moreover, almost 89% (55 out of 62) felt that “the project and its activities are potentially replicable”. The following figure summarizes the responses: Figure 10 Beneficiaries’ perceptions of sustainability Source: Prepared by the author. 101. Nevertheless, as illustrated in figure 10, the results of the survey show that less than 54% of the respondents (29 out of 54) considered that the political support existed or that the environment would be conducive to further action along the same lines. Of the respondents, 25 (46%) considered that failure to address the problem in an integrated manner suggested a lack of, or weak, political support. This, together with the survey responses, indicates a need for further support. Almost 45% of the respondents (20 out of 45) think that beneficiaries had not taken ownership of the project. 102. During the interviews, all project managers stressed that the Commission’s involvement did not end with the termination of the contract. In keeping with its mandate, ECLAC continues, albeit in a more limited way, providing support to strengthen the capacities of governmental and non-governmental organizations (for example, by taking advantage of in-country missions to hold discussions on the project results). 25 29 19 55 40 43 31 31 28 20 25 21 7 14 18 19 15 11 29 20 34 12 20 13 24 28 35 0 10 20 30 40 50 60 70 80 There is appropiation of the project by beneficiaries There exists political support to continue with similar actions Partnerships have been established to ensure sustainability of results The project or its activities are potentially replicable The events had or will have a multiplier effect The publications had or will have a multiplier effect The technical assistance had or will have a multiplier effect JUVeLAC had or will have a multiplier effect ECLAC has implemented adequate mechanism to ensure the sustainability of the project Agree (totally or partly) Disagree (totally or partly) Without sufficient knowledge to answer FINAL ASSESSMENT REPORT 31 103. Further technical assistance is currently being explored for El Salvador. Later in 2016, the Focal Point on Youth in the Department of Economic and Social Affairs will organize training workshops with the same national counterparts of the project. These workshops will provide follow-up by monitoring and evaluating youth policies in the region. Crucial to sustaining the effects in the long term is the need for individual capacities to be translated into enhanced institutional capacities. ECLAC has been requested to provide assistance and technical support based on the experience gained with the project. In addition, the United Nations Educational, Scientific and Cultural Organization (UNESCO) and CLACSO are conducting workshops with youth parliamentarians and have also requested ECLAC to provide technical support. 3.4.2 DISSEMINATION 104. The project made a huge effort to disseminate outputs and results. The regional and national workshops were crucial to this strategy as they allowed the dissemination not only of experiences but also of country specific methodologies and comparative findings: 193 participants attended the six national workshops and 100 participants, the two subregional workshops. ECLAC also sought to disseminate the project publications. As mentioned above, over 85% of the respondents to the survey were familiar with at least one of these publications. The regional study (published as an ECLAC book) has been downloaded over 2,500 times and the presentation video has been viewed completely 2,086 times. The other four publications have been downloaded 856 times and the toolkit, 1,018 times. During the interviews, the success of this strategy was confirmed by beneficiaries and project managers alike. The former pointed out that the knowledge gained at the workshops and the content of the publications have been discussed at numerous conferences and seminars. 105. Despite the assistance provided by ECLAC, several stakeholders solicited additional support to continue with the dissemination of results. They advocated giving more publicity to the work accomplished and disseminating the publications to a wider readership. The JUVeLAC portal is expected to be a valuable tool for this dissemination. As mentioned in the final report, partnering with the Ibero-American Youth Organization (OIJ) and interested national counterparts, and the inclusion of the update of youth-related statistical indicators in the Social Development Division’s work plan will ensure the sustainability and pertinence of JUVeLAC. The maintenance costs of the portal have been incorporated into the regular ECLAC budget, along with provision for the recruitment of trainees to update the information. Project managers have already presented JUVeLAC in several forums. 106. According to the project document, sustainability would be encouraged by developing the online databases, by fostering the exchange of experiences between countries to forge links between practitioners and by seeking additional funding and partners. Although the three initiatives could add value in terms of sustainability, the design cannot be said to assemble a comprehensive strategy for ensuring sustainability. Not surprisingly, little has been achieved in terms of additional funding or partners due to the lack of concrete guidelines. Linking sustainability with the exchange of experiences is too vague and limited as a sustainability strategy. The most coherent measure was guaranteeing the maintenance of the databases under the regular ECLAC budget (and other sources). All sources of information indicated that the outputs and results were disseminated successfully during project implementation (finding 33). The regional and national workshops enabled participants to share experiences, country-specific methodologies and comparative findings (finding 34). The data indicate that the work done needs to be more widely disseminated (finding 35). FINAL ASSESSMENT REPORT 32 4. CONCLUSIONS RELEVANCE AND DESIGN 107. The project responded to concerns expressed in Latin America and the Caribbean and in participating countries, in particular to the need to strengthen the capacity within countries to promote the social inclusion of youth against a backdrop of violence and insecurity. The project design identified the main bottlenecks as being the lack of skills, information and practical tools, which hindered decision-makers from designing effective and targeted policies and programmes. Although the needs of each country had not been thoroughly analysed, the project adapted its activities to each specific context. Thus, most beneficiaries were satisfied that the methodology and the implemented activities were relevant for their work and the national context (conclusion 1, based on findings 1, 2 and 3). 108. The project was fully in line with the resolutions and agreements reached at several United Nations Conferences and Summits and clearly contributed to the ECLAC mandate by coordinating actions towards economic development and reinforcing economic relationships within the region. However, the project seemed to overlook gender-related issues. As a result, the design was not gender- responsive. Nevertheless, the human rights and gender perspective were partially integrated into the project activities and products (conclusion 2, based on findings 4, 5 and 6). 109. The design of the project did include a description of the various stakeholders. However, a more thorough analysis would have allowed a better understanding of the rules and incentives that governed the implementation of policy reform and a clearer definition of the stakeholders’ roles. Although the analysis revealed credible cause-effect relationships, which proved that the project was capable of addressing the challenges, the project design would have benefited from a more thorough description of its logic (for example, more comprehensive problem and objective trees). Ideally, stakeholders should have been engaged in a process to visualize what the future would look like if the problems were resolved, including testing crucial cause-effect assumptions (conclusion 3 based on findings 7, 8 and 9). 110. The simplified logical framework was useful at the project proposal stage but it should have been improved so as to serve as an effective management tool (conclusion 4, based on finding 10). EFFICIENCY 111. The project management structure contributed to effective implementation and responded to the changing needs of the beneficiaries. Most information sources indicated that the collaboration (including the coordination between implementing bodies and their counterparts) was outstanding. This collaboration was seen as one of the main strengths and successes of the project as it helped to build synergies. The management structure of the project together with the collaboration within ECLAC and with other partners facilitated implementation and generated efficiency gains (conclusion 5, based on findings 11, 12, 13, 14, 15, 16 and 17). 112. The project was implemented as planned and additional activities were implemented at the request of beneficiaries. Most stakeholders were satisfied with the Commission’s support and considered that the activities and outputs were of a high quality. Others thought that broader dissemination would have encouraged a stronger participation and might have augmented the benefits of the project (conclusion 6, based on findings 18, 19, 20 and 35). FINAL ASSESSMENT REPORT 33 EFFECTIVENESS 113. The project increased the participants’ knowledge and skills and, in turn, strengthened the capacity of Central American governments to design and assess more effective policies on youth exclusion and violence. By opening up public dialogue and promoting face-to-face interaction, the project helped deepen the understanding of national governments and, to some extent, civil society organizations, enabling them to analyse and design more effective policies and programmes (conclusion 7, based on findings 21 and 22). 114. At least two new policies were implemented to foster youth inclusion in Central America. These were among several initiatives (launched in at least three countries) that benefited from the project activities and ECLAC support. Currently, each of the countries that participated in the project has in place policies on youth exclusion and/or violence (conclusion 8, based on findings 23, 24 and 25). 115. Most stakeholders agreed that the project contributed to enhancing data- and information-sharing on critical economic and social challenges for youth in the region. The JUVeLAC portal is online and available to the public. It has started to gain momentum and has the potential to further improve collaboration within the region (conclusion 9, based on findings 26, 27 and 28). SUSTAINABILITY 116. Despite its limited resources, the project has contributed to long-term processes such as the design of new policies and programmes in several countries. The activities are already generating multiplier effects such as follow-up activities implemented by ECLAC partners (conclusion 10, based on findings 29 and 30). 117. ECLAC has sought to ensure the sustainability of the results by implementing a successful strategy to disseminate outputs and results and by promoting the exchange of experiences and comparative findings at the regional and national workshops. Beneficiaries should, however, take stronger ownership of the project results and every effort should be made to increase political support. Data also indicate that the work done needs to be disseminated more widely, particularly among civil society organizations (conclusion 11, based on findings 31, 33, 34 and 35). 118. Although sustainability was not fully addressed at the project design phase, ECLAC was aware of this need and will continue to sustain the impacts over the long term and to ensure that individual capacities are transformed into enhanced institutional capacity. The JUVeLAC online portal should play a crucial role in this effort (conclusion 12, based on finding 32). FINAL ASSESSMENT REPORT 34 5. LESSONS LEARNED 119. The close collaboration between the two ECLAC offices was broadly recognized as a crucial element in the successful implementation of the project. As highlighted in the final report and confirmed during the interviews, this close and consistent collaboration was achieved by setting clear goals, ensuring open and frequent communication, and establishing roles and responsibilities that matched each team member’s particular strengths (lesson 1). 120. ECLAC is an excellence-driven organization with an outstanding record and reputation in the region. It has the potential to bring about significant efficiency gains by encouraging dialogue, facilitating access to cutting-edge knowledge and attracting additional contributions (in cash or in kind) to the projects. In keeping with its mandate, ECLAC promotes multilateral dialogue, knowledge-sharing and networking at the regional level, and fosters joint intra- and interregional cooperation. It has been working closely with other multilateral organizations (such as SISCA and OIJ) with a view to a wider dissemination of the project results and the sustainability of the project (lesson 2). 121. Throughout the project, the Development Account acted as a channel through which member countries could tap into the normative and analytical expertise of the United Nations Secretariat. By offering distinctive knowledge and skills rarely accessible through other development partners, ECLAC was well placed to act as a game changer, promoting dialogue between government officials and civil society groups and the exchange of knowledge and transferr of skills between countries. In this context, ECLAC was perceived as a key actor contributing to a shared United Nations vision (lesson 3). 122. The publication of the book increased visibility and involved collaboration and constructive peer- review sessions. Staff from the Social Development Division and the Social Development Unit of the ECLAC subregional headquarters in Mexico drafted several chapters in their area of expertise. The final report describes it as an excellent, insightful book that showcases these offices’ grasp of various dimensions of social inclusion as it relates to youth (lesson 4). FINAL ASSESSMENT REPORT 35 6. RECOMMENDATIONS 124. Based on the findings and conclusions arrived at in this assessment, the five recommendations put forward in this section are intended as practical measures for addressing the identified challenges. Some of these recommendations may, however, stretch the Commission’s current capacity. 125. The recommendations are directed primarily at ECLAC divisions and subregional headquarters, as well as the Department of Economic and Social Affairs, which are the main beneficiaries of this evaluation. Some recommendations concern other project partners whose involvement is crucial to bringing about the desired changes. A number of recommendations, if accepted and implemented, will also impact on the beneficiaries and their relationships with ECLAC. RECOMMENDATION 1 (BASED ON CONCLUSION 1) 126. Developing and maintaining an evaluative culture in an organization is often seen as the key to building more effective approaches to results-based management and evaluation. Projects designed to achieve complex change must therefore be underpinned by a robust theory of change. ToC is essential for demonstrating what has been achieved, for facilitating monitoring and for sharing information. It allows senior managers to challenge the logic of the project and the evidence gathered on performance in order to oversee the results-based management regime, thus, ensuring that the results are realistic, transparent and reliable. 127. The analysis should explain country and sector specificities (for example, different policy areas), even developing specific theories of change, if necessary. A systemic approach during the design makes it possible to investigate unintended effects (either positive or negative), power relationships and/or possible conflicts at the boundaries of the system. Different stakeholders should be involved in the identification of the most critical problems (including underlying causes) and credible cause-effect relationships. This should include identifying their different roles, positions, strengths, weaknesses and influences. This process is essential for building stakeholder consensus and forging the necessary partnerships to effectively address the problems identified, as well as assessing the roles that each stakeholder should play to ensure a more efficient and effective project design and implementation. 128. However, a solid results-based management (RBM) system rests on what is commonly referred to as a “life cycle”, where “results” are central to planning, implementation, monitoring and evaluation, reporting and ongoing decision-making. By focusing on “results” rather than “activities”, RBM articulates more clearly the vision and the support for expected results and monitors progress using indicators, targets and baselines. It is essential, therefore, that the project proposal include a robust and comprehensive logical framework matrix that spells out clear process, results and impact indicators that are specific, measurable, achievable, relevant and time-bound (SMART), including targets, baselines and means of verification, risks, assumptions and the role of partners. This would enhance both the design and the evaluability of each project. 129. The evaluator recommends that ECLAC divisions and subregional headquarters (with support from the Programme Planning and Evaluation Unit of the Programme Planning and Operations Division) provide ongoing training to managers and staff in the various aspects of results-based management, including ToC, logical framework approach, indicators and self-evaluation. To enhance the culture of evaluation and effective results-based management by providing ongoing training to managers and staff in the various aspects of this tool, including self-evaluation. FINAL ASSESSMENT REPORT 36 RECOMMENDATION 2 (BASED ON CONCLUSION 2) 130. There is broad agreement that gender-related issues should be mainstreamed in any development project. Target entry points should be identified for mainstreaming the gender perspective in ECLAC activities through advocacy, project and policy development, implementation, and monitoring and evaluation. 131. ECLAC divisions and subregional headquarters (with support from the Division for Gender Affairs) should include a comprehensive gender analysis in their project proposals in order to identify gender-specific roles and responsibilities, gender-related differences and the differences in impact on men and women. The identification of gender-specific measures would increase the effectiveness and impact of the project (and the gender balance achieved would benefit both men and women. In addition, the results would serve other development objectives, such as economic development and poverty reduction. 132. Many sectoral experts come from technical or scientific backgrounds and may have had little exposure to gender issues, which are more commonly raised in political and social contexts. Therefore, the concept of gender mainstreaming may not seem particularly relevant to their work. To address this issue, it is important to promote the value added of mainstreaming gender into their work, as well as its relevance to strengthening replication and sustainability. Inviting gender or human rights analysts from partner development agencies or representatives of gender-focused non-governmental organizations (NGOs) to be involved as stakeholders could be an effective way of ensuring an ongoing focus on this issue. RECOMMENDATION 3 (ON THE BASIS OF CONCLUSION 3) 133. In accordance with its mandate, the main ECLAC counterparts are the various government institutions in Latin American and Caribbean countries. ECLAC is, therefore, less used to working directly with civil society groups. The policymaking process is primarily about seeking a balance between the broader goals of equity and the welfare and interests of various groups. Once it is acknowledged that one project alone cannot address all related issues, it becomes clear that developing synergies and collaborating with different stakeholders and programmes during the project design, throughout implementation and after completion are of paramount importance. 134. The special capacities of civil society groups should be recognized and cooperation arrangements established to identify the key actors and ensure their participation in the activities. Therefore, ECLAC divisions and subregional headquarters will need to undertake a thorough stakeholder analysis from the start of the project in order to include specific activities targeting civil society. Focus group discussions and consultations with various stakeholders may suffice, but it is recommended that the different stakeholders be brought together in one place. To maximize the benefits to civil society organizations by undertaking a thorough stakeholder analysis at project outset and by including targeted activities. To ensure that gender-related issues are mainstreamed through a comprehensive gender analysis undertaken at the outset and including targeted activities. FINAL ASSESSMENT REPORT 37 RECOMMENDATION 4 (BASED ON CONCLUSION 4) 135. Failure to review the results and the underlying theory of change on a regular basis might lead to confusing behaviour and inaccuracies. The system should focus on the substantive development of intended results (outputs and outcomes). It should provide real-time answers about the outcomes rather than delaying communication until a project is completed and the outputs produced. 136. Self-evaluation, in principle, provides information about many more projects than could possibly be processed by independent evaluators. It is also broadly accepted that if managers and staff are involved in the process of measuring results and analysing information, they are more likely to see the value of such efforts and to make use of the information gathered. Seeing positive results in terms of better design or delivery will stimulate interest in learning from such information. However, if evaluation and monitoring are perceived primarily as a means of checking up on managers and staff, then the learning is less likely to be supported (John Mayne, 2008). 137. It is recommended that ECLAC divisions and subregional headquarters (with support from the Programme Planning and Evaluation Unit in the Programme Planning and Operations Division) establish a system for checking the evaluability of project proposals. This should involve monitoring and evaluation at the planning stage (including conducting regular monitoring of results and/or mid- term evaluations). In this framework, structured learning events should be routinely organized to discuss the future direction, using available results data and information. 138. An independent validation should be conducted by the Programme Planning and Evaluation Unit into the system to counteract the natural bias of self-evaluation. For example, the project team should submit either an evaluability report (including credibility of the intervention theory and causal attribution and clarity of the indicators) or a completion report (including self-ratings of outcomes and the performance of ECLAC) At the inception, some organizations routinely commission the development of a monitoring and evaluation framework which addresses evaluability questions. This type of analysis would also identify possible weaknesses and any capacity-building required by ECLAC. RECOMMENDATION 5 (BASED ON CONCLUSIONS 6, 9, 10, 11 AND 12) 139. The assurance of a lasting impact of the results and achievements in this type of project in terms of sustained access to knowledge and the enhanced technical capacity of beneficiaries is crucial. Funding cycles are rarely aligned with needs, imposing artificial timelines on programme phase-out. It is therefore sensible to develop a sustainability plan that considers how the project intends to withdraw its resources, while ensuring that achievement of the goals is not jeopardized and that progress towards those goals will continue. 140. The evaluation has demonstrated that ECLAC is already making an effort to ensure sustainability. ECLAC divisions and subregional headquarters are advised to outline a strategy to advertise the JUVeLAC portal widely and link it with other initiatives. Similarly, future projects should outline an explicit “exit strategy” at the outset and develop it further during the project implementation. In the context of the current evaluation, the term “exit strategy” refers to the end of project funding. Its To maximize the sustainability of project impacts by designing an “exit strategy” at the outset and/or during project implementation, including targeted activities. To strengthen the learning focus by regularly assessing project evaluability, implementing results- based monitoring and/or mid-term evaluations and organizing structured learning events. FINAL ASSESSMENT REPORT 38 aim should be to ensure that individual capacities are translated into enhanced institutional capacities. It should define the change from one type of assistance (for example, a Development Account project) to another (such as the regular ECLAC programme of work). Targeted activities would need to be included and the implemented dissemination activities with future initiatives by ECLAC and its to link the project results partners. Initially, judicious proposals as to how the project results might be further sustained should be articulated at least in the termination reports. FINAL ASSESSMENT REPORT 39 ANNEXES ANNEX 1 TERMS OF REFERENCE ANNEX 2 DOCUMENTS REVIEWED ANNEX 3 PARTICIPANTS ANALYSIS ANNEX 4 SIMPLIFIED LOGICAL FRAMEWORK ANNEX 5 EVALUATION MATRIX ANNEX 6 INTERVIEW GUIDELINES ANNEX 7 SURVEY QUESTIONNAIRE ANNEX 8 LIST OF INTERVIEWEES ANNEX 9 EVALUATOR’S REVISION MATRIX ANN T E R M S I. Introduc 1. This a and 5 Planni (PPBM regula streng Secre differ implem work. Evalua II. Assess 2. This a the d identi perta III. Object 3. The o of the attain 4. The p partic and i good poten 5. The le for the EX 1 O F R E F E Ass S tion ssessment is 4/474 of A ng, Aspects E). In this r, periodic thening of th tariat in gene ent oversight enting an e This is theref tion Unit (PP ment Topic ssessment is a esign, implem fying, analyz ining to violen ive of the As bjective of th project imp ed in relation roject object ularly in Cent nsecurity. The practices tha tial of replica ssons learned future plann R E N C E essment of t ocial inclusio and ins in accordance pril 2000, of the Budge context, the basis, coverin e evaluation ral and ECL bodies end valuation stra ore a discret EU) of ECLAC n end-of-cyc entation, an ing, and diss ce preventio sessment is assessment lementation a to its overal ive was to s ral America, evaluation t were deriv ting them to and good ing and impl TERMS he Developm n of youth in ecurity with with the G which endor t, the Monit General Ass g all areas function to AC in particu orsed by t tegy that inc ionary intern ’s Programme le review of a d monitoring eminating suc n. is to review nd more pa l objectives a trengthen the to promote e will place an ed from the other countrie practices in a ementation o OF REFEREN ent Account the context a focus on C eneral Assem sed the Reg oring of Imp embly reque of work un support and lar and with he General ludes period al evaluation Planning an regional pr of youth inc cessful youth the efficiency rticularly doc nd expected capacity o ffective socia important e implementat s. ctual project f ECLAC proje CE Project ROA of increasing entral Amer bly resolutio ulations and lementation a sted that pr der their pu inform the d in the normat Assembly, E ic evaluations managed b d Operations oject aiming lusion policie social inclusi , effectivene ument the re results as def f Latin Amer l inclusion of mphasis in i ion of the pr implementat cts. FINAL AS 254-8 violence ica ns 54/236 o Rules Gove nd the Meth ogrammes be rview. As pa ecision-makin ive recomme CLAC’s Exec of different y the Progra division (PPO at facilitating s and progr on initiatives, ss, relevance, sults and imp ined in the pr ican and Ca youth in the dentifying le oject, its sust ion will in tur SESSMENT RE f December rning Progra ods of Evalu evaluated rt of the ge g cycle in th ndations mad utive Secreta areas of EC mme Planning D). the use of d ammes as w particularly and sustaina act of the p oject docume ribbean cou context of vio ssons learned ainability an n be used as PORT 40 1999 mme ation on a neral e UN e by ry is LAC’s and ata in ell as those bility roject nt. ntries, lence and d the tools FINAL ASSESSMENT REPORT 41 IV. Background The Development Account 6. The Development Account (DA) was established by the General Assembly in 1997, as a mechanism to fund capacity development projects of the economic and social entities of the United Nations (UN). By building capacity on three levels, namely: (i) the individual; (ii) the organizational; and (iii) the enabling environment, the DA becomes a supportive vehicle for advancing the implementation of internationally agreed development goals (IADGs) and the outcomes of the UN conferences and summits. The DA adopts a medium to long-term approach in helping countries to better integrate social, economic and environmental policies and strategies in order to achieve inclusive and sustained economic growth, poverty eradication, and sustainable development. Projects financed from the DA aim at achieving development impact through building the socio- economic capacity of developing countries through collaboration at the national, sub-regional, regional and inter-regional levels. The DA provides a mechanism for promoting the exchange and transfer of skills, knowledge and good practices among target countries within and between different geographic regions, and through the cooperation with a wide range of partners in the broader development assistance community. It provides a bridge between in-country capacity development actors, on the one hand, and UN Secretariat entities, on the other. The latter offer distinctive skills and competencies in a broad range of economic and social issues that are often only marginally dealt with by other development partners at country level. For target countries, the DA provides a vehicle to tap into the normative and analytical expertise of the UN Secretariat and receive on-going policy support in the economic and social area, particularly in areas where such expertise does not reside in the capacities of the UN country teams. The DAs operational profile is further reinforced by the adoption of pilot approaches that test new ideas and eventually scale them up through supplementary funding, and the emphasis on integration of national expertise in the projects to ensure national ownership and sustainability of project outcomes. DA projects are being implemented by global and regional entities, cover all regions of the globe and focus on five thematic clusters1. Projects are programmed in tranches, which represent the Accounts programming cycle. The DA is funded from the Secretariats regular budget and the Economic Commission for Latin America and the Caribbean (ECLAC) is one of its 10 implementing entities. The UN Department of Economic and Social Affairs (DESA) provides overall management of the DA portfolio. 7. ECLAC undertakes internal assessments of each of its DA projects in accordance with DA requirements. Assessments are defined by ECLAC as brief end-of-project evaluation exercises aimed at assessing the relevance, efficiency, effectiveness and sustainability of project activities. They are undertaken as desk studies and consist of a document review, stakeholder survey, and a limited number of telephone- based interviews. The project 8. The project under evaluation is part of the projects approved under this account for the 8th Tranche (2012-2015), under the coordination of the Economic Commission for Latin America and The Caribbean (ECLAC), specifically by its Social Development Division and it Subregional Office in Mexico. 1 Development Account projects are implemented in the following thematic areas: advancement of women; population/countries in special needs; drug and crime prevention; environment and natural resources; governance and institution building; macroeconomic analysis, finance and external debt; science and technology for development; social development and social integration; statistics; sustainable development and human settlement; and trade. See also UN Development Account website: http://www.un.org/esa/devaccount/projects/active/theme.html. FINAL ASSESSMENT REPORT 42 9. The original duration of this project was of 3 years (2013-2015), having started activities in July 2013 and ending in December 2015 as originally planned. 10. The overall logic of the project against which results and impact will be assessed contains an overall objective and a set of expected accomplishments and indicators of achievement that will be used as signposts to assess its effectiveness and relevance. 11. The project’s objective as stated above is “to strengthen the capacity of Latin American and Caribbean countries, particularly in Central America, to promote effective social inclusion of youth in the context of violence and insecurity.”2 12. The expected accomplishments were defined as follows: (a) Strengthened capacity of Central American governments to assess and design more effective policies to confront youth exclusion and violence. (b) Enhanced sharing of data and information on critical economic and social challenges for youth in Latin America and the Caribbean to support national and regional decision-making. 13. To achieve the expected accomplishments above, the following activities were originally planned: • Organizing consultative meetings in Central America with UN Country Teams and national counterparts to discuss and gather information on youth social inclusion, human security, and violence prevention policies and assess ways to feed project results into UN Common Country Assessments (CCA). • Providing technical cooperation to strengthen the capacity of National Youth Institutes to process and use available economic and social data to: i) identify economic and social issues critical for youth development and social inclusion, ii) design public policies promoting youth inclusion, and iii) assess the effectiveness of policies implemented. This will include assistance provided via virtual distance learning tools, as well as in person. • Prepare one regional comparative study covering Latin America and the Caribbean and two national case studies covering Central American countries that provide a better understanding and knowledge of innovative policies and programmes addressing social inclusion among youth, with a particular focus on human security and violence prevention. The knowledge that is generated from these studies will be used to develop a toolkit that will form the basis of the training activity in the national and sub-regional workshops. • Organize six national and two sub-regional workshops to discuss and analyze with public and private authorities, effective policies and programmes oriented to social inclusion, human security, and violence prevention among youth. During these workshops participants will be trained on the use of a toolkit that will be developed based on the knowledge and experiences garnered through activities 1-3. • Creating a regional online database gathering valuable experiences on policies and programmes related to youth development–including violence prevention– relevant to meet the MDGs/IADGs. • Generating a regional online database with quantitative and qualitative data on economic and social issues critical for youth development and social inclusion. • Establishing a network of policy-makers, experts, and representatives from national youth and civil society organizations and a forum to exchange experiences, policies, and programmes addressing violence among youth in Central America. Electronic communications and knowledge sharing will be sustained by RISALC, the Network of Social Institutions in Latin America and the Caribbean (www.risalc.org). 2 See Annex 1: Project Document. FINAL ASSESSMENT REPORT 43 • Organizing a regional seminar to present and discuss findings and recommendations on social inclusion, human security, and violence prevention. 14. The budget for the project totalled US$ 500,000. Progress reports were prepared on a yearly basis. Stakeholder Analysis: 15. The primary stakeholders were high-level public sector decision-makers and senior advisors in the National Youth Institutes and similar governmental agencies. Other stakeholders include experts, practitioners, civil society organizations and community members related to youth and violence among youth, including young people themselves. In order to facilitate the inclusion of young people in the project, youth-led organizations and youth leaders in the areas of violence and social inclusion were to be identified with the guidance of National Youth Institutes and the OIJ (the Iberoamerican Youth Organization) and invited to participate in certain project activities, as deemed appropriate. V. Guiding Principles 16. The assessment will seek to be independent, credible and useful and adhere to the highest possible professional standards. It will be consultative and engage the participation of a broad range of stakeholders. The unit of analysis is the project itself, including its design, implementation and effects. The assessment will be undertaken in accordance with the provisions contained in the Project Document. The assessment will be conducted in line with the norms, standards and ethical principles of the United Nations Evaluation Group (UNEG).3 17. Although this exercise should not be considered a fully-fledged evaluation (e.g. less extensive data collection and analysis involved, etc.), it is expected that ECLAC’s guiding principles to the evaluation process are applied. 4 In particular, special consideration will be taken to assess the extent to which ECLAC’s activities and outputs respected and promoted human rights. 5 This includes a consideration of whether ECLAC interventions treated beneficiaries as equals, safeguarded and promoted the rights of minorities, and helped to empower civil society. 18. The assessment will also examine the extent to which gender concerns were incorporated into the project — whether project design and implementation incorporated the needs and priorities of women, whether women were treated as equal players, and whether it served to promote women’s empowerment. 19. Moreover, the evaluation process itself, including the design, data collection, and dissemination of the evaluation report, will be carried out in alignment with these principles. 6 20. Evaluators are also expected to respect UNEG’s ethical principles as per its “Ethical Guidelines for Evaluation”: 7 • Independence: Evaluators shall ensure that independence of judgment is maintained and that evaluation findings and recommendations are independently presented. 3 Standards for Evaluation in the UN System, UNEG, April 2005, http://www.unevaluation.org/document/detail/22, Norms for Evaluation in the UN System, UNEG, April 2005; http://www.uneval.org/document/detail/2; UNEG Ethical Guidelines for Evaluation, UNEG, March 2008; http://www.unevaluation.org/document/detail/102. 4 See ECLAC, “Preparing and Conducting Evaluations: ECLAC Guidelines” (2009) and ECLAC, “Evaluation Policy and Strategy”(2014) for a full description of its guiding principles. 5 For further reference see UNEG “Integrating Human Rights and Gender Equality in Evaluations” (2014). 6 Human rights and gender perspective. 7 UNEG Ethical Guidelines for Evaluation, UNEG, March 2008 (http://www.unevaluation.org/ethicalguidelines). FINAL ASSESSMENT REPORT 44 • Impartiality: Evaluators shall operate in an impartial and unbiased manner and give a balanced presentation of strengths and weaknesses of the policy, program, project or organizational unit being evaluated. • Conflict of Interest: Evaluators are required to disclose in writing any past experience, which may give rise to a potential conflict of interest, and to deal honestly in resolving any conflict of interest which may arise. • Honesty and Integrity: Evaluators shall show honesty and integrity in their own behavior, negotiating honestly the evaluation costs, tasks, limitations, scope of results likely to be obtained, while accurately presenting their procedures, data and findings and highlighting any limitations or uncertainties of interpretation within the evaluation. • Competence: Evaluators shall accurately represent their level of skills and knowledge and work only within the limits of their professional training and abilities in evaluation, declining assignments for which they do not have the skills and experience to complete successfully. • Accountability: Evaluators are accountable for the completion of the agreed evaluation deliverables within the timeframe and budget agreed, while operating in a cost effective manner. • Obligations to Participants: Evaluators shall respect and protect the rights and welfare of human subjects and communities, in accordance with the UN Universal Declaration of Human Rights and other human rights conventions. Evaluators shall respect differences in culture, local customs, religious beliefs and practices, personal interaction, gender roles, disability, age and ethnicity, while using evaluation instruments appropriate to the cultural setting. Evaluators shall ensure prospective participants are treated as autonomous agents, free to choose whether to participate in the evaluation, while ensuring that the relatively powerless are represented. • Confidentiality: Evaluators shall respect people’s right to provide information in confidence and make participants aware of the scope and limits of confidentiality, while ensuring that sensitive information cannot be traced to its source. • Avoidance of Harm: Evaluators shall act to minimize risks and harms to, and burdens on, those participating in the evaluation, without compromising the integrity of the evaluation findings. • Accuracy, Completeness and Reliability: Evaluators have an obligation to ensure that evaluation reports and presentations are accurate, complete and reliable. Evaluators shall explicitly justify judgments, findings and conclusions and show their underlying rationale, so that stakeholders are in a position to assess them. • Transparency: Evaluators shall clearly communicate to stakeholders the purpose of the evaluation, the criteria applied and the intended use of findings. Evaluators shall ensure that stakeholders have a say in shaping the evaluation and shall ensure that all documentation is readily available to and understood by stakeholders. • Omissions and wrongdoing: Where evaluators find evidence of wrong-doing or unethical conduct, they are obliged to report it to the proper oversight authority. VI. Scope of the assessment 21. In line with the assessment objective, the scope of the assessment will more specifically cover all the activities implemented by the project. The assessment will review the benefits accrued by the various stakeholders, as well as the sustainability of the project interventions. The assessment will also assess and review the interaction and coordination modalities used in its implementation within ECLAC and between/among other implementing partners participating in the implementation of the project. FINAL ASSESSMENT REPORT 45 22. In summary, the elements to be covered in the assessment include: • Actual progress made towards project objectives • The extent to which the project has contributed to outcomes in the identified countries whether intended or unintended. • The efficiency with which outputs were delivered. • The strengths and weaknesses of project implementation on the basis of the available elements of the logical framework (objectives, results, etc) contained in the project document • The validity of the strategy and partnership arrangements. Coordination among the two implementing Divisions/Offices and other implementing partners. • The extent to which the project was designed and implemented to facilitate the attainment of the goals. • Relevance of the project’s activities and outputs towards the needs of Member States, the needs of the region/subregion and the mandates and programme of work of ECLAC. 23. It will also assess various aspects related to the way the project met the following Development Account criteria: • Result in durable, self-sustaining initiatives to develop national capacities, with measurable impact at field level, ideally having multiplier effects; • Be innovative and take advantage of information and communication technology, knowledge management and networking of expertise at the sub regional, regional and global levels; • Utilize the technical, human and other resources available in developing countries and effectively draw on the existing knowledge/skills/capacity within the UN Secretariat; • Create synergies with other development interventions and benefit from partnerships with non- UN stakeholders. VII. Methodology 24. The assessment will use the following data collection methods to assess the impact of the work of the project: (a) Desk review and secondary data collection analysis: of the programme of work of ECLAC, DA project criteria, the project document, annual reports of advance, workshops and meetings reports and evaluation surveys, other project documentation such as project methodology, country reports, consolidated report, webpage, etc. (b) Self-administered surveys: The following surveys should be considered as part of the methodology: a) Surveys to beneficiaries and Member States in each of the five regions; and b) Survey to implementing partners and stakeholders within the United Nations and the countries participating in the project. PPEU will provide support to manage the online surveys through SurveyMonkey. PPEU will distribute the surveys among project beneficiaries to the revised lists facilitated by the consultant. PPEU will finally provide the evaluator with the consolidated responses. (c) Semi-structured interviews and focus groups to validate and triangulate information and findings from the surveys and the document reviews, a limited number of interviews (structured, semi-structured, in-depth, key informant, focus group, etc.) may be carried out via tele- or video-conference with project partners to capture the perspectives of managers, beneficiaries, FINAL ASSESSMENT REPORT 46 participating ministries, departments and agencies, etc. PPEU will provide assistance to coordinate the interviews, including initial contact with beneficiaries to present the assessment and the evaluator. Following this presentation, the evaluator will directly arrange the interviews with available beneficiaries project managers and implementing partners. 25. Methodological triangulation is an underlying principle of the approach chosen. Suitable frameworks for analysis and evaluation are to be elaborated —based on the questions to be answered. The experts will identify and set out the methods and frameworks as part of the inception report. VIII. Evaluation Issues/ Questions 26. This assessment encompasses the different stages of the given project, including its design, process, results, and impact, and is structured around four main criteria: relevance, efficiency, effectiveness, and sustainability. Within each of these criteria, a set of evaluation questions will be applied to guide the analysis. 8 The responses to these questions are intended to explain “the extent to which,” “why,” and “how” specific outcomes were attained. 27. The questions included hereafter are intended to serve as a basis for the final set of evaluation questions, to be adapted by the evaluator and presented in the inception report. Efficiency (a) Collaboration and coordination mechanisms between and within the ECLAC that ensure efficiencies and coherence of response; (b) Provision of services and support in a timely and reliable manner, according to the priorities established by the project document; Effectiveness (a) How satisfied are the project’s main clients with the services they received? (b) How much more knowledgeable are the participants in workshops and seminars? (c) What are the results identified by the beneficiaries? (d) Has the project made any difference in the behavior/attitude/skills/ performance of the clients? (e) How effective were the project activities in enabling capacities and influencing policy making? (f) Are there any tangible policies that have considered the contributions provided by ECLAC in relation to the project under evaluation? Relevance (a) How in line were the activities and outputs delivered with the priorities of the targeted countries? (b) How aligned was the proposed programme of work with the subprogrammes activities? (c) Were there any complementarities and synergies with the other work being developed? 8 The questions included here will serve as a basis for the final set of evaluation questions, to be adapted by the evaluator and presented in the inception report. FINAL ASSESSMENT REPORT 47 Sustainability With beneficiaries: (a) How did the project utilize the technical, human and other resources available in developing countries? (b) How have the project’s main results and recommendations been used or incorporated in the work and practices of beneficiary institutions after completion of the project’s activities? What were the multiplier effects generated by the project? (c) What mechanisms were set up to ensure the follow-up of networks created under the project? Within ECLAC: (a) How has the project contributed to shaping / enhancing ECLAC’s programme of work / priorities and activities? The work modalities and the type of activities carried out? How has ECLAC built on the findings of the project? IX. Deliverables 28. The assessment will include the following outputs: (a) Work Plan. No later than five days after the signature of the contract, the consultant must deliver to PPOD a detailed Work Plan of all the activities to be carried out related to the evaluation of project ROA254-8, schedule of activities and outputs detailing the methodology to be used, etc. (b) Inception Report. No later than 4 weeks after the signature of the contract, the consultant should deliver the inception report, which should include the background of the project, an analysis of the Project profile and implementation and a full review of all related documentation as well as project implementation reports. Additionally, the inception report should include a detailed evaluation methodology including the description of the types of data collection instruments that will be used and a full analysis of the stakeholders and partners that will be contacted to obtain the evaluation information. First drafts of the instruments to be used for the survey, focus groups and interviews should also be included in this first report. (c) Draft final evaluation Report. No later than 12 weeks after the signature of the contract, the consultant should deliver the final draft report for revision and comments by PPOD and the ERG, which should include the main draft results and findings of the evaluation, lessons learned and recommendations derived from it, including its sustainability, and potential improvements in project management and coordination of similar DA projects. (d) Final Evaluation Report. No later than 14 weeks after the signature of the contract, the consultant should deliver the final evaluation report which should include the revised version of the preliminary version after making sure all the comments and observations from PPOD and the ERG, have been included. Before submitting the final report, the consultant must have received the clearance on this final version from PPOD, assuring the satisfaction of ECLAC with the final evaluation report. (e) Presentation of the results of the evaluation. A final presentation of the main results of the evaluation to ECLAC staff involved in the project will be delivered at the same time of the delivery of the final evaluation report. FINAL ASSESSMENT REPORT 48 All documents related to the present evaluation should be delivered by the consultant in its original version, two copies and an electronic copy. X. Payment schedule and conditions 29. The duration of the consultancy will be initially for 14 weeks during the months of January and April 2016. The consultant will be reporting to and be managed by the Programme Planning and Evaluation Unit (PPEU) of the Programme Planning and Operations Division (PPOD) of ECLAC. Coordination and support to the evaluation activities will be provided by the Social Development Division in Santiago and the Subregional Office in Mexico. 30. The contract will include the payment for the services of the consultant as well as all the related expenses of the evaluation. Payments will be done according to the following schedule and conditions: (a) 30% of the total value of the contract will be paid against the satisfactory delivery of the inception report which should be delivered as per the above deadlines. (b) 30% of the total value of the contract will be paid against the satisfactory delivery of the draft final evaluation report which should be delivered as per the above deadlines. (c) 40% of the total value of the contract will be paid against the satisfactory delivery and presentation of the Final Evaluation Report which should be delivered as per the above deadlines. 31. All payments will be done only after the approval of each progress report and the final report from the Programme Planning and Evaluation Unit (PPEU) of the Programme Planning and Operations Division (PPOD) of ECLAC. XI. Profile of the Evaluator 32. The evaluator will have the following characteristics: Education • MA in political science, public policy, development studies, sociology, economics, business administration, or a related social science. Experience • At least seven years of progressively responsible relevant experience in programme/project evaluation are required. • At least two years of experience in areas related to social inclusion, especially on matters related to the inclusion of youth in violence and insecurity contexts is desirable. • Experience in at least three evaluations with international (development) organizations is required. Experience in Regional Commissions and United Nations projects, especially Development Account projects is highly desirable. • Proven competency in quantitative and qualitative research methods, particularly self-administered surveys, document analysis, and informal and semi-structured interviews are required. • Working experience in Latin America and the Caribbean is desirable. FINAL ASSESSMENT REPORT 49 Language Requirements • Proficiency in English and Spanish is required. XII. Roles and responsibilities in the evaluation process 33. Commissioner of the evaluation Î (ECLAC Executive Secretary and PPOD Director) • Mandates the evaluation • Provides the funds to undertake the evaluation • Safeguards the independence of the evaluation process 34. Task manager Î (PPEU Evaluation Team) • Drafts evaluation TORs • Recruits the evaluator/evaluation team • Shares relevant information and documentation and provides strategic guidance to the evaluator/evaluation team • Provides overall management of the evaluation and its budget, including administrative and logistical support in the methodological process and organization of evaluation missions • Coordinates communication between the evaluator/evaluation team, implementing partners and the ERG, and convenes meetings • Supports the evaluator/evaluation team in the data collection process • Reviews key evaluation deliverables for quality and robustness and facilitates the overall quality assurance process for the evaluation • Manages the editing, dissemination and communication of the evaluation report • Implements the evaluation follow-up process 35. Evaluator/Evaluation team Î (External consultant) • Undertakes the desk review, designs the evaluation methodology and prepares the inception report • Conducts the data collection process, including the design of the electronic survey and semi- structured interviews • Carries out the data analysis • Drafts the evaluation report and undertakes revisions 36. Evaluation Reference Group (ERG) Î (Composed of representatives of each of the implementing partners) • Provides feedback to the evaluator/evaluation team on preliminary evaluation findings and final conclusions and recommendations • Reviews draft evaluation report for robustness of evidence and factual accuracy FINAL ASSESSMENT REPORT 50 XIII. Other Issues 37. Intellectual property rights. The consultant is obliged to cede to ECLAC all authors rights, patents and any other intellectual property rights for all the work, reports, final products and materials resulting from the design and implementation of this consultancy, in the cases where these rights are applicable. The consultant will not be allowed to use, nor provide or disseminate part of these products and reports or its total to third parties without previously obtaining a written permission from ECLAC. 38. Coordination arrangements. The evaluation team comprised of the consultant and the staff of the Programme Planning and Evaluation Unit of ECLAC will confer and coordinate activities on an on-going basis, ensuring a bi-monthly coordination meeting/teleconference to ensure the project is on track and that immediate urgencies and problems are dealt with in a timely manner. If any difficulty or problem develops in the interim the evaluation team member will raise it immediately with the rest of the team so that immediate solutions can be explored and decisions taken. XIV. Assessment use and dissemination 39. This assessment seeks to identify best practices and lessons learned in the implementation of development account projects and specifically the capacity of the countries to promote social inclusion, especially of youth in contexts of violence. The evaluation findings will be presented and discussed to ECLAC with the participation of the Divisions/offices participating in the implementation of the project. An Action Plan will be developed to implement recommendations when appropriate in future development account projects. The evaluation report will also be circulated through ECLAC’s webpage and intranet (and other knowledge management tools), and a final copy will be submitted to DESA, as the programme manager for the Development Account, so as to constitute a learning tool in the organization. FINAL ASSESSMENT REPORT 51 ANNEX 2 D O C U M E N T S R E V I E W E D • Project Document ROA 254-8 “Social inclusion of youth in the context of increasing violence and insecurity with a focus on Central America”, ECLAC, June 2013. • ROA 254-8 Progress report, ECLAC, December 2014. • ROA 254-8 Progress report, ECLAC, December 2013. • Actividades y Publicaciones del Proyecto ROA 254-8-A: Inclusión social de la juventud en contexto de creciente violencia e inseguridad, con foco en Centroamérica. • ROA 254-8 Final report, ECLAC, April 2016. • Draft Programme of Work of the ECLAC System 2012-2013, ECLAC, 2010. • Draft Programme of Work of the ECLAC System 2014-2015, ECLAC, 2012. • Juventud: realidades y retos para un desarrollo con igualdad, ECLAC. • Políticas y prácticas de prevención de la violencia para la juventud en Centroamérica, ECLAC. • Las juventudes centroamericanas en contextos de inseguridad y violencia: realidades y retos para su inclusión social, ECLAC. • Las pandillas en El Salvador: propuestas y desafíos para la inclusión social juvenil en contextos de violencia urbana, ECLAC. • Un extremo de nosotros. Lo público y la paz en El Salvador y Nicaragua, ECLAC. • Estudio de profundización sobre la oferta de educación técnico- profesional para fortalecer el eje educación-empleo, ECLAC. • Hacia la inclusión social juvenil: herramientas para el análisis y el diseño de políticas, ECLAC. • Strategy For Mainstreaming Gender at ECLAC 2013-2017, ECLAC, October 2013. • Standards for Evaluation in the UN System, UNEG, April 2005. • Norms for Evaluation in the UN System, UNEG, April 2005. • UNEG Ethical Guidelines for Evaluation, UNEG, March 2008. • Global status report on violence prevention” (WHO, 2014). FINAL ASSESSMENT REPORT 52 ANNEX 3 P A R T I C I P A N T S A N A L Y S I S The events organised by the project counted with over 700 participants. The following graph shows the number of participants in the six consultative meetings (169), six national workshops (189), two sub- regional workshops (130) and one regional seminar (221). Participants in the events ‘ Source: Elaborated by the evaluator. In line with the Project Document, 41% of the participants were high-level public sector decision-makers and senior advisors in the National Youth Institutes and similar governmental agencies. 36% were experts, practitioners, representatives of civil society organizations and community members related to youth and violence among youth, including young people themselves. The rest of the participants worked at the UN System, academia and regional organisations (e.g. Iberoamerican Youth Organization, Central American Integration System, Caribbean Community and Common Market, Organization of Ibero-American States, etc.). Country of work of the participants in the consultative meetings Source: Elaborated by the evaluator. 29 209 238 20 29 47 1461 Academia Civil Society (and local administration) Public Administration Regional Organisation Unknown United Nations ECLAC Consultive meetings (169) National workshops (189)Sub-regional workshops (130) Regional seminar (221) FINAL ASSESSMENT REPORT 53 Approximately 95% of the participants in these events work in eight countries: El Salvador (32%), Dominican Republic (14%), Guatemala (10%), Costa Rica (9%), Nicaragua (8%), Panama (8%), Honduras (7%) and Chile (5%).9 The remaining participants worked in another 17 countries10 or at regional level. The following figures show the distribution of participants in each type of event. Country of work of the participants in the consultative meetings Source: Elaborated by the evaluator. Country of work of the participants in the national workshops Source: elaborated by the evaluator. 9 The distribution is similar if the regional seminar is not considered: Dominican Republic (20%), El Salvador (16%), Guatemala (13%), Costa Rica (12%), Nicaragua (12%), Panama (11%), Honduras (9%) and Chile (6%). 10 Mexico, Colombia, Argentina, Saint Vincent and the Grenadines, Peru, Jamaica, Spain, Brazil, Uruguay, Trinidad and Tobago, Portugal, Italy, Ecuador, Cuba, Plurinational State of Bolivia, Belize and Barbados. 23 27 26 30 42 21 Costa Rica El Salvador Guatemala Honduras Nicaragua Panama Consultive meetings 34 51 38 15 17 34 Costa Rica El Salvador Guatemala Honduras Nicaragua Panama National workshops FINAL ASSESSMENT REPORT 54 Country of work of the participants in the regional seminar Source: Elaborated by the evaluator 1 1 2 4 2 1 2 3 7 2 1 7 5 2 149 1 1 10 5 7 2 1 1 1 3 Uruguay Trinidad and Tobago Saint Vincent and the Grenadines Dominican Rep. Regional Portugal Peru Panama Mexico Jamaica Italy Honduras Guatemala Spain El Salvador Ecuador Cuba Costa Rica Colombia Chile Brazil Bolivia (Plur. State of) Belize Barbados Argentina Regional Seminar FINAL ASSESSMENT REPORT 55 ANNEX 4 S I M P L I F I E D L O G I C A L F R A M E W O R K Objective To strengthen the capacity of Latin American and Caribbean countries, particularly in Central America, to promote effective social inclusion of youth in the context of violence and insecurity. Intervention logic Indicators Means of verification Risks/Assumptions Expected accomplishment 1 Strengthened capacity of Central American governments to assess and design more effective policies to face youth exclusion and violence. 1. Increased number of policies or policy initiatives on youth exclusion and violence in the target region 2. Number of programmes at the national, sub-regional, and regional level which support specific action for social inclusion of youth (i) Review of information contained in the database on policies and programmes related to youth development policies and programmes. (ii) Questionnaires Project partners and stakeholders are committed to the project and participate actively in the project activities. Main activities A1. Organize consultative meetings in Central America with UN Country Teams and national counterparts to discuss and gather information on youth social inclusion, human security, and violence prevention policies and assess ways to feed project results into UN Common Country Assessments (CCA). A2. Provide technical cooperation to strengthen the capacity of National Youth Institutes to process and use available economic and social data to: i) identify economic and social issues critical for youth development and social inclusion, ii) design public policies promoting youth inclusion, and iii) assess the effectiveness of policies implemented. This will include assistance provided via virtual distance learning tools, as well as in person. A3. Prepare and disseminate one regional comparative study covering Latin America and the Caribbean and two national case studies covering Central American countries that provide a better understanding and knowledge of innovative policies and programmes addressing social inclusion among youth, with a particular focus on human security and violence prevention. The knowledge that is generated from these studies will be used to develop a toolkit that will form the basis of the training activity in A4. A 4. Organize six national and two sub-regional workshops to discuss and analyze with public and private authorities effective policies and programmes oriented to social inclusion, human security, and violence prevention among youth. During these workshops participants will be trained on the use of a toolkit that will be developed based on the knowledge and experiences garnered through activities 1-3. FINAL ASSESSMENT REPORT 56 Intervention logic Indicators Means of verification Risks/Assumptions Expected accomplishment 2 Enhanced sharing of data and information on critical economic and social challenges for youth in Latin America and the Caribbean to support national and regional decision-making. 1. Enhanced availability to stakeholders of timely data on relevant aspects of youth-related social and economic challenges for youth. 2. Increased number of Latin American and Caribbean countries for which information on innovative programmes addressing youth policy priorities relevant to meet MDGs/IADG –including violence– is analysed. (i) Review of information contained in the database on economic and social issues critical for youth development and social inclusion (ii) Review of RISALC records Project partners and stakeholders are committed to the project and are prepared to share experiences and knowledge. Main activities A 5. Create a regional online database gathering valuable experiences on policies and programmes related to youth development —including violence prevention— relevant to meet the MDGs/IADGs. A 6. Generate a regional online database with quantitative and qualitative data on economic and social issues critical for youth development and social inclusion. A7. Establish a network of policy-makers, experts, and representatives from national youth and civil society organizations and a forum to exchange experiences, policies, and programmes addressing violence among youth in Central America. Electronic communications and knowledge sharing will be sustained by RISALC, the Network of Social Institutions in Latin America and the Caribbean (www.risalc.org). A 8. Organize a regional seminar to present and discuss findings and recommendations on social inclusion, human security, and violence prevention contained in the studies undertaken under A.3, as well as to present information gathered under A.5 and A.6. FINAL ASSESSMENT REPORT 57 ANNEX 5 E V A L U A T I O N M A T R I X RELEVANCE The extent to which the project and its activities were suited to the priorities, policies and needs of the region and countries at the time of formulation and to what extent they were linked or related to ECLAC’s mandate and programme of work. (EQ1) How in line were the activities and outputs delivered with the priorities of the targeted countries? Indicators Collection Methods Sources Quality of the problem and objective analysis Document review Interviews Surveys Project Document Project Progress Reports Meeting Reports ECLAC Project Managers UN / International Partners Beneficiaries Level of alignment of the problem analysis with major problem conditions (including the cause and effect links between the problem conditions) Evidence of alignment of objectives and EAs with the region and countries’ needs and priorities Level of satisfaction of relevant stakeholders with the design and content of the Project Degree of relevance of the Project objectives throughout implementation Logic and plausibility of the means-end or cause effect relationship (EQ2) How aligned was the proposed programme of work with the sub-programmes activities? Indicators Collection Methods Sources Evidence of coherence against main ECLAC mandate and policies Document review Interviews Surveys Project Document Project Progress Reports Meeting Reports Programmes of Work of the ECLAC System 2012-2013 and 2014-2015 Strategy for Mainstreaming Gender at ECLAC 2013-2017 ECLAC Project Managers UN / International Partners Beneficiaries Degree of alignment with the overall DA mandate Contribution and consistency with the Programme of Work of the ECLAC System (2012-2013 and 2014-2015) Evidence that the project design took into consideration human rights and gender issues FINAL ASSESSMENT REPORT 58 EFFICIENCY Measurement of the outputs (qualitative and quantitative) in relation to the inputs, including complementarity (the extent to which the activities and the outcomes of the project have been able to establish and/or exploit synergies with other actions implemented by ECLAC, other UN bodies or local organizations) and value added (the extent to which the project’s activities and outcomes have confirmed the advantages of ECLAC’s involvement, specially by promoting human rights and gender equality). (EQ3) Were collaboration and coordination mechanisms put in place to ensure efficiencies and coherence of response? Indicators Collection Methods Sources Extent to which the governance and management structures of the project facilitated the implementation Document review Interviews Surveys Project Document Project Progress Reports Meeting Reports ECLAC Project Managers UN / International Partners Number and type of processes and/or procedures that were enacted to improve the implementation Evidence of clarity in definition of roles and responsibilities with regard to ECLAC’s procedures and reporting requirements Extent to which the management of the project was based on results, including the existence of a RBM policy (EQ4) Were services and support provided in a timely and reliable manner according to the priorities established in the project document? Indicators Collection Methods Sources Planned vs. actual allocation of expenses Document review Interviews Surveys Project Document Project Progress Reports Meeting Reports ECLAC Project Managers UN / International Partners Beneficiaries Implementation delays due to lack of resource allocation timeliness Responses and actions taken to expedite processes Planned versus actual work plan Nature of delays that affected the implementation Degree to which the project beneficiaries feel that project activities were delivered in a timely manner FINAL ASSESSMENT REPORT 59 (EQ5) Were there any complementarities and synergies with the other work being developed? Indicators Collection Methods Sources Evidence of joint programming with other development partners or institutions Document review Interviews Surveys Project Document Project Progress Reports Meeting Reports ECLAC Project Managers UN / International Partners Beneficiaries Evidence of joint implementation of activities with other development partners or institutions Evidence of links with similar initiatives implemented by other Regional Commissions and other UN entities Evidence of the project successfully tapping regionally-generated knowledge (e.g. to identify good practices, to establish indicators, to generate policies, etc.) Evidence of the project contribution to the UNDAF action plans or the CCAs Evidence of active involvement of civil society and youths EFFECTIVENES The extent to which the project attained its objectives and expected accomplishments. (EQ6) To what extent has the project strengthened the capacity of Central American governments to assess and design more effective policies to confront youth exclusion and violence? (EA1) Indicators Collection Methods Sources Level of satisfaction of the project’s main clients with the services received, including use of the knowledge generated in the events and publications Document review Interviews Surveys Project Document Project Progress Reports Meeting Reports ECLAC Project Managers UN / International Partners Beneficiaries Extent to which the knowledge of the participants in workshops and seminars has increased Results identified by the beneficiaries Increased number of policies or policy initiatives on youth exclusion and violence in the target region (Simplified Logical Framework) Number of programmes at the national, sub-regional, and regional level which support specific action for social inclusion of youth (Simplified Logical Framework) FINAL ASSESSMENT REPORT 60 (EQ7) To what extent has the project enhanced sharing of data and information on critical economic and social challenges for youth in Latin America and the Caribbean to support national and regional decision-making? (EA2) Indicators Collection Methods Sources Level of satisfaction of the project’s main clients with the services received, including use of the information available in JUVeLAC Document review Interviews Surveys Project Document Project Progress Reports Meeting Reports ECLAC Project Managers UN / International Partners Beneficiaries Differences in the clients’ behavior, attitude, skills and/or performance Results identified by the beneficiaries Enhanced availability to stakeholders of timely data on relevant aspects of youth- related social and economic challenges for youth (Simplified Logical Framework) Increased number of Latin American and Caribbean countries for which information on innovative programmes addressing youth policy priorities relevant to meet MDGs/IADG –including violence– is analysed (Simplified Logical Framework) (EQ8) To what extent has the project contributed to strengthening the capacity of Latin American and Caribbean countries, particularly in Central America, to promote effective social inclusion of youth in the context of violence and insecurity? (Objective) Indicators Collection Methods Sources Extent to which the project has influenced policy making Document review Interviews Surveys Project Document Project Progress Reports Meeting Reports ECLAC Project Managers UN / International Partners Beneficiaries Evidence of policies that have considered the project results Results identified by the beneficiaries FINAL ASSESSMENT REPORT 61 SUSTAINABILITY The extent to which the benefits of the project are likely to continue after funding has been withdrawn, including long-term impact, dissemination and replication. (EQ9) How was sustainability embedded into the project logic? Indicators Collection Methods Sources Evidence of an Exit Strategy Document review Interviews Surveys Project Document Project Progress Reports Meeting Reports ECLAC Project Managers UN / International Partners Beneficiaries Level of satisfaction of beneficiaries with their involvement during implementation Extent to which project design factored in strengthening local ownership and commitment among key stakeholders Quality of partnerships with new donors or partners to improve after-project financial capacity Evidence of a scaling or replication plan Budget for scaling out to other locations (EQ10) To what extent has beneficiary countries implemented measures to enhance the sustainability of the results of the project? Indicators Collection Methods Sources Extent to which the project utilized the technical, human and other resources available in the beneficiary countries Document review Interviews Surveys Project Document Project Progress Reports Meeting Reports ECLAC Project Managers UN / International Partners Beneficiaries Evidence of the project’s main results and recommendations being used by beneficiary institutions after project end Evidence of multiplier effects generated by the project Mechanisms set up to ensure the follow-up of the networks created by the project Perception of an enabling environment to carry on by government officials after the project ends (EQ11) To what extent has ECLAC implemented measures to enhance the sustainability of the results of the project? Indicators Collection Methods Sources Evidence of the project contribution to shaping/enhancing ECLAC’s programme of work/priorities and activities Document review Interviews Surveys Project Document Project Progress Reports Meeting Reports ECLAC Project Managers Work modalities and the type of activities carried out Evidence of ECLAC’s use of the findings of the project FINAL ASSESSMENT REPORT 62 ANNEX 6 I N T E R V I E W G U I D E L I N E S QUESTIONS EC LA C P ro je ct M an ag er s C o- op er at in g ag en ci es N at io na l i ns tit ut io ns C iv il So ci et y EQ1 1) Do the project objectives and expected accomplishments respond to the region and country needs and priorities? 9 9 9 9 2) Do you think that the cause-effect relationships identified at project design are logic and plausible? (e.g. Do you think that the policy- makers lack of knowledge, skills and information are the main limitations to promote effective youth social inclusion initiatives?) 9 9 9 9 EQ2 3) Do you think that the project has contributed to ECLAC’s Programme of Work? 9 4) Do you think that human rights and gender issues were sufficiently considered during project design? 9 9 9 9 EQ3 5) Do you think that the governance and management structures of the project facilitated its implementation? Were any specific procedures put in place? 9 9 6) Were the roles and responsibilities sufficiently clear (e.g. reporting requirements)? 9 9 7) Was the Logical Framework used as a management tool? Was it reviewed when necessary? Were the indicators useful? Was information collected as prescribed? 9 EQ4 8) Do you think that the services and support were provided in a timely and reliable manner? 9 9 9 9 9) Were there any delays during implementation? Do you know the cause of the delay? Were there any actions taken to expedite processes? 9 9 9 EQ5 10) Were there any complementarities and synergies with the other work being developed? 9 9 9 9 11) Do you think that ECLAC collaborated with other institutions? Were any activities implemented jointly with other partners? 9 9 9 12) Were the activities linked with similar initiatives implemented by other Regional Commissions and other UN entities? 9 9 13) Do you consider that the project used regionally-generated knowledge (e.g. to identify good practices, to establish indicators, to generate policies, etc.)? 9 9 9 9 14) Do you think that civil society and particularly youths were actively involved in the activities? 9 9 9 9 FINAL ASSESSMENT REPORT 63 QUESTIONS EC LA C P ro je ct M an ag er s C o- op er at in g ag en ci es N at io na l i ns tit ut io ns C iv il So ci et y EQ6 15) To what extent do you think that your knowledge has increased after your participation in the events? Has it been useful to improve your work? 9 9 16) Are you familiar with the project publications? Are they useful to improve your work? 9 9 17) Do you know if there are any new policy initiatives or programmes on youth exclusion and violence? Do you think there is any link with the project activities and results? 9 9 9 9 EQ7 18) To what extent has the project enhanced sharing of data and information on critical economic and social challenges for youth? 9 9 19) Do you know the Youth Observatory for Latin America and the Caribbean (JUVeLAC)? Are you registered? Have you uploaded information? Have you used the available information? 9 9 20) Do you think that JUVeLAC provides timely data on relevant aspects of youth-related social and economic challenges for youth? 9 9 9 9 21) Do you think that the project has contributed to increase the number of countries for which information on innovative programmes addressing youth policy priorities relevant to meet MDGs/IADG –including violence– is analysed? How? Which ones? 9 9 9 9 EQ8 22) To what extent has the project contributed to strengthening the capacity to promote effective social inclusion of youth in the context of violence and insecurity? 9 9 9 9 EQ9 23) To what extent did the project factored in strengthening local ownership and commitment among key stakeholders? 9 24) Are you satisfied of your involvement in the project? 9 9 9 25) Does the project implemented an exit strategy? Are you aware of any partnerships to improve after-project financial capacity? Are you aware of any scaling or replication plan? Is there any budget available? 9 9 9 EQ10 26) Do you think that the project utilized the resources available in the beneficiary countries (technical, human, etc.)? 9 9 9 27) Are you aware of the project’s main results and recommendations being used by beneficiary institutions? 9 9 9 28) Have any mechanisms been put in place to ensure the follow-up of the networks created by the project? 9 9 9 29) Do you think that the project has generated multiplier effects? Which ones? 9 9 9 9 EQ11 30) Has ECLAC implemented measures to enhance the sustainability of the results? 9 9 9 31) Has ECLAC used the findings of the project? Has it contributed to shaping / enhancing ECLAC’s programme of work / priorities and activities? 9 FINAL ASSESSMENT REPORT 64 ANNEX 7 S U R V E Y Q U E S T I O N N A I R E Questionnaire overview QUESTIONS Pr oj ec t M an ag er s Be ne fi ci ar ie s Section A.1 (for all) 1) ¿En qué evento(s) organizado(s) por el proyecto ha participado? 9 9 2) Indique su sexo 9 9 3) ¿En qué país trabaja? 9 9 4) ¿Cuál es su cargo actual? 9 9 5) ¿ En qué tipo de institución trabaja? 9 9 Section B (only for project managers) 6) ¿ Los objetivos del proyecto responden a las necesidades y prioridades de los países y la región? 9 7) ¿Son lógicas y creíbles las relaciones causales que establece el proyecto (entre actividades, resultados y objetivos)? 9 8) ¿Ha contribuido el proyecto al Programa de Trabajo de la CEPAL? 9 9) ¿El proyecto ha incorporado un enfoque de derechos humanos e igualdad de género (durante su diseño e implementación)? 9 10) ¿Fueron adecuadas las estructuras de gestión y gobernanza del proyecto para facilitar una implementación eficiente? 9 11) ¿Fue útil el ‘Simplified Logical Framework’ como herramienta de gestión del proyecto? 9 12) ¿Hubo retrasos durante la implementación del proyecto? 9 13) ¿Se han aprovechado sinergias entre el proyecto y otras iniciativas similares? 9 14) ¿El proyecto utilizó de forma efectiva los conocimientos disponibles en la región? (p.ej. identificación de buenas prácticas, establecimiento de indicadores, impulso de políticas, etc.) 9 15) ¿Participó activamente la sociedad civil - y los jóvenes en particular - en las actividades organizados por el proyecto? 9 Section C (only for beneficiaries) Sub-section C.1 (about events) 16) ¿Hasta qué punto le parece que el/los evento(s) fue/fueron relevante(s), teniendo en cuenta el contexto de su país? 9 FINAL ASSESSMENT REPORT 65 QUESTIONS Pr oj ec t M an ag er s Be ne fi ci ar ie s 17) ¿Cuál es su nivel de satisfacción respecto a los temas tratados en el/los evento(s)? 9 18) ¿Cuán eficientes considera que fue/fueron el/los evento(s) desarrollado(s)? 9 19) ¿Participó activamente la sociedad civil —y los jóvenes en particular— en el/los evento(s)? 9 20) Por favor, indique en qué medida cree que en el futuro se organizarán eventos similares sin el apoyo de la CEPAL 9 21) ¿Utiliza los conocimientos adquiridos a través de su participación en el/los evento(s) organizado(s) en el marco de este proyecto, en el desarrollo de su trabajo habitual? 9 22) ¿En su opinión participó activamente la sociedad civil - y los jóvenes en particular - en el/los evento(s)? 9 23) ¿En su opinión hubo igualdad en la participaron de mujeres y hombres en el/los evento(s)? 9 24) ¿Considera que en los temas tratados en el/los evento(s) incorporaron un enfoque de derechos humanos e igualdad de género? 9 Sub-section C.2 (about studies and publications) 25) ¿Cual(es) de las siguientes publicaciones elaboradas en el marco del proyecto conoce usted? 9 26) ¿Ha utilizado esta(s) publicación(es) en el desarrollo de su trabajo habitual? 9 27) ¿Considera que esta(s) publicación(es) es/son de buena calidad? 9 28) ¿Considera que esta(s) publicación(es) tiene(n) un enfoque de derechos humanos e igualdad de género? 9 29) ¿Considera que esta(s) publicación(es) incorporan suficientemente el punto de vista de la sociedad civil? 9 Sub-section C.3 (about JUVeLAC) 30) ¿Conoce el Observatorio de juventud para América Latina y el Caribe (JUVeLAC [en línea]: http://dds.cepal.org/juvelac/)? 9 31) ¿Participa activamente en el JUVeLAC? 9 32) ¿Cree que el JUVeLAC dispone de información actualizada y relevante sobre los desafíos socio-económicos de los jóvenes? 9 33) ¿Ha utilizado la información del JUVeLAC en el desarrollo de su trabajo habitual? 9 Sub-section C.4 (about technical assistance) 34) ¿Participó usted en la asistencia técnica al Consejo para la Política Púbica de la Persona Joven de Costa Rica durante 2014? 9 35) ¿Cuál es su nivel de satisfacción con la calidad de la asistencia brindada por la CEPAL? 9 36) ¿Considera que su organización fue eficiente? 9 FINAL ASSESSMENT REPORT 66 QUESTIONS Pr oj ec t M an ag er s Be ne fi ci ar ie s 37) Indique en qué medida cree que en el futuro se organizarán eventos similares sin el apoyo de la CEPAL 9 38) ¿Considera que la asistencia técnica benefició igualmente a hombres y mujeres? 9 Sub-section C.5 (about all activities) 39) ¿En qué medida ha/han contribuido el/los evento(s), las publicaciones y/o la asistencia técnica a aumentar su capacidad para…? - Procesar y/o análisis información sobre juventud - Establecer vínculos entre las políticas de inclusión social y otras relacionadas con juventud - Diseñar e implementar programas y políticas eficaces para promover la inclusión de los jóvenes - Descubrir e interpretar políticas de inclusión y juventud exitosas en diferentes escenarios 9 Section A.2 (for all) 40) ¿En qué medida cree que la falta de capacidades y/o de información de los responsables políticos de la región es la principal limitación para la elaboración de políticas de integración de jóvenes eficaces? 9 9 41) ¿En qué medida cree que han contribuido las actividades del proyecto ha aumentar y/o mejorar el intercambio de información en la región sobre exclusión y violencia juvenil? 9 9 42) ¿En qué medida cree que han contribuido las actividades del proyecto a mejorar las capacidades de los gobiernos de la región para promover y diseñar políticas más eficaces para la juventud? 9 9 43) ¿Sabe si existen nuevas iniciativas políticas o programas relacionados con la exclusión y violencia juvenil que hayan resultado de la implementación o contado con insumos de este proyecto? 9 9 44) ¿Existe apoyo político y/o un ambiente favorable para continuar con acciones similares a las implementadas por este proyecto? 9 9 45) Por favor indique su grado de acuerdo o desacuerdo con las siguientes afirmaciones - Existe apropiación del proyecto por parte de los beneficiarios - Existe apoyo político para continuar con acciones similares - Se han establecido partenariados que aseguran la sostenibilidad de los resultados - El proyecto o alguna de sus actividades tienen potencial para ser replicados - Los eventos organizados por el proyecto han tenido o tendrán un efecto multiplicador - Las publicaciones del proyecto han tenido o tendrán un efecto multiplicador - La asistencia técnica organizada por el proyecto ha tenido o tendrá un efecto multiplicador - La puesta en marcha del JUVeLAC ha tenido o tendrá un efecto multiplicador - La CEPAL ha puesto en marcha mecanismos adecuados para asegurar la sostenibilidad del proyecto 9 9 46) ¿Tiene alguna recomendación para futuras actividades? 9 9 FINAL ASSESSMENT REPORT 67 Questionnaire Section A.1 (for all) 1) ¿En qué evento(s) organizado(s) por el proyecto ha participado? (elija tantas opciones como sean necesarias) o Reunión de Expertos: “Inclusión Social Juvenil en el Contexto de la Creciente Inseguridad y Violencia”, 10 de diciembre de 2013, Panamá o Reunión de Expertos: “Inclusión Social Juvenil en el Contexto de la Creciente Inseguridad y Violencia”, 12 de diciembre de 2013, El Salvador o Reunión de Expertos: “Inclusión Social Juvenil en el Contexto de la Creciente Inseguridad y Violencia”, 25 de marzo de 2014, Costa Rica o Reunión de Expertos: “Inclusión Social Juvenil en el Contexto de la Creciente Inseguridad y Violencia”, 22 de abril de 2014, Guatemala o Reunión de Expertos: “Inclusión Social Juvenil en el Contexto de la Creciente Inseguridad y Violencia”, 24 de abril de 2014, Nicaragua o Reunión de Expertos: “Inclusión Social Juvenil en el Contexto de la Creciente Inseguridad y Violencia”, 25 de junio de 2014, Honduras o Taller Nacional:“Herramientas para el Diseño e Implementación de Políticas y Programas de Inclusión Social Juvenil”, 21 de abril de 2015, Costa Rica o Taller Nacional:“Herramientas para el Diseño e Implementación de Políticas y Programas de Inclusión Social Juvenil”, 23 de abril de 2015 en Honduras o Taller Nacional:“Herramientas para el Diseño e Implementación de Políticas y Programas de Inclusión Social Juvenil”, 19 de mayo de 2015, Guatemala o Taller Nacional:“Herramientas para el Diseño e Implementación de Políticas y Programas de Inclusión Social Juvenil”, 21 - 22 de mayo de 2015, Nicaragua o Taller Nacional:“Herramientas para el Diseño e Implementación de Políticas y Programas de Inclusión Social Juvenil”, 23 de junio de 2015, El Salvador o Taller Nacional:“Herramientas para el Diseño e Implementación de Políticas y Programas de Inclusión Social Juvenil”, 25 de junio de 2015, Panamá o Taller subregional “Inclusión Social y Juventud en Contextos de Violencia”, 18 - 19 de agosto de 2015, Chile o Taller subregional “Inclusión Social y Juventud en Contextos de Violencia en Centroamérica”, 1 - 2 de septiembre de 2015, República Dominicana o Seminario internacional “Inclusión Social y Juventud en América Latina y el Caribe”, 13 - 14 de octubre de 2015, El Salvador 2) Indique su sexo o Hombre o Mujer 3) ¿En qué país trabaja? (elija una opción) o Argentina o Barbados o Belice o Bolivia o Brasil o Chile o Colombia o Costa Rica o Cuba o Ecuador o El Salvador o España FINAL ASSESSMENT REPORT 68 o Guatemala o Honduras o Italia o Jamaica o México o Nicaragua o Panamá o Perú o Portugal o República Dominicana o San Vicente o Trinidad y Tobago o Uruguay 4) ¿Cuál es su cargo actual? (elija una opción) o Gerente / Director o Oficial técnico o Oficial administrativo o Investigador 5) ¿ En qué tipo de institución trabaja? (elija una opción) o CEPAL Ö To question 6 (section B) o Organización de Naciones Unidas (distinta de la CEPAL) Ö To question 17 (section C) o Institución gubernamental Ö To question 17 (section C) o Agencia regional intergubernamental Ö To question 17 (section C) o Organización de la sociedad civil (ONG, Fundación, etc.) Ö To question 17 (section C) o Academia Ö To question 17 (section C) Section B (only for project managers) 6) ¿Los objetivos del proyecto responden a las necesidades y prioridades de los países y la región? o Sí o No o Sin conocimiento suficiente para responder Comentarios: ______________ 7) ¿Son lógicas y creíbles las relaciones causales que establece el proyecto (entre actividades, resultados y objetivos)? o Sí o No o Sin conocimiento suficiente para responder Comentarios: ______________ 8) ¿Ha contribuido el proyecto al Programa de Trabajo de la CEPAL? o Sí o No o Sin conocimiento suficiente para responder Comentarios (p.ej. cuál fue la contribución o como podría haber contribuido si no lo hizo): ______________ FINAL ASSESSMENT REPORT 69 9) ¿El proyecto ha incorporado un enfoque de derechos humanos e igualdad de género (durante su diseño e implementación)? o Sí o No o Sin conocimiento suficiente para responder Comentarios (p.ej. como se ha materializado este enfoque o como podría haberse mejorado): ______________ 10) ¿Fueron adecuadas las estructuras de gestión y gobernanza del proyecto para facilitar una implementación eficiente? o Sí o No o Sin conocimiento suficiente para responder Comentarios (p.ej. se utilizó algún procedimiento especial, los roles estaban claros, etc.): ______________ 11) ¿Fue útil el ‘Simplified Logical Framework’ como herramienta de gestión del proyecto? o Sí o No o Sin conocimiento suficiente para responder Comentarios (p.ej. se revisó para adaptarse a los cambios, se recogió información regularmente para dar respuesta a los indicadores, etc.): ______________ 12) ¿Hubo retrasos durante la implementación del proyecto? o Sí o No o Sin conocimiento suficiente para responder Comentarios (p.ej. causas, etc.): ______________ 13) ¿Se han aprovechado sinergias entre el proyecto y otras iniciativas similares? o Sí, con actividades desarrolladas por otras Comisiones Regionales o Sí, con actividades desarrolladas por otros organismos de las Naciones Unidas o Sí, con actividades desarrolladas por otras instituciones o No o Sin conocimiento suficiente para responder Comentarios: ______________ 14) ¿El proyecto utilizó de forma efectiva los conocimientos disponibles en la región? (p.ej. identificación de buenas prácticas, establecimiento de indicadores, impulso de políticas, etc.) o Sí o No o Sin conocimiento suficiente para responder Comentarios: ______________ 15) ¿Participó activamente la sociedad civil - y los jóvenes en particular - en las actividades organizadas por el proyecto? o Sí o No o Sin conocimiento suficiente para responder Comentarios: ______________ Ö To question 40 (section A.2) FINAL ASSESSMENT REPORT 70 Section C (only for beneficiaries) Sub-section C.1 (about events) 16) ¿Hasta qué punto le parece que el/los evento(s) fue/fueron relevante(s), teniendo en cuenta el contexto de su país? o Muy relevante(s) o Relevante(s) o Algo relevante o No relevante o Sin conocimiento suficiente para poder responder Comentarios: ______________ 17) ¿Cuál es su nivel de satisfacción respecto a los temas tratados en el/los evento(s)? o Muy satisfecho o Satisfecho o Algo satisfecho o No satisfecho o Sin conocimiento suficiente para poder responder Comentarios: ______________ 18) ¿Cuán eficiente(s) considera que fue/fueron el/los evento(s)? o Muy eficiente(s) o Eficiente(s) o Algo eficiente(s) o Nada eficiente(s) o Sin conocimiento suficiente para poder responder Comentarios: ______________ 19) ¿Participó activamente la sociedad civil - y los jóvenes en particular - en el/los evento(s)? o Sí o No o Sin conocimiento suficiente para responder Comentarios: ______________ 20) Por favor, indique en qué medida cree que en el futuro se organizarán eventos similares sin el apoyo de la CEPAL o Seguramente o Probablemente o Probablemente no o Seguramente no Comentarios: ______________ 21) ¿Utiliza los conocimientos adquiridos a través de su participación en el/los evento(s) organizado(s) en el marco de este proyecto, en el desarrollo de su trabajo habitual? o Si o No o Sin conocimiento suficiente para poder responder Favor especificar de qué manera ha aplicado los conocimientos adquiridos en el desarrollo de su trabajo habitual: ______________ FINAL ASSESSMENT REPORT 71 22) ¿En su opinión hubo igualdad en la participación de mujeres y hombres en el/los evento(s)? o Sí o No o Sin conocimiento suficiente para responder Comentarios: ______________ 23) ¿Considera que en los temas tratados en el/los evento(s) incorporaron un enfoque de derechos humanos e igualdad de género? o Sí, estos temas se trataron adecuadamente o No, estos temas no se trataron adecuadamente o No, estos temas no eran relevantes o Sin conocimiento suficiente para responder Comentarios: ______________ Sub-section C.2 (about studies and publications) 24) ¿Cual(es) de las siguientes publicaciones elaboradas en el marco del proyecto conoce usted? (puede marcar más de una opción) o Juventud: realidades y retos para un desarrollo con igualdad o Políticas y prácticas de prevención de la violencia para la juventud en Centroamérica o Las juventudes centroamericanas en contextos de inseguridad y violencia: realidades y retos para su inclusión social o Las pandillas en El Salvador: propuestas y desafíos para la inclusión social juvenil en contextos de violencia urbana o Un extremo de nosotros. Lo público y la paz en El Salvador y Nicaragua o Estudio de profundización sobre la oferta de educación técnico- profesional para fortalecer el eje educación-empleo o Hacia la inclusión social juvenil: herramientas para el análisis y el diseño de políticas o Ninguna de las anteriores 25) ¿Ha utilizado esta(s) publicación(es) en el desarrollo de su trabajo habitual? o Mucho o Bastante o Poco o Nada o Sin conocimiento suficiente para poder responder Favor brindarnos ejemplos de cómo las ha utilizado: ______________ 26) ¿Considera que esta(s) publicación(es) es/son de buena calidad? o Sí o No o Sin conocimiento suficiente para poder responder Comentarios: ______________ 27) ¿Considera que esta(s) publicación(es) tiene(n) un enfoque de derechos humanos e igualdad de género? o Sí, estos temas se trataron adecuadamente o No, estos temas no se trataron adecuadamente o No, estos temas no eran relevantes o Sin conocimiento suficiente para responder Comentarios: ______________ FINAL ASSESSMENT REPORT 72 28) ¿Considera que esta(s) publicación(es) incorpora(n) suficientemente el punto de vista de la sociedad civil? o Sí o No o No, pero no era necesario/relevante o Sin conocimiento suficiente para responder Comentarios: ______________ Sub-section C.3 (about JUVeLAC) 29) ¿Conoce el Observatorio de juventud para América Latina y el Caribe (JUVeLAC [en línea]: http://dds.cepal.org/juvelac/)? o Sí o No Ö To question 34 (sub-section c.4) 30) ¿Participa activamente en el JUVeLAC? o Sí o No Comente de qué forma lo hace o las razones para no hacerlo: ___________ 31) ¿Cree que el JUVeLAC dispone de información actualizada y relevante sobre los desafíos socio-económicos de los jóvenes? o Sí o No Comentarios: ______________ 32) ¿Ha utilizado la información del JUVeLAC en el desarrollo de su trabajo habitual? o A menudo o De vez en cuando o Casi nunca o Nunca o Sin conocimiento suficiente para poder responder Favor brindarnos ejemplos de cómo la ha utilizado: ______________ Sub-section C.4 (about technical assistance) 33) ¿Participó usted en la asistencia técnica al Consejo para la Política Púbica de la Persona Joven de Costa Rica durante 2014? o Sí o No Ö To question 39 (sub-section c.5) 34) ¿Cuál es su nivel de satisfacción con la calidad de la asistencia brindada por la CEPAL? o Muy satisfecho o Satisfecho o Algo satisfecho o No satisfecho o Sin conocimiento suficiente para poder responder Comentarios: ______________ 35) ¿Considera que la asistencia se brindo de forma eficiente? o Si o No o Sin conocimiento suficiente para poder responder ¿Por qué? ________ FINAL ASSESSMENT REPORT 73 36) ¿Considera que la asistencia técnica benefició igualmente a hombres y mujeres? o Si o No o Sin conocimiento suficiente para poder responder Comentarios: ________ Sub-section C.5 (about all activities) 37) ¿En qué medida ha/han contribuido el/los evento(s), las publicaciones y/o la asistencia técnica a aumentar su capacidad para…? M uc ho Ba st an te Po co N ad a Si n co no ci m ie nt o su fic ie nt e pa ra re sp on de r Procesar y/o análizar información sobre juventud Establecer vínculos entre las políticas de inclusión social y otras relacionadas con juventud Diseñar e implementar programas y políticas eficaces para promover la inclusión de los jóvenes Descubrir e interpretar políticas de inclusión y juventud exitosas en diferentes escenarios Section A.2 (all) 38) ¿Considera que la falta de capacidades y/o de información de los responsables políticos de la región es la principal limitación para la elaboración de políticas eficaces para mejorar la integración de los jóvenes? o Si o No o Sin conocimiento suficiente para poder responder Comentarios: ________ 39) ¿En qué medida considera que han contribuido las actividades del proyecto ha aumentar y/o mejorar el intercambio de información en la región sobre exclusión y violencia juvenil? o Mucho o Bastante o Poco o Nada o Sin conocimiento suficiente para responder Comentarios: ________ FINAL ASSESSMENT REPORT 74 40) ¿En qué medida cree que han contribuido las actividades del proyecto a mejorar las capacidades de los gobiernos de la región para promover y diseñar políticas más eficaces para la juventud? o Mucho o Bastante o Poco o Nada o Sin conocimiento suficiente para responder Comentarios: ________ 41) ¿Sabe si existen nuevas iniciativas políticas o programas relacionados con la exclusión y violencia juvenil que hayan resultado de la implementación o contado con insumos de este proyecto? o Si o No o Sin conocimiento suficiente para poder responder Comente cuales son: ___________ 42) ¿Existe apoyo político y/o un ambiente favorable para continuar con acciones similares a las implementadas por este proyecto? o Si o No o Sin conocimiento suficiente para poder responder Comentarios: ________ 43) Por favor indique su grado de acuerdo o desacuerdo con las siguientes afirmaciones M uc ho Ba st an te Po co N ad a Si n co no ci m ie nt o su fic ie nt e pa ra re sp on de r Existe apropiación del proyecto por parte de los beneficiarios Existe apoyo político para continuar con acciones similares Se han establecido partenariados que aseguran la sostenibilidad de los resultados El proyecto o alguna de sus actividades tienen potencial para ser replicados Los eventos organizados por el proyecto han tenido o tendrán un efecto multiplicador Las publicaciones del proyecto han tenido o tendrán un efecto multiplicador La asistencia técnica organizada por el proyecto ha tenido o tendrá un efecto multiplicador La puesta en marcha del JUVeLAC ha tenido o tendrá un efecto multiplicador La CEPAL ha puesto en marcha mecanismos adecuados para asegurar la sostenibilidad del proyecto Comentarios: ______________ 44) ¿Tiene alguna recomendación para futuras actividades? _______________ FINAL ASSESSMENT REPORT 75 ANNEX 8 L I S T O F I N T E R V I E W E E S • Ms. Daniela Trucco Project Manager / ECLAC Social Development Division • Mr Humberto Soto Project Manager / ECLAC Sub-regional Office Mexico • Mr. Paul Giovanni Rodríguez Iberoamerican Youth Organization (OIJ) • Ms. Adriana Velasquez Secretaría de la Integración Social Centroamericana • Ms. Natalia Camacho Viceministerio de Juventud y Consejo de la Persona Joven (Costa Rica) • Mr. John Anthony Cruz Instituto Nacional de la Juventud (El Salvador) • Mr. Marvin Espinoza DIJUVE (Honduras) FINAL ASSESSMENT REPORT 76 ANNEX 9 E V A L U A T O R ’ S R E V I S I O N M A T R I X A. COMMENTS ERG SPECIFIC COMMENTS PARAGRAPH NUMBER COMMENTS ERG EVALUATOR’S RESPONSE Resumen 3.1.1 No entiendo a qué se refiere el F2; at Project design. Se refiere en el diseño de este proyecto o de programas/políticas de juventud de parte de las instituciones de juventud? It makes reference to the design of the evaluated Project. The evaluation never intended to assess the performance of the local institutions in designing programmes or policies. Resumen 3.1.1 En F3 se menciona que las especificidades no se analizaron a profundidad, pero ese fue uno de los objetivos de las reuniones consultivas, el conocer con mayor profundidad las especificidades de los países en el tema. I agree. The text has been modified to better reflect this issue. 45 (the question was unfortunately skipped by the other recipients) This is not surprising as respondents from outside of ECLAC would not be aware of ECLAC’s mandate I agree. The text has been modified. Table 3 Do you think that there was equal participation of women and men at the events. Aside from their perceptions, shouldn’t we be taking into account more objective measures of this, for example, the number of female to male presenters (or participants) at the sub-regional workshops and final project seminar? Plus a review of the presence of gender and human perspective in the documents produced by the Project, which is the most tangible product that shows what the Project promoted in terms of ideas. The report is not only based on perceptions / answer to this specific question. Several sources of information have been used. The evaluator tried to balance the findings using the most relevant ones (including all the sub-questions mentioned in the comment). Nevertheless, the evaluator did not count the number of male and female presenters as there probably exists a (significant) margin of error in the agendas and the balance could be more easily confirmed by other means. Most importantly, the evaluator thought that even if a balanced presence was confirmed (it was), it did not ensure equal participation. The qualitative sources used in the evaluation were deemed indispensable for this. I am not sure that I understand the comment related to the review of the presence of gender and human perspective in the documents (why not the presentations at the events as well). Does it mean the review by the evaluator? If so, I do not agree that this would be a more objective measure than the ones used. Moreover, this would have involved an assessment from a technical point of view that the project stakeholders could do better (more reliably) than the evaluator himself. Finally, the resources available for the evaluation were limited and did not allow for such a thorough review of the quality of the products. FINAL ASSESSMENT REPORT 77 SPECIFIC COMMENTS PARAGRAPH NUMBER COMMENTS ERG EVALUATOR’S RESPONSE 47 it would be helpful to quantify the participation of men and women in the events. The evaluator did not count the number of male and female participants on purpose. First, I considered that there probably exists a (significant) margin of error in the lists. Second, the balance could be more easily confirmed by other means. Third, I think that ECLAC has already done this exercise so it would not add any value. Fourth, the evaluator thought that even if a balanced presence was confirmed (it was), it did not ensure equal participation. The qualitative sources used in the evaluation were deemed indispensable for this. 47 The main concern expressed about human rights and gender was that more effort could be made to translate into practice the very important concepts developed by the project. El tema ya tiene de por sí un enfoque transversal por ciclo de vida, y aún así se abordó la perspectiva de género y el enfoque de derechos en varios de los productos. I am not sure that I fully understand the comment but I do not see and discrepancy with the report. One of the findings of the evaluation is that the products were elaborated with a gender and human rights perspective. The report provides several examples to support this idea. 50 While this is interesting, I am not sure how it is relevant or applicable to our project since ours preceded the creation of these guidelines, as it is a project from the 8th tranche, 2012. The evaluation report surely does not target (at least not primarily) the evaluated (closed) project. The evaluator thinks that the guideless approved by ECLAC during the implementation of the project (presumable there existed a discussion within ECLAC for many months before) were very relevant as a benchmark / best practice. 50 Si bien es importante hacer un esfuerzo por identificar los stakeholders en la etapa de diseño, en ocasiones surgen algunos actores relevantes en etapas posteriores, como fue el caso del proyecto. I agree. It is exactly what the repost says. 58 But again, these guidelines were not available in 2012 when the PD was developed. The evaluation report surely does not target (at least not primarily) the evaluated (closed) project. The evaluator thinks that the guideless approved by ECLAC during the implementation of the project (presumable there existed a discussion within ECLAC for many months before) were very relevant as a benchmark / best practice. 58 This was included as the first part of the work to be done in the Project, the country specific analysis. I agree. The text has been modified to better reflect this issue. 59 Judging the project based on guildelines that were established after the PD was approved doesn’t make very much sense. The project has not been judged based on the guidelines. They were only used as a benchmark / best practice and other sources /analysis have been used. 62 “Approaches and instruments for youth social inclusion and related capacity building are developed” refers to the objective of developing approaches and instruments for capacity building purposes, which related to EA1 I agree to some extent. The text has been modified to better reflect this issue. FINAL ASSESSMENT REPORT 78 SPECIFIC COMMENTS PARAGRAPH NUMBER COMMENTS ERG EVALUATOR’S RESPONSE 64 We wonder what our colleagues in the DPPO think about this point and point 67 regarding Results Based Management. These practices would have indeed been helpful, had we known about them when we were developing the PD. Not really a comment for the evaluator. 66 IBID *59 The project has not been judged based on the guidelines. They were only used as a benchmark / best practice and other sources /analysis have been used. 71 Dado que esto se menciona también en las lecciones aprendidas pero allá solo se menciona la participación de colegas de la SDD y aquí se hace referencia a colegas de otras divisiones, creo que sería importante aclarar que en el libro participaron en la redacción funcionarios de SDD y de UDS Mex, mas no de otras divisiones I agree. The text has been modified to better reflect this issue. 72 All managers (5) … Me surge la duda, a quienes se refiere??? Quienes son los cinco??? En la Tabla 2 menciona que entrevistó a 2?? Table 2 has been corrected: 5 project managers were surveyed and 2 interviewed. Table 10 Creo que es incorrecto el año, toda vez que acabamos el proyecto en diciembre de 2015??? The date has been corrected. 89 Same extent o some extent? Done. Now in paragraph 86. 90 Política Nacional de Prevención... Honduras? Done 96 Editorial: the political support and a favorable environment exist to continue with similar actions. The paragraph has been modified. 103 Editorial: Human rights and gender equality were considered Done. Now in paragraph 102. 111 Editorial: ECLAC continued support guarantees that the effects are sustained The paragraph has been modified. 117 Toda vez que me sentí integrado y parte de este proceso, y que creo que es un elemento adicional que muestra nuestra buena coordinación, a mi me gustaría que se mencionara que también la oficina de México participó en este proceso de gestión del libro. Done 126 Me parece extraña esta recomendación, después que en las páginas anteriores se reconoció que el proyecto había tenido una fuerte mirada de género en lo sustantivo y en lo práctico (participantes, etc) The report highlights that gender equality was considered throughout the project. Nevertheless, it also conclude tha it could have been more consistently mainstreamed in the design. Please see C2, F4, F5 and F6. 127 La mayor parte del staff involucrado en la implementación del proyecto si tiene bastante incorporado los conocimientos de gender mainstreaming en su formación. Es uno de los temas centrales en el abordaje de los temas que se hace desde desarrollo social. The evaluation never intended to assess the capacities or knowledge of ECLAC’s staff. It only aims at drawing useful recommendations based on the assessment of a single project. FINAL ASSESSMENT REPORT 79 SPECIFIC COMMENTS PARAGRAPH NUMBER COMMENTS ERG EVALUATOR’S RESPONSE Annex 3 Figure: Country of work of the participants in the sub-regional workshops (Source: elaborated by the evaluator).This is not correct; the sub-regional workshops were held in the DR and Chile, but the participants came from different countries; for example, at the Chile sub-regional workshops there were participants from Argentina, Bolivia, Brazil, Chile, Colombia, Cuba, Ecuador, Mexico, Peru, Paraguay, Uruguay; similarlry the participants in the DR sub-regional workshops represented the six Central American countries and the DR. I agree. The table has been deleted. The table was included in the inception report and prepared on the basis of the data tabulated by DPPO. There were a misleading column named ‘country’. B. COMMENTS PPOD GENERAL COMMENTS REPORT SECTION (if applicable) COMMENTS PPOD EVALUATOR’S RESPONSE Please correct the numbering of the pages of the annexes in the table of contents Done Please revise the numbering of the pages, as some have numbers and others do not. I do not see any problem with the page numbers Please edit the report as several typos and grammar errors have been encountered. Done Conclusions Please make sure to make reference to the findings that support each conclusion in the Conclusions section. Done Recommenda tions The consistency between the recommendations and the conclusions to which they are linked needs to be further revised, as there is not always a clear link between what the conclusion states and the recommendation being made. Done SPECIFIC COMMENTS PARAGRAPH NUMBER COMMENTS PPOD EVALUATOR’S RESPONSE 2 Please eliminate the following sections in this paragraph as they do not correspond to the evaluation in reference: “… who, in parallel, has coordinated the final assessment of another four DA projects. The report is based on the information collected by another consultant under the guidance of Mr Guerrero.” Done 56 The first two lines of the paragraph together are a bit confusing, as first it says that the “analysis presented in the text of the Project Document is more clear than the figure above”, and the next sentence starts with “in this sense…”, but then talks about the hierarchies not being evident. Please revise the text to make it clearer. Done 68 In paragraph 68, the evaluator jumps from an explanation of the types of indicators to measure performance at different levels to a concluding sentence stating that “the project though did not developed indicators that comprehensively capture its performance”, without providing further details. We therefore request the evaluator to provide more context on why this conclusion or “finding” was reached to facilitate its comprehension and objectively confirm the finding. Done FINAL ASSESSMENT REPORT 80 SPECIFIC COMMENTS PARAGRAPH NUMBER COMMENTS PPOD EVALUATOR’S RESPONSE 79 Please include information on how many more national case studies were produced in this paragraph. Done 86 and 89 The following sentences are repeated in both paragraphs 86 and 89: The above has been confirmed both by the interviews and the survey. 82% of the beneficiaries (59 out of 72 respondents) use the knowledge obtained in the events in their daily work. Only 4 respondents did not use it (9 did not have sufficient knowledge to answer). Please avoid the repetition and leave them only in the most appropriate paragraph. Done Section 3.4.1 F32 In Finding 32 is stated that “Most stakeholders believe that ECLAC has implemented adequate mechanisms to ensure the sustainability of the project”. However, in the main text of this section there is no information on what were the mechanisms implemented by ECLAC to ensure the sustainability of the project, or at least they are not easily identifiable within the text. We suggest making a more explicit presentation of what those mechanisms were and why they have been assessed as being adequate. Done 99 In paragraph 99 it is stated that “The majority of the respondents to the survey think that there is appropriation of the project by beneficiaries and that ECLAC has implemented adequate mechanisms to ensure the sustainability of the project”. However, when one reviews the table below there is only 25% and 28% of respondents respectively, which seem quite low to be regarded as majority, especially in the first case where there is 20% that disagrees. We recommend revising the text. In the case of the second statement it may suffice to clarify that it is the majority of those who actually responded to the question (by differentiating form those with not enough knowledge to respond), while in the first case further analysis is recommended. The whole section has been revisited 103 C2 states that “Human rights and gender equality were considered throughout the project”. However, there is better little information in the main body of the report (especially in the Findings section) to support this conclusion, especially in the area of gender equality. We would therefore appreciate that you include more details on how where these two perspectives included throughout the project in the Findings section to strengthen and provide evidence to support this conclusion. Please see F6, paragraphs 45 and 46 and Table 3 104 In C3 it is mentioned that “Both the problem and objective analysis included credible cause-effect relationships that demonstrate the adequacy of the project to address the challenges”. However, in paragraph 56 it is stated “the hierarchy of the different levels and their causal relationships are not always evident. For example, the ‘lack of awareness among policy- makers…’ seems rather a cause than a consequence of the ‘economic, social and environmental shocks threaten youth’, the ‘limited linkages between social inclusion and other policies and programmes directed at youths’ and the ‘progress with regard to equity and the MDGs is stifled’”, which seems a bit contradictory. Please the revise both texts to ensure consistency between the findings and the conclusion. C3 revisited FINAL ASSESSMENT REPORT 81 SPECIFIC COMMENTS PARAGRAPH NUMBER COMMENTS PPOD EVALUATOR’S RESPONSE 106 IN C5 it is mentioned that the project management structure contributed to an effective implementation and allowed to respond to the changing needs of the beneficiaries. However, there is very little information of the project management structure in Findings section. We would therefore recommend including more information in the findings section to further strengthen this conclusion, specially more details on what was the management structure, division of roles, coordination mechanisms etc. and in what way such structure and mechanisms contributed to the effective implementation of the project (paragraph 71). Please see F12, F13, paragraphs 70, 71 and 72, table 9 and figure 5 Recommenda tions 1. The consistency between the recommendations and the conclusions to which they are linked needs to be further revised, as there is not always a clear link between what the conclusion states and the recommendation being made. 2. In this section, we still consider more effort needs to be put to make sure there is a clear link between the conclusions and the recommendations. For example, it is recommended to have exit strategies to increase the probabilities of sustainability; however, in both conclusions and findings the project was positively evaluated in terms of its sustainability. The same issue is identified in the case of mainstreaming gender and human rights perspectives which was also mentioned in one of the comments of the ERG. A line has been added to the related conclusion but it doesn’t exactly reflects the related finding. Findings and conclusions related to sustainability and gender have been revised to clarify the evaluator’s argument. It is true that in general both gender and sustainability have been addressed to some extent during implementation. Nevertheless, it is also clear throughout the text that they were not sufficiently incorporated into the design. The recommendations go in this direction. If specific activities are planned (at design – overall and consistent strategy) in order to promote gender equality and sustainability, the project would achieve a bigger impact. 86 and 89 1. The following sentences are repeated in both paragraphs 86 and 89: The above has been confirmed both by the interviews and the survey. 82% of the beneficiaries (59 out of 72 respondents) use the knowledge obtained in the events in their daily work. Only 4 respondents did not use it (9 did not have sufficient knowledge to answer). Please avoid the repetition and leave them only in the most appropriate paragraph. 2. Please revise the text again as it is still repeated in paragraphs 86 and 89. The repeated paragraph has been deleted. FINAL ASSESSMENT REPORT 82 SPECIFIC COMMENTS PARAGRAPH NUMBER COMMENTS PPOD EVALUATOR’S RESPONSE Table 2 1. Please revise table 2 as it still contains errors. Implementing Partners (including UN agencies and intergovernmental organizations) Project Beneficiaries (including governmental institutions, civil society and academia) ECLAC Project Managers Total Interviews: # of stakeholders contacted 4 16 2 22 # of stakeholders interviewed 2 3 2 7 Surveys: # of stakeholders contacted 14 350 346 5 9 369 # of survey responses 12 (86%) 65 (19%) 5 (100% 56%) 82 (22%) The table has been revised. 103 1. C2 states that “Human rights and gender equality were considered throughout the project”. However, there is better little information in the main body of the report (especially in the Findings section) to support this conclusion, especially in the area of gender equality. We would therefore appreciate that you include more details on how where these two perspectives included throughout the project in the Findings section to strengthen and provide evidence to support this conclusion. 2. Even though there are some examples of how human rights and gender perspectives were considered throughout the project, we would appreciate a little more background information in this section, based not only on the survey responses, but presenting more information on how this was done based on the documentary analysis of the project and other sources. An analysis has been included based on the documentary analysis. 106 1. IN C5 it is mentioned that the project management structure contributed to an effective implementation and allowed to respond to the changing needs of the beneficiaries. However, there is very little information of the project management structure in Findings section. We would therefore recommend including more information in the findings section to further strengthen this conclusion, specially more details on what was the management structure, division of roles, coordination mechanisms etc. and in what way such structure and mechanisms contributed to the effective implementation of the project (paragraph 71). 2. Same comment as above. The evaluator feels that all the evidence yielded by the evaluation is already in the report ( please see F12, F13, paragraphs 70, 71 and 72, table 9 and figure 5). As highlighted, the project was implemented under the usual ECLAC-DA procedures and structures. The evaluation never intended to assess FINAL ASSESSMENT REPORT 83 SPECIFIC COMMENTS PARAGRAPH NUMBER COMMENTS PPOD EVALUATOR’S RESPONSE this. The findings make reference to arguably the most remarkable management issue, i.e. collaboration within ECLAC. 47 1. it would be helpful to quantify the participation of men and women in the events. 2. Please reconsider. We think that including an overall figure (not necessarily by event by based on the consolidated list) would help in strengthening the objectivity of the comment for the reader. The requested quantification has been included. 58 1. But again, these guidelines were not available in 2012 when the PD was developed. 2. We think that it would be advisable to include this explanation in the text of the report. As it did happen with the members of the ERG, when one reads the report as structured right now it does seem as if they were being evaluated based on guidelines that were not available when they drafted their project documents. Furthermore, the evaluator’s response needs to be revised, as the guidelines were not produced by ECLAC but by DESA and were not discussed with ECLAC during its drafting. Explanations have been included where relevant.