Terms of reference

Evaluation of the UNFPA-UNESCO-UNODC Interagency Collaboration to Promote Youth Led Initiatives in Response to the Beirut Port Explosion

Background & Introduction

UNFPA-UNESCO-UNODC Interagency collaboration seeks to evaluate a project conducted post the Beirut port blast. The said project aimed at supporting and empowering affected community members, especially the youth. Towards this, a request for proposal was released and 13 local organizations were selected out of a pool of more than 100. Each organization had a unique approach and supported youth through a range of activities, involving Psychosocial interventions, Team Building, Art projects and so on.

Since the completion of the project, the Inter-agency collaboration now aims to evaluate the interventions to comprehend the key achievements in terms of the overall impact, sustainability, relevance, reproducibility, efficiency, effectiveness and the project’s synergy (coherence) with the larger mandates of the three agencies. The detailed objectives of the evaluation are listed in the subsequent section.

The primary intended users of this evaluation are expected to be the (i)UNFPA Country office in Lebanon (ii) UNESCO country office in Lebanon (iii) UNODC country office in Lebanon. The 13 engaged local organizations are also expected to gain from the results of the evaluation in an extended fashion, that allows them to gather and share learning and reflect back on the interventions.  The wider UN and civil society community as well as donors are also secondary users of the findings of this evaluation in terms of potential support/intentions to scale up and expand these interventions. Most importantly, the evaluation is a step towards accountability to the affected communities of the blast and will serve as a learning exercise to gather inputs and insights for similar projects undertaken in the future.

Purpose, Objectives and Scope of the Evaluation

The evaluation is expected to serve as an end of project exercise to reflect on the activities, their relevance, sustainability, key impacts, effectiveness, efficiency of inter-agency coordination and synergy (coherence) with the larger mandates of each of these organizations. It is aimed at bringing a holistic perspective and look at the overall and (possible) long-term results of all the isolated activities that were undertaken.

It will also record lessons learnt and good practices, that can later be reproduced/scaled up in similar contexts.

The specific objectives of the evaluation are listed below.

• To provide an objective overview of the interventions undertaken, their relevance,  appropriateness against the context and the concrete impact achieved.
• To evaluate each individual intervention and analyze them as a part of a wider  strategic objective, aimed at empowering the affected youth, post the Beirut post explosion.
• To evaluate the inter-agency coordination and coordination between local actors and IA collaboration during the length of the project, with special attention to human resources investment/staff time and cost efficiency
• To record lessons learnt and identify good practices that can be reproduced/scaled up in similar contexts.
• To provide a platform for the local actors involved as well as the community affected to voice their impressions of the entire initiative.

The scope of the evaluation covers all the activities undertaken, the 13 implementing partners engaged and the community targeted through the inter-agency collaboration with the aim of youth empowerment post the Beirut port explosion.

Evaluation criteria and indicative areas for investigation

The evaluation will use internationally agreed evaluation criteria, drawn from UNEG norms and standards, OECD/DAC and the ALNAP criteria for the evaluation of humanitarian action.

Along with the overall evaluation, collateral attention will be given to gender, disability, inclusivity of both, protection and accountability to affected population, making sure that these issues are included and highlighted, as necessary.

The below list of indicative areas for investigation, structured around the above-mentioned evaluation criteria, will form the basis for the formulation of evaluation questions by the evaluation team at inception stage. The final list of evaluation questions will be limited to a maximum of 10. Based on the agreed list of evaluation questions, the evaluation team will prepare an evaluation matrix, linking questions with associated assumptions to be assessed, indicators, data sources and data collection tools.

1. Appropriateness/Relevance

• To what extent did the interventions address the needs of the affected population, especially the youth as a response to the Beirut port explosion?
• Rationale: What were the efforts /activities undertaken to identify these needs before deciding on the plan of action?
• To what extent were the agencies involved able to adapt to the changing needs of the evolving context, especially with regards to the financial crisis and Covid-19?

2. Coverage

• To what extent did the interventions reach the population groups with greatest need for youth empowerment, in particular, the most vulnerable?

3. Impact

• What was the overall and the specific concrete impact?
• Were the planned impacts achieved and to what extent?
• Were there any un-anticipated impacts (positive or negative) of the interventions? What were the driving forces for these?

4. Sustainability

• Were there exit strategies in place for the interventions undertaken? Were the exit strategies effectively planned? Were these strategies followed?
• To what extent would the communities be able to sustain the impacts created without any external assistance?
• Are there any additional resources required that may enhance the sustainability of the impacts?

5. Effectiveness

• To what extent did the projects contribute to youth empowerment in the affected community?
• To what extent was the larger community affected positively through its youth being engaged in the interventions?
• To what extent did the activities contribute to managing/reducing the trauma of the incident for the engaged youth and by extension, the affected community?
• To what extent could the interventions reach out to the most vulnerable within the youth and what was the degree of their inclusion?
• To what extent was the project successful in creating youth leaders within the community?

6. Efficiency (Interagency Coordination)

• How efficient was the project in terms of being a quick and timely response to the Beirut blast? How quickly were the agencies able to deploy essential services/ staff to undertake the project?
• To what extent did the Inter Agency Collaboration contribute to enhanced coordination, organizational flexibility, and the achievement of the intended results of the project?
• What were the challenges identified in coordination and communication between the three agencies as well with the 13 local organizations involved, through the course of the project?
• What was the degree of the coordination between the 13 involved agencies for purposes such as referrals, geographic feasibilities of outreach, etc?

7. Synergy (Coherence)

• To what extent do the activities integrate into the larger mandates of the three agencies- UNFPA, UNODC, UNESCO and align with the priorities of the wider recovery response plan  post the explosion?
• To what extent did the activities complement each-other for achieving the final desired outcome of the project?

8. Replicability/ Scale up potential

• Does the project have the potential to be reproduced in a similar context or be contextualized elsewhere?
• To what extent can the interventions be scaled up?
• List the potential challenges for scaling up and/or reproducing the interventions.
• What were the lessons learnt and good practices identified through the course of the project and how could they influence the involved actors (interagency collaboration, UNFPA, UNODC, UNESCO, implementing partners involved, direct beneficiaries of the interventions (youth), larger community members?
• What do the lessons learnt and good practices identified contribute to the knowledge base of the recovery processes?

Methodology and Approach

• The evaluation team will design the evaluation methodology (including data collection methods and tools), which will be presented in the inception report.
• This will be a rapid evaluation, primarily focusing on qualitative methodologies and tools, complemented with primary data collection as necessary and feasible with due consideration to the ongoing fuel crisis and Covid-19 restrictions across the country.
• Suggested methodologies include Focus Group Discussions (FGD)/ Key Informant Interviews (KII) involving the 13 implementing partners and small targeted groups from the community, apart from KII with stakeholders at the inter-agency collaboration.
• It is strongly recommended that short and effective tools are utilized for the purpose of FGDs, KII, etc in the interest of achieving the desired outputs efficiently.
• The evaluation must focus on qualitative aspects such as case analysis, capturing human interest stories and success stories, as applicable.

*Particular attention will be paid to triangulation of information, both in terms of data sources and methods and tools for data collection*

Evaluation Process, Timeline and Deliverables

The evaluation will unfold in a phase-wise manner moving from Preparatory phase to Inception, Data Collection, Analysis and Report Development, Review & Feedback and finally Completion.

Each phase will have distinct outcomes and deliverables that feed into the next phase. The deliverables per phase shall also govern the release of funds, as detailed in the section of Budget and Payment Modalities.

While the infographic above provides a brief glimpse of the phases, the specific tasks and deliverables for each phase are listed in the table below. It is to be noted that “preparatory phase” will be internal and has not been listed below. Also, “review and Feedback” by the 3 agencies will be a continued process throughout the course of the evaluation.

Final Deliverables:

• One report in English in mutually agreed upon format, consisting of the rapid qualitative evaluation results and recommendations with annexes (i) case studies and (ii) documentation good practices. The report must also have the FGD/KII transcripts annexed to it.
• One executive summary in English of 2-3 pages with key findings
• One power point presentation with key results and findings

Management and Governance

The responsibility for the management and supervision of the evaluation will rest with the UNFPA on behalf of the InterAgency Collaboration Group. UNFPA will have overall responsibility for the management of the evaluation process and will be responsible for ensuring the quality and independence of the evaluation (in line with UNEG Norms and Standards and Ethical Guidelines). The main responsibilities of the evaluation entity are:

• draft the terms of reference
• procure the services on behalf of and in collaboration with the 3 agencies as part of the technical evaluation committee
• chair the debriefings, convene review meetings with the evaluator and coordinate the processes
• supervise and guide the evaluator through the evaluation process.
• provide guidance as needed, in the data collection process.
• review, provide substantive comments and approve the draft report.
• review and provide substantive feedback on the case studies, documentation of key learning and good practices.
• approve the final evaluation report.
• disseminate the evaluation results and contribute to learning and knowledge sharing.

The progress of the evaluation will be followed closely by the evaluation reference group consisting of members of UNFPA, UNODC and UNESCO who are directly interested in the results of this evaluation. The main responsibilities of the reference group are to:

• provide feedback and comments on the terms of reference of the evaluation;
• provide feedback and comments on the inception report
• provide comments and substantive feedback from a technical expert perspective on the draft and final evaluation reports;
• act as the interface between the evaluators and key stakeholders of the evaluation, notably to facilitate access to informants and documentation;
• participate in review meetings with the evaluator as required;
• play a key role in learning and knowledge sharing from the evaluation results, contributing to disseminating the results of the evaluation as well.

Evaluation Expert

• At least 6-8 years of experience working in the humanitarian/development sector, including previous/current experience of undertaking similar evaluations of humanitarian/recovery/post recovery assistance, especially youth-based programming.
• Capable of organizing and analyzing both quantitative and qualitative data, especially qualitative analysis of data obtained. Extensive evaluation experience of policies, strategies and programmes including in complex conflict situations, internal displacement, refugee programmes and transition settings;
• Experience with/knowledge of UN and NGO actors is an asset
• Substantial familiarity with subjects like Psychosocial Wellbeing, Psychosocial interventions in humanitarian/recovery programming, youth led community programming, gender, disability and inclusion, etc.
• Extensive knowledge of youth-based programmes and working with young adults
• Excellent research and analytical skills including case analysis and documentation
• Excellent communication skills (written, spoken) in English and Arabic;

Budget and payment modalities

1. The available budget for the overall cost of the evaluation is USD 8,500 . The costs of the evaluation include:
   
   • The evaluation as defined in the Terms of Reference (including other expenses associated with the editing, design (final evaluation report and evaluation briefs) and translation (evaluation brief);
   • The travel related costs as part of the data collection/stakeholder consultation/visits to projects activities
2. The bidder shall not bear any costs (including any related travel) associated with the preparation and submission of the bid. These cannot be included as a direct cost of the assignment. The inter-agency collaboration group shall in no case be responsible or liable for those costs, regardless of the conduct or outcome of the solicitation process.
3. The maximum cost for travel will be used in the financial evaluation and will be included in the contract. The inter-agency collaboration group reserves the right to request less than the maximum number of visits and/or visits shorter than the indicated number of days, should the project needs change as work progresses. Should this occur, the group will pay only for the actual number of visits and actual duration of visits requested.
4. The payment schedules will be as follows:
   
   • 30% upon signing contract, revie3weing documents, submitting an inception report and action plan and finalization of data collection tools and methodologies.
   • 40% post data collection and analysis completion and submission of draft evaluation report.
   • 30% upon final submission of reports/ppt including transcripts of FGDs/KIIs/ etc.

Note that no payment will be processed until the corresponding deliverables are satisfactory and formally approved by the evaluation manager on behalf of the IA team.

Annexes to follow (Formats if we want- of Report, documentation of good practices, case study analysis, ppt), Humanitarian Action Evaluation Criteria, suggestive tools.

Individual consultant who meet the required qualifications set forth in the TORs may apply on line by sending a cover letter expressing interest, curriculum vitae and P11 form.  All documents must be sent to the following email: bids_lbn@unfpa.org  by no later than July 31, 2021 at 11:59 pm.