Joint Impact Assessment of CTA's support to NARI (2008-2014)

JOINT IMPACT ASSESSMENT OF CTA’S SUPPORT TO NARI (2008-2014) Dr Birte Komolong and Maxie Makambo INCLUDING: • Brief on the Capacity-centred Impact Pathway Analysis (CcIPA) model • Abstracts on five studies Ibrahim Khadar, Tarikua Woldetsadick, Jan Brouwers and Eunike SpieringsCTA-CAFAN JOINT IMPACT ASSESSMENT 1 About NARI NARI was established by an Act of National Parliament of PNG in July 1996 as a public funded, statutory research organization, to conduct and foster applied and adaptive research into: i. any branch of biological, physical and natural sciences related to agriculture; ii. cultural and socioeconomic aspects of the agricultural sector, especially of the smallholder agriculture; and iii. matters relating to rural development and of relevance to PNG. Besides, NARI is responsible for providing technical, analytical, diagnostic and advisory services and up-to-date information to the agriculture sector in PNG. The Institute’s purpose (strategic objective) is to accomplish enhanced productivity, efficiency, stability and sustainability of the smallholder agriculture sector in the country so as to contribute to the improved welfare of rural families and communities who depend wholly or partly on agriculture for their livelihoods. This is intended to be accomplished through NARI’s mission of promoting innovative agricultural development in PNG through scientific research, knowledge creation and information exchange. About CTA The vision of the Technical Centre for Agricultural and Rural Cooperation (CTA) is “strengthen policy and institutional capacity development and information and communication management capacities of ACP agricultural and rural development organisations”. To realise its vision, CTA is working with a number of partner organisations in the African, Caribbean and Pacific regions. CTA provides support, usually financial grants, to enable its partner organisations strengthened their institutional and information and communication management capacities. About the Joint Impact Assessment Reports CTA initiated this joint impact study with the aim of promoting learning for development impact with its long-term ACP partner organisations and networks. The study has been carried out in two phases between October 2012 and June 2015, with the first phase that was completed in 2014, involving nine partners: CaFAN and CARDI in the Caribbean region, and ANAFE, EAFF, FANRPAN, IPACC, KENAFF, RTN and RUFORUM in Africa. The second phase, which was launched in 2014, concerned five partners: NARI and SPC in the Pacific region, and AFRACA, PROPAC and WOUGNET in Africa. Close to 50 ACP and EU experts participated in the study. A key achievement of the joint impact study is that the LME Unit has successfully spearheaded the development and application of an innovative impact assessment methodology, referred to as the Capacity-centred Impact Pathway Analysis (CcIPA) model, with support from CDI-WUR, ECDPM, and MDF and the nine ACP partner organisations and networks involved in the first phase. The study has provided baseline information for future impact studies and also identified opportunities for organisational capacity development. CTA and its partners are committed to sharing the lessons from this joint study widely. The joint impact study represents one of the various forms of evaluations and impact assessments which CTA undertakes to generate information necessary for learning, accountability and decision-making. JOINT IMPACT ASSESSMENT OF CTA’S SUPPORT TO NARI (2008-2014) Dr Birte Komolong and Maxie Makambo INCLUDING: • Brief on the Capacity-centred Impact Pathway Analysis (CcIPA) model • Abstracts on five studies Ibrahim Khadar, Tarikua Woldetsadick, Jan Brouwers and Eunike Spierings Table of contents Table of Contents Contacts Acknowledgements Acronyms Executive Summary 4 6 7 8 9 3 In-Depth Report 35 3.1 3.2 3.3 36 37 40 Sampling frame and key questions Findings: Impact of Selected Activities Analysis 3.3.1 Effects of capacity enhancement in ASTI systems 3.5.2 Effects of capacity enhancement in Web 2.0 and social media for development PART A The Joint impact Assessment of CTA’s support to CaFAN 10 1 Introduction and Background 12 4 Conclusion 1.1 1.2 1.3 1.4 1.5 About NARI NARI organisational structure Growth of the network NARI projects Objectives of the Joint Impact Assessment 13 15 16 16 17 4.1 52 19 23 2.3 The 5 Core capabilities of CaFAN 2.3.1 Capability to act and commit 2.3.2 Capability to adapt and renew 2.3.3 Capability to deliver products and services 2.3.4 Capability to relate 2.3.5 Capability to achieve coherence 4 JOINT IMPACT ASSESSMENT OF CTA’S SUPPORT TO NARI 48 5 Overview of the CCIPA model 2.1 2.2 26 47 50 18 5.1 25 43 PART B Explanation on the CCIPA model & process 2 The Quick Scan Design and Approach The logical framework analysis of NARI 2.2.1 Outcomes at the Organisational Level 2.2.2 Outcomes with the Direct Beneficiaries 2.2.3 Outcomes with the Indirect Beneficiaries Where to from here? 40 5.2 5.3 5.4 a. b. Introduction Context in which the CcIPA model was designed Implementing the impact study through different phases The CcIPA model explained Conceptualisation of CcIPA Application of CcIPA 53 54 55 57 57 60 27 6 Lessons learned from Impact 28 Study and Next Steps 29 30 30 32 34 62 PART C Key findings from all nine reports 7 Brief descriptions of the organisations 10 Recommendations for future CTA 64 support/engagement 11 Annexes 70 8.1 8.2 8.3 8.4 8.5 71 72 74 76 77 9 Key findings from logical framework analysis 80 9.1 9.2 9.3 9.4 9.5 81 83 85 87 91 AFRACA NARI SPC WOUGNET PROPAC 100 66 11.1 List of Participants in the CcIPA 8 Selected findings on the capacity of the organisations Capability to act and commit Capability to adapt and self renew Capability to deliver Capability to relate Capability to achieve coherence 95 11.2 11.3 11.4 11.5 impact study Sources Sample Survey Average Scores and Composite Scores Details of CTA Funded Projects/ Activities within NARI 101 102 102 106 110 Table of contents 5 Contacts Dr Birte Komolong Mrs Maxie Makambo Address: NARI Head Office Kana Aburu Haus Sir Alkan Tololo Research Centre P O Box 4415 LAE 411 Morobe Province Address: NARI Head Office Kana Aburu Haus Sir Alkan Tololo Research Centre P O Box 4415 LAE 411 Morobe Province Phone: (675) 478 4000 / 478 4001 Phone: (675) 478 4000 / 478 4001 Fax: (675) 475 1450 Fax: (675) 475 1450 Email: naripng@nari.org.pg Email: naripng@nari.org.pg 6 JOINT IMPACT ASSESSMENT OF CTA’S SUPPORT TO NARI Acknowledgements This report has been prepared by Maxie Makambo Dominic (as the External Resource Person) and Dr Birte Komolong (as the NARI Internal Resource Person). The team is grateful for the valuable support and insights provided by the staff of the National Agricultural Research Institute of Papua New Guinea in Lae, especially Dr Sergie Bang (Director General of NARI) and Dr Norah Omot (Programme Director, Enabling Environment). The External Resource Person is very grateful for the thoughtful leadership and guidance provided by Dr Birte Komolong throughout the process including; field visits to the NARI headquarters in Lae, facilitating consultations with NARI staff and collating relevant data from various sources in NARI, and attending various international workshops organised by CTA to discuss the findings of the study as well as provide feedback on improving the outcomes of the joint impact assessment study. Acknowledgements 7 Acronyms ACP African, Caribbean and Pacific Countries MOUs Memorandum of Understanding PNG Papua New Guinea AIS Agricultural Innovation Systems PNGWIA ASTI Agricultural Science Technology Innovations Papua New Guinea Women in Agriculture R&D Research and Development ACIAR Australian Centre for International Agricultural Research ST&ICT Science and Technology and Information Communication Technology CCIPA Capacity-Centred Impact Pathway Analysis TOT Training of Trainers CTA Technical Centre for Agricultural and Rural Cooperation EU European Union UN United Nations HR Human Resources IT Information Technology NAIS National Agricultural Information Systems NARI National Agricultural Research Institute 8 JOINT IMPACT ASSESSMENT OF CTA’S SUPPORT TO NARI Executive Summary The partnership, spanning ten (10) years, between the Technical Centre for Agricultural and Rural Cooperation (CTA) and the National Agricultural Research Institute (NARI) of Papua New Guinea has been fruitful. CTA, in fulfilling its mandated function, has been supporting a partner organisation in the Pacific Region and NARI, on the other part, has been a successful recipient of CTA’s support and making good use of the invaluable supported offered by CTA. CTA has funded a number of activities within NARI and these activities range from case studies, training of trainer events and learning events. The total value of these activities is 267,596.00 EURO. The CcIPA model provided by CTA has been used to assess the impacts, both in terms of immediate, short and long term, resulting from the supported offered by CTA to NARI. The findings of this study reveal that a wide range of people and organisations benefited from CTA’s support through NARI. Examples of those who benefited include: farmers, extension agents, private sector, R&D organisations, women’s groups, staff of NARI, various PNG Government Departments, other Pacific Island institutions and individuals, regional agricultural research and development organisations, and those parties involved in the information technology industry. These individuals and organisations have benefited through: • Enhanced capacity in a given topic / subject area • Increased visibility • Improved farming practices • New approaches to doing business • Improved income, etc.. NARI, as an organisation, has also benefited enormously from the ongoing support of CTA. NARI’s staff capacity in various subject areas has been enhanced, NARI’s visibility has been increased /improved, NARI is the leader in agricultural research and is also the preferred partner for many donor agencies, NARI has a track record in successful implementing projects resulting in its overall budget increase for projects. NARI has moderate to high levels of organisational capability to: act and commit, deliver on development objectives including NARI’s organisational strategic objective, adapt and self-renew, relate to external stakeholders, and achieve coherence. Executive Summary 9 PART A: JOINT IMPACT ASSESSMENT OF CTA’S SUPPORT TO CaFAN The Capacity-centred Impact Pathway Analysis Model CHANGES 5 CORE CAPABILITIES PROJECTS & ACTIVITIES OUTCOMES AT BENEFICIARIES LEVEL OUTPUTS / DELIVERABLES PRODUCTS & SERVICES 10 CTA-CAFAN JOINT IMPACT ASSESSMENT OUTCOMES AT LEVEL OF ORGANISATION OUTCOMES ON WIDER SOCIETY CTA-CAFAN JOINT IMPACT ASSESSMENT 11 INTRODUCTION & BACKGROUND 1 12 CTA-CAFAN JOINT IMPACT ASSESSMENT 1.1 About NARI The National Agricultural Research Institute of Papua New Guinea was established by an Act of National Parliament of Papua New Guinea (PNG) in July 1996... ...as a public funded, statutory research organization, to conduct and foster applied and adaptive research into: NARI implements the following main strategies2 to achieve its organisational objective, and these include: (i) any branch of biological, physical and natural sciences related to agriculture; • Effective use and sharing of information and knowledge in the agricultural sector (ii) cultural and socioeconomic aspects of the agricultural sector, especially of the smallholder agriculture; and • Improve the productivity, efficiency and stability of agricultural production systems (iii) matters relating to rural development and of relevance to Papua New Guinea. Besides, NARI is responsible for providing technical, analytical, diagnostic and advisory services and up-to-date information to the agriculture sector in PNG. The organisational goal of NARI is ‘improved welfare of rural families and communities who depend wholly or partly on agriculture for their livelihood’ whilst the organisational objective of NARI is to ‘enhance productivity, efficiency, stability and sustainability of the smallholder in the agriculture sector’. • Influence the Enabling environment for sustainable agricultural development • Enhance effective and congenial institutional environment for effective AR4D. NARI’s activities are spread throughout the four (4) regions of the country to cater for the varying agro-ecological conditions that prevail in the country. Whilst NARI’s headquarter is located in Lae, Morobe Province, NARI has five (5) regional centres throughout the country. The following map depicts the geographical spread of NARI’s activities in the country. 2 NARI Strategic Programme Implementation Plan 2012-2020 Part A: Introduction & background 13 “...Nari is to enhance productivity, efficiency, stability and sustainability of the smallholder in the agriculture sector” Figure 1: Map demonstrating NARI’s Regional Centres throughout the Country NE W BISMARCK SEA PACIFIC OCEAN IR EL A ND 8 1 NEW BRITAIN 2 5 3 PAPUA NEW GUINEA GULF OF PAPUS 4 SOLOMON SEA 6 7 CORAL SEA 1. High Altitude Highlands Sub-Regional Centre - Kandep 5. Momase Regional Centre / GIS (Head Office) - Bubia 2. High Altitude Highlands Regional Centre - Tambul 6. Southern Regional Centre - Laloki 3. Highlands Regional Centre - Aiyura 7. Chemistry Laboratory Insect Collection - Kilakila 4. Livestock - Labu 8. Islands Regional Centre - Keravat 14 JOINT IMPACT ASSESSMENT OF CTA’S SUPPORT TO NARI 1.2 NARI organisational structure Figure 2: The Organisational Chart of NARI NARI COUNCIL DIRECTOR GENERAL: DEPUTY DIRECTOR GENERAL: AGRICULTURAL SYSTEMS IMPROVEMENT PROGRAMME DIRECTOR INFORMATION AND KNOWLEDGE PROGRAMME DIRECTOR ENABLING ENVIRONMENT PROGRAMME DIRECTOR INSTITUTIONAL DEVELOPMENT & MANAGEMENT PROGRAMME DIRECTOR NARI REGIONAL CENTRES MOMASE REGIONAL CENTRE (BUBIA AND LABU) HIGHLANCS REGIONAL CENTRE (AIYURA) ISLANDS REGIONAL CENTRE (KERAVAT) SOUTHERN REGIONAL CENTRE (LALOKI) REGIONAL CENTRE MANAGER REGIONAL CENTRE MANAGER REGIONAL CENTRE MANAGER REGIONAL CENTRE MANAGER HIGHLANCS REGIONAL SUBCENTRE (TAMBUL, KANDEP) NARI CHEMISTRY LABORATORY SUB-CENTRE MANAGER LABORATORY MANAGER NARI is a medium size organisation with a linear organisational structure with four (4) distinct layers. At the very top, there is the NARI Council. This Council governs the activities of NARI. The Council is comprised of representatives from other Government Departments (Treasury, Department of Agriculture and Livestock), Universities (University of PNG, University of Technology), Growers Associations, smallholder farmers, and a women’s representative. The Executive Management comprising of the Director General and the Deputy Director General of NARI report to the Council. The Director General of NARI is ultimately responsible for delivering on the organisational objective of NARI. The Program Directors, four (4) in total, who manage the four key programmes of NARI occupy the third layer of NARI’s organisational structure. The Program Directors report to the Deputy Director General of NARI. There are five (5) Regional Centres and each of these regional centres is managed by a Regional Centre Manager who is reporting to the Programme Director Institutional Management and Development. Part A: Introduction & background 15 1.3 Growth of the network 1.4 NARI projects NARI has developed rapidly into a well recognised institution, both nationally and internationally, since its establishment in 1996. NARI has expanded its growth in many areas including: financial, staff numbers, projects / activities and is considered a preferred partner organisation to work with in the country by both national and international development agencies including the Government of PNG. For instance, NARI’s budget increased from 4.2 million Kina (US$1.68) in 2000 to 22.7 million Kina (US$9.1) in 2013. NARI has also witnessed growth in its staff numbers, from 101 (2002) to 152 (2013). These include both scientific and technical staff. (Table 1.) NARI has been and continues to implement a wide range of projects, from small studies to large-scale multi-component projects including regional partners. Many of these projects are formalized through MOUs. NARI’s ability to manage projects is growing by the years, for instance, NARI’s projects and activities increased by 130% between 2002 and 2014. NARI collaborates and partners with more than fifty (50) organisations, both national and international, in implementing many of its projects. Table 1. NARI’s Financial Growth during the period 2000 to 2013. 25 20 PGK Mill 15 10 5 0 2000 2002 2004 2006 2008 Budget Trend ( Total Expenditure) 16 JOINT IMPACT ASSESSMENT OF CTA’S SUPPORT TO NARI 2010 2012 2013 1.5 Objectives of the Joint Impact Assessment The National Agricultural Research Institute (NARI) of Papua New Guinea is a partner organisation of CTA. The partnership between CTA and NARI spans ten (10) years. During this period, CTA funded several projects /activities within NARI, the total value of these investments being 267,596 EURO. The activities funded by CTA ranged from case studies to learning events and training of trainer sessions. CTA and NARI have now agreed to undertake a joint impact assessment of the various projects that CTA has funded within NARI during the 10 year period. To assist with the impact assessment, CTA has introduced a new tool, known as the ‘Capacity Centred Impact Pathway Analysis’ (CcIPA). The CCIPA will be used to assess the impact of CTA’s projects within NARI. The impact assessment exercise has been divided into two (2) phases; (a) a quick scan report, and (b) an in-depth study. This report covers the first phase of the assessment within NARI. c) This report is based on: (i) a review of available literature aimed at facilitating a better understanding of the long standing relationship between CTA and NARI, including a review of the MOUs that were signed by both parties to effect the commissioning of each project; (ii) in-house consultations with key staff of NARI and field visits to meet with key stakeholders; and (iii) the use of a survey questionnaire. This survey questionnaire, designed by CTA, was used for assessing NARI’s organisational capabilities. The staff of NARI, using the key questions in this questionnaire as a guide, assessed NARI’s capabilities. A scale of 1 to 5 was used for this exercise where: 1=Completely Disagree; 2=Disagree to an extent; 3=Moderate; 4=Agree to an extent; 5=Fully Agree. A sample of the survey form used in this assessment is attached as Annex 1. In light of the ongoing relationship between CTA and NARI and also taking into consideration the recent agreement between The overall objective of this report is to CTA and NARI to undertake this joint impact contribute to the learning for development assessment, this report is structured as impact within CTA and its ACP partner follows: section 2 focuses on CTA and NARI organisations and networks. as partners and briefly describes who CTA The specific objectives of this report are and NARI are and what they do; section 3 two-fold: discusses the CcIPA model including its various components and their application in a) For CTA, the objective is to have a clear clarification and an in-depth understanding the current impact assessment process; section 4 analyses and discusses the key of the impact pathways of CTA-supported findings of the joint impact assessment study; projects among its Partnership and the final section summarises the key Organisations and Networks, which in findings of this study and highlights several this case is NARI; and b) For the Partner Organisations and Networks, points for noting when undertaking similar studies in the future. including NARI, the objective is to have an in-depth understanding of the change(s) they bring about and how they bring about such change(s). Part A: Introduction & background 17 THE QUICK SCAN 2 18 CTA-CAFAN JOINT IMPACT ASSESSMENT 2.1 Design and Approach The CcIPA model is a synthesis model based on the premise that the performance and impact of organisations or networks depend to a large extent on the state of their capabilities3. The CcIPA model is built around three main conceptual components: the Five Core Capabilities (5 CCs) model, the Logic Model and a framework for categorising impact indicators. Each of these models or framework is adapted to a certain degree to fit into CcIPA, with the logic model undergoing the most far-reaching modification4. During the planning phase, participants agreed that instead of attempting to construct an entirely new model, the team should draw on the existing array of established evaluation frameworks/models. In view of the emphasis on the capabilities of organisations, it was decided that the 5 Core Capabilities (5Cs) model be given a central place in the CcIPA model. The five core capabilities featured in the CcIPA model can be summarised as follows: A: Capability to act and commit: concerns the ability to work properly, including planning, taking decisions and acting on these decisions collectively. B: Capability to deliver on development objectives: concerns the organisations’ skill to ensure that it is producing what it is established to do. C: Capability to adapt and self-renew: concerns the ability of an organisation to learn internally and to adjust to shifting contexts and relevant trends. D: Capability to relate to external stakeholders: this is about building and maintaining networks with external actors (including governmental structures, private sector parties, civil society organisations and in the end their constituencies) E: Capability to achieve coherence: concerns the strength of an organisations’ identity, self-awareness and discipline. Each of these five categories was further divided into sub-categories referred to as domains. A total of fifteen (15) domains were identified, with two or three key questions per domain to help collect the information needed for assessing an organisation’s capability. The participants agreed to complement the 5Cs model by the Logic model with the latter being adapted (see Figure 1 below) to reflect the fact that organisations/ networks learn and change as they engage in the development processes, which in turn affects their ability to cause an impact. 3 CTA Handout, March 2014. 4 Section 4 of this report, which basically provides the justification and relevance of the CcIPA model in this impact assessment study, is based on the Handout that was provided by CTA to participants in the second group in September, 2014. Part A: The Quick Scan 19 The CcIPA framework combines activities from different programmes, projects or services, whereas the conventional use of the logic model is limited to a single project or programme. Outcomes at the level of the implementing organisation/ network are separated from outcomes at the level of beneficiaries, whereas the common practice is to only differentiate outcomes along the time line (short-term, medium-term, and long-term). Regarding the impact indicators the participants drew on previous work undertaken under the auspices of Oxfam to develop a template, referred to as the ‘impact categorisation table, with a similar format to the 5Cs, containing the following heading: Figure 3: Logic Model adapted for the CcIPA framework Activities Deliverables Outcomes Outcomes Outcomes Involving multiple programmes and actors (including CTA and partner) over a given period (e.g. the last 10 years). Resulting from the grouped activities. Deliverables may be project outputs or direct services not related to any partnership project. (Organisation / network level) • Short-term • Medium-term • Long-term (Direct beneficiaries level) • Short-term • Medium-term • Long-term (Indirect beneficiaries level) • Short-term • Medium-term • Long-term Table 2: Categories of impact Categories of Impact (fixed) Dimension of Impact (Not fixed, not exhaustive) Descriptive of Impact (Not fixed, not exhaustive) Wealth / Capital Income Increased Revenue for farmers Political empowerment Policy New policy adopted Human / Technical Empowerment New skills Use of new ICT tools Social capital Network New organisations joined Natural / Environment Climate Change New measures taken to mitigate changes 20 JOINT IMPACT ASSESSMENT OF CTA’S SUPPORT TO NARI Questions Why? What for? Why not? For whom? Unplanned? Etc... “...a key feature running through the different constituents is the categorisation of indicators aimed at making the information more manageable...” Impact Categories: which are a list of possible general types of impact Dimensions of impact: which are a list of possible changes or specific domains within the general type of impact/impact category Description of impact: which are examples of possible changes/impact Questions: which are a list of questions to help first investigate the impact/finding/story and then to narrate the story. As shown in Table 2, the five categories of impact included in the CcIPA model are: (i) wealth/ capital, (ii) political empowerment, (iii) human/ technical empowerment, (iv) social capital and (v) natural resources/ environment. Overall a key feature running through the different constituents is the categorisation of indicators aimed at making the information more manageable as well as facilitating communication among the participants. Part A: The Quick Scan 21 The Capacity-centred Impact Pathway Analysis (CcIPA) framework incorporates three inter-related elements, as shown in the horizontal segments of the model (moving from left to right): (i) Outputs/Deliverables: These are outputs at the level of the partner organisation or network as a direct result of CTA’s partnership with the organisation or network. They are defined in relation to the set of project interventions or products and services. Deliverables may be directed at the partner’s capacity building needs or made up of products/ services aimed at the partner’s beneficiaries. CTA’s support may be only partial but must be considered significant. (ii) Changes in the capabilities: of the CTA partner organisations/ networks. These can be also described as outcomes at level of the organisation or network. (iii) Effects (medium-, long-term outcomes / impact): on the partner’s direct and indirect beneficiaries or on the wider society that can be attributed to the deliverables and/or changes in the partner’s core capabilities. The application of the CcIPA framework will produce a graphical representation of the changes resulting from diverse interventions over time. CcIPA also takes account of the interactions among the key actors (direct and indirect beneficiaries, stakeholders). The Capacity-centred Impact Pathway Analysis Model CHANGES 5 CORE CAPABILITIES PROJECTS & ACTIVITIES OUTCOMES AT BENEFICIARIES LEVEL OUTPUTS / DELIVERABLES PRODUCTS & SERVICES 22 JOINT IMPACT ASSESSMENT OF CTA’S SUPPORT TO NARI OUTCOMES AT LEVEL OF ORGANISATION OUTCOMES ON WIDER SOCIETY 2.2 The Logical Framework Analysis of NARI The framework therefore addresses the complex human and institutional relationships that may have a crucial bearing on how impact occurs. In this joint impact assessment study (between CTA and NARI), the CcIPA model is used to assess NARI’s institutional capability. There are numerous dimensions of institutional capacity which requires assessment and further strengthening within NARI to enable NARI progress towards achieving its organisational strategic objective. However, for the purposes of this study, the CcIPA model will only focus attention on the 5 core capabilities of NARI and these include: • The capability to act and commit, • The capability to deliver on development objectives, • The capability to adapt and self-renew, • Capability to relate to external stakeholders; and • The capability to achieve coherence. In the following section, the results of NARI’s institutional capacity assessment is discussed. The CcIPA model, as highlighted earlier, integrates several components. These include: the logic model but with an added emphasis on outcomes at the organisational level, the assessment of an organisation’s capability against the 5 Core Capabilities model; and the assessment of benefits at the direct and indirect beneficiaries levels. In light of this, the discussions under this section will be divided into three (3) main parts to reflect the findings under the various components constituting the CcIPA model. The first section will concentrate on the logic model and provide the details of the activities supported by CTA within NARI and will describe the expected outcomes at the organisation or NARI level. The next section will analyse and report on NARI’s organisational capabilities against the five (5) core capabilities model; and the final section will describe the expected outcomes at both the direct and indirect beneficiaries level. A total of seven (7) activities, ranging from case studies, learning workshops, re-fresher learning workshops and training of trainer (TOTs) events were supported by CTA within NARI in the period 2004 to 2014. The total value of this activities stands at 267,596EURO. Refer to Annex 2 for details on the projects / activities funded by CTA within NARI in the period 2004 to 2014. During the period 2004 to 2009, a total of four (4) TOT sessions on agricultural science, technology and innovations (ASTI) systems, and identifying and analysing farmer’s innovations were supported by CTA within NARI. A TOT for ACP Professionals in the Pacific Region to enhance understanding and analysis within the ASTI systems was also conducted. Furthermore, two (2) subregional TOT workshops on ASTI systems were conducted between 2007 and 2009. Part A: The Quick Scan 23 The TOT sessions on ASTI systems attracted a total of fifty five (55) participants (with representatives from six (6) other Pacific Island countries, two (2) regional organisations and ten (10) NARI staff). The refresher training on management of the National Agricultural Information System (NAIS) attracted a total of five (5) participants, from various institutions within PNG. The Web2.0 learning events have only commenced this year (2014) and these learning events are still ongoing. For almost all workshops and learning events, workshop reports have been compiled. However, in some instances, these reports require finalisation. 24 JOINT IMPACT ASSESSMENT OF CTA’S SUPPORT TO NARI The application of the knowledge gained in the TOT sessions on ASTI systems resulted in the compilation of several case studies. Out of the total of four (4) case studies, three (3) of the case studies relate directly to TOT sessions on ASTI systems which were basically undertaken to test /apply the knowledge and skills gained in the TOT sessions. These case studies featured innovations systems in the rice and banana subsectors, farmer experiments and innovations. The case study on extension and delivery systems in PNG was a lone study undertaken within the theme of extension and delivery systems. All, except one, case study was led and managed by NARI. The key deliverables from each of these case studies is the study report. In all instances, a case study report with relevant recommendations has been compiled. These reports are available within NARI. “NARI’s ability to interface between scientists, farmers and other stakeholders improved and strengthened” 2.2.1 Outcomes at the Organisational Level The key question here is ‘how did NARI as an organisation benefited following the successful implementation and completion of the various activities funded by CTA or put it slightly differently, ‘what changes, if any, has occurred within NARI following the successful implementation and completion of the CTA funded activities within NARI?’. • Informed decision making by NARI in linking research to extension • New information /findings available to NARI for use in policy briefs, new research proposals, contributions to development forums, etc. Some of the outcomes expected at the NARI organisational level resulting from CTA supported learning workshops and TOT sessions include: In an effort to answer this question, the activities supported by CTA within NARI have been divided into two (2) main groups, and these are: Aggregation 1 -Case studies; and Aggregation 2- Learning Workshops /TOTs. Another point worth noting at this stage is that the outcomes at the NARI organisational level described in this section are merely ‘expected outcomes’ and not the ‘actual outcomes’. It is anticipated that the actual outcomes will be assessed and reported in the next phase of the joint CTA and NARI impact assessment exercise. • NARI staff who have attended the learning workshop /TOTs in ASTI systems have acquired relevant skills /knowledge in conducting TOTs within the ASTI systems theme, A number of outcomes are expected at the NARI organisational level following the successful planning, implementation and completion of the CTA supported case studies. These changes include: • Increased visibility for NARI, particularly among the other participating ACP professionals and regional organisations, • Capacity of NARI staff enhanced in terms of undertaking case studies in the context of ASTI systems, • Reduced communication costs • New knowledge among NARI staff on agricultural innovations systems • Capacity of NARI enhanced to build and maintain Partnerships and Networks • New actors identified for partnerships in ASTI systems In the next phase of the joint CTA and NARI impact assessment process, the above listed outcomes should be fully validated and properly verified using relevant data. • NARI’s ability to interface between scientists, farmers and other stakeholders improved and strengthened • NARI has improved vision to undertake demand-led design of research for development programmes and projects, • Improved management of NARI’s NAIS as well as NAIS operators identified and established for the maintenance and up-keep of the NAIS, • Improved capacities for NARI to generate and disseminate information, • Increased efficiency of NARI staff in the management of NARI’s NAIS Part A: The Quick Scan 25 2.2.2 Outcomes with the Direct Beneficiaries One of the first group of people who have benefited directly from the case studies funded by CTA include NARI staff who took part in the planning, execution, analysis and write up of the case study reports. The staff of NARI have benefited by: • Greater confidence in application of AIS concept, in analyzing and evaluating farmers’ experimentation and innovation NARI staff and other participants who participated in the CTA sponsored learning and TOTs events are the direct beneficiaries. Workshop and TOT participants, including NARI staff, benefited in terms of: • Greater confidence of NARI staff in analyzing and evaluating farmers’ experimentation and innovation and the application of the AIS concept • Improved networking & partnerships • Improved approaches • Staff capacity enhanced in use of ASTI systems (3 staff) • Staff individual capacity enhanced in use of ASTI systems • Exposure to and awareness of Ag Innovation Systems approach for participating NARI staff Another group of people who benefited directly from the CTA funded learning and TOT events include the NAIS partner institutions and NAIS operators. Participants who participated in the NAIS re-fresher training event benefited in the following ways: • Farmers get recognition of their innovative capabilities The other group of people who have benefited directly from the case studies include NARI, PNG agriculture policy makers (Department of Agriculture and Livestock, and Department of National Planning and Monitoring), and international policy and extension researchers and practitioners. These groups of people have benefited by: • Further debate on improving extension and advisory services • Skilled workshop participants better able and more willing to manage and strengthen the existing information network • Networking and collaboration among existing partners enhanced • Opportunities (strategic and technical) for further development and enlargement of the network identified • Issues to do with sustainability addressed • Development of improved strategies and policies on extension and advisory services “As a direct result of the workshop a number of agreements were made between farmers and growers for export of produce or the initiation of domestic markets” 26 JOINT IMPACT ASSESSMENT OF CTA’S SUPPORT TO NARI Participants from other organisations, including regional organisations as well as PNG organisations, also benefited directly from the learning events and TOTs sessions. The benefits for these groups of people may be summed as: • Increased understanding and knowledge as to how ST& ICT can contribute to supporting the development process • insights into proven strategies for research, teaching and innovation in agriculture • knowledge and awareness of potential opportunities for research and innovation in Pacific agriculture and build on successful initiatives • Stakeholders have information on successful technologies, practices, strategies applied in the practice • Opportunities for out-scaling of successful farmer innovations The other groups of organisations and individuals that have benefited indirectly from the case studies include women’s groups, civil society, extension service agents, agricultural policy makers, and research organisations. These groups, in particular, the PNG Women in Agriculture (PNGWIA), benefited in the following ways: • Visibility for PNGWIA enhanced • Increased capacity for PNGWIA President • Improved efficiency in accessing, developing, A number of organisations and individuals exchanging and disseminating information within these organisations have benefited • Improved IT knowledge indirectly from the various learning events and TOT sessions that were supported by CTA. 2.2.3 Outcomes with Indirect These organisations and individuals include: Beneficiaries farmers, research organisations, women’s groups, extension agents, policy makers, With regard to case studies, a number of regional organisations including R&D organisations and individuals benefited institutions and universities, and the private indirectly from it. In the first instance, those sector companies. These organisations and actors who are primarily involved in the individuals have benefited through: subsectors in which the case studies were conducted benefited from the case studies. • Increased visibility for PNGWIA The benefits to the actors in the respective • Increased capacity for PNGWIA President subsectors (rice, banana, farmer innovations • Better access to scientific and technical and, extension services and policy makers) information in agricultural research and include: development • New knowledge and information on • Improved quality of research proposals and stakeholders role in Rice and Banana scientific and technical reports addressing Innovation system research gaps and needs of stakeholders • Change of practices by actors • Access to a network of expertise (academics, • Change in attitudes in working together scientists, agro-entrepreneuers..) in areas of agricultural development Part A: The Quick Scan 27 2.3 The 5 Core Capabilities of SPC • Access to information from the Pacific Academic, Research, Innovation and Entrepreneurship electronic platform / virtual network information on best practices, marketing of opportunities and solutions , and Other important information. NARI’s organisational capacity was assessed using the five (5) capabilities model featured in the CcIPA model. NARI staff, through a self assessment exercise, assessed NARI’s capability against each of the five (5) core competencies. For details on NARI’s selfassessment scores, refer to Annex 3. These five (5) core capabilities are: • Capacity to act and commit, • Capacity to adapt and self renew, • Capacity to deliver products and service, • Capacity to relate to others; and • Capacity to achieve coherence within NARI. 28 JOINT IMPACT ASSESSMENT OF CTA’S SUPPORT TO NARI “NARI’s organisational culture is committed to change and agreed priorities for achieving its organisational outcomes” 2.3.1 Capability to Act and Commit The capacity to act and commit refers to NARI’s ability to: make informed decisions through participatory process, thorough planning by integrating the views of others, an organisational culture that is committed to change and decisions been made, and having an action oriented leadership and management style that is willing to commit in order to achieve identified organisational objectives. The main domains featured under this area of competency include: action oriented leadership and management; an organisational culture that is committed; and relevant structures /mechanisms established for informed decision making. NARI’s capability to act and commit was rated fairly moderately, represented by a value of 3.75. The overall score, as indicated by the value 3.75 is moderate. However, when assessing each of the domains separately, the results reveal that NARI’s leadership is inspiring and action oriented, and NARI’s organisational culture is committed to change and agreed priorities for achieving its organisational outcomes. The results also reveal that NARI’s decisions are evidence based but they are not necessarily participatory. Staff of NARI felt, as expressed in their scoring, that the decision making process in NARI is less participatory. This was reflected by a low score of 3 against the main domain ‘decision making structures / mechanisms’ and key question ‘decision making happens participatory’ in NARI. Table 3: Assessment of NARI’s Capability to Act and Commit Main Domain Key Words Key Question Average Score Leadership and management Inspiring, action oriented leadership Does NARI have an action oriented leadership 4 Organizational commitment Committed ownership, related to culture NARI’s culture is committed 4 Decision making structures/ mechanisms Information /intelligence based, collective, participatory Decisions are made based on information? 4 Decision making happens participatory 3 Composite Score 3.75 (Scale: 1=Completely Disagree; 2=Disagree to an extent; 3=Moderate; 4=Agree to an extent; 5=Fully Agree) Part A: The Quick Scan 29 2.3.2 Capability to Adapt and Renew The main domains featured under this area of capability include: monitoring and evaluation, internal environment and external environment. Some of the key questions associated with these main domains include: does NARI take external environment changes into account?, are NARI’s internal communication channels open?, does NARI take feedback from stakeholders into account?, and does the M&E system of NARI contributes to improving projects?. The capability to adapt and self-renew is about NARI’s ability to set up mechanisms for analysing and updating changes taking place in the external environment and incorporating such changes in the internal decision making process of NARI, NARI’s internal systems and processes including HR, communication, finance, staff culture etc are conducive for facilitating the achievement of agreed organisational outcomes, and the M&E systems and processes within NARI are robust and flexible enough to allow for integrating feedback from stakeholders and enables effective learning among staff on the both positive and negative outcomes of projects. The capability of NARI to adapt and self-renew was rated fairly high but not too high, represented by a value of 3.2. In fact, the ability to adapt and self-renew was rated the lowest amongst the five (5) core capabilities within NARI. The ratings against the main domains within this capability area reveal that NARI’s internal systems and processes are not sufficiently transparent and participatory. The ratings further reveal that NARI’s communication channels are less open and inclusive, NARI’s M&E system does not allow feedback from its stakeholders, and the M&E system generates minimal support towards improving project’s performance. 30 JOINT IMPACT ASSESSMENT OF CTA’S SUPPORT TO NARI Although NARI’s capacities in the areas of M&E and internal environment were rated fairly moderately, NARI’s ability to analyse the external environment and incorporate these changes in the internal decision making process of NARI was rated high. (Table 4) 2.3.3 Capability to Deliver Products and Services The capability to deliver products and services relates basically to NARI’s ability to take a project through the entire project cycle, from an initial concept note through to designing, implementing, monitoring /reporting, project evaluations and readjustments, project completion and final termination. This area of capability also assesses NARI’s ability to develop and maintain quality assurance measures throughout the project life cycle. The main domains comprising this area of capability include project implementation, project initiation, phasing and termination; and maintaining quality assurance mechanisms. NARI’s capability to deliver products and service was rated high, represented by a value of 4. NARI has a long history of successful designing and implementing projects, both with national and international partners. In fact, NARI is a preferred partner for many funding agencies to implement projects aimed at improving the livelihoods of rural women and men. As a result, this high rating of NARI’s capability to deliver products and services is not a surprise. In fact, NARI’s capability to deliver products and services could have been rated very, very high with a score of 5. Yet, NARI has assessed itself moderately in this area of capability. (Table 5) “NARI has a long history of successful designing and implementing projects, both with national and international partners” Table 4: Assessment of NARI’s Capability to Act and Commit Main Domain Key Words Key Question Average Score External Environment Updating mechanisms, staff assigned, what happens to the information Does NARI have an action oriented leadership 4 Internal Environment Incentives to apply learning, self-assessment, culture NARI has adequate incentives 3 for a learning culture Staff culture, HR resources, finances Is NARI’s HR policy well implemented 3 Internal marketing, communication of issues. Own constrains to implement policies, being open about failure Is NARI’s internal communication channels open 3 Feedback from stakeholders, what happens to information, relevance of products & services, Feedback from stakeholders is taken into account in daily processes 3 Performance appraisal The M&E process contributes to improving projects 3 Monitoring and Evaluation Involvement in baseline assessment Involvement in baseline assessment Composite Score 3.2 Table 5: NARI’s Capability to Deliver Products and Services – Assessment Rating Main Domain Key Words Key Question Average Score Implementation of activities /projects Competencies of staff, quality of services and products Staff are competent for the job that they need to do 4 Resources The infrastructure is adequate to perform the job 4 The types of projects undertaken fit in the overall strategy 4 Resources, processes, steps The quality of NARI’s work is well assured 4 Operational performance appraisal Composite Score Project/activity initiation Frequency of projects, source / or phasing out /termination initiator, number, size, type, reasons for phasing out Quality assurance Composite Score 3 3.75 (Scale: 1=Completely Disagree; 2=Disagree to an extent; 3=Moderate; 4=Agree to an extent; 5=Fully Agree) Part A: The Quick Scan 31 2.3.4 Capability to Relate This area of capability assesses an organisation’s ability to develop and maintain networks with external actors (including government organisations, private sector organisations, civil society organisations and in the end their constituencies). In the case of NARI, this area of capability relates to NARI’s ability to develop and maintain relationships or networks with its external stakeholders, and NARI’s reputation and visibility /credibility among its stakeholders as a result of NARI’s engagement with external stakeholders. NARI is a national institution specialising in agricultural research in Papua New Guinea. Its mandate comes from the Government of PNG and NARI is widely recognised as a leading agricultural research institution in the country. NARI, as a highly technical and scientific research organisation, is staffed by highly qualified staff, some of the best brains in the country as well as international experts on a number of subject areas. NARI is a preferred partner for collaborative efforts among many national and international development partners. NARI collaborates in development efforts with many partners including the Government of PNG, the Government of Australia through the Department of Foreign Affairs and Trade, agricultural research and development agencies such as ACIAR, other regional bodies such as SPC and of course, CTA. NARI undertakes research in various areas and releases its research products and services to the wider community through a number of mechanisms. The main event in which NARI show cases most of its products and services is known as the NARI Agricultural Show. This is an annual event in which NARI displays many of its products, including new releases in technologies and other materials. NARI’s capability to relate was rated fairly high, represented by a value of 4. This high rating reflects NARI’s ongoing commitment to agricultural research and its leading role as a leader in the agricultural research field in the country (and possible also in the Pacific Region). NARI has no problems developing and maintaining networks and relationships with its external stakeholders. Furthermore, NARI has a high level of influence among many of its national collaborators and takes all necessary precautions to ensure its image / credibility is not diminished in anyway. “NARI, as a highly technical and scientific research organisation, is staffed by highly qualified staff, some of the best brains in the country as well as international experts on a number of subject areas” 32 JOINT IMPACT ASSESSMENT OF CTA’S SUPPORT TO NARI Table 6: Assessment of NARI’s Capability to Act and Commit Main Domain Key Words Key Question Average Score Level of engagement and influence of organisation in networks, alliances and collaborative efforts Networks /alliances/partnerships, changes, reasons NARI is involved in all networks 4 Ability to negotiate, ensure there is flexibility, ensure what you need for yourself… The networks add value to NARI 4 Does NARI have a strong reputation among its stakeholders? 4 Mandate Acquiring mandate as an organisation, political legitimacy, social credibility, image of the organisation Visibility /credibility, reputation or status /image Incentives, measures taken, why/what triggered this reputation, tools, opportunities Composite Score 4 Table 7: NARI’s Capability to Achieve Coherence – Assessment Rating Main Domain Key Words Key Question Average Score Governance Structure Score Internal processes, comprehensiveness, perception of staff of these rules The internal management is supportive for staff to operate 4 Vision and Strategy Link between operational / administrative rules and vision, mission, strategy, making strategy happen Staff are well aware of NARI’s strategy 4 People Human resources qualification, growth, staff turnover..respect for diversity Is NARI’s HR policy competency based? 3 Composite Score 3.67 (Scale: 1=Completely Disagree; 2=Disagree to an extent; 3=Moderate; 4=Agree to an extent; 5=Fully Agree) Part A: The Quick Scan 33 2.3.5 Capacity to Achieve Coherence The capability to achieve coherence relates to the strength of an organisation’s identity, self-awareness and discipline5. One of the key things an organisation must establish and have in place all the time is, its identify in terms of: who is it?, why it exists?, and how will it achieve its purpose for which it exits?. Once an organisation takes the necessary steps to identify itself first, it becomes relatively easier to add/amend /develop its internal systems and processes in order to achieve its purpose whilst at the same time be confident in communicating with those in its operating environment and justifying its relevance. NARI, following a five (5) year (2007-mid 2012) institutional capacity development assistance from the Australian Government’s Department of Foreign Affairs and Trade, has developed all the necessary elements to achieve coherence. NARI’s strategic objective(s), developed through the lens of more science and research for development impact as opposed to science for scientific excellence, has improved NARI’s visibility and its justification for existence in the country. NARI, subsequent to this paradigm shift, aligned and realigned its internal systems and processes so that its internal processes can enable the achievement of its new strategic vision – science and research for development impact. NARI also realigned its human resources including their enabling structures to facilitate the achievement of NARI’s organisational objective. 34 JOINT IMPACT ASSESSMENT OF CTA’S SUPPORT TO NARI An integral part of NARI’s HR realignment processes resulted in NARI identifying and developing competencybased TORs for its staff. NARI’s capability to achieve coherence was considered very high, particularly in light of the past work that has been done with the Australian Government’s assistance in the area of corporate planning and institutional capacity development. However, when assessing NARI’s capability using the five (5) core capabilities under the CCIPA model, NARI’s performance was rated as merely high (and not very high), represented by a value of 3.67. A possible reason for this score is that many of the staff members who participated in this survey were new to NARI – meaning that these of group people were not aware of the processes that NARI has developed since 2007 and is currently in use within NARI including NARI’s M&E system and competency-based TORs. 5 CTA Handout, 2014 IN-DEPTH REPORT 3 CTA-CAFAN JOINT IMPACT ASSESSMENT 35 3.1 Sampling frame and key questions The in-depth study assesses the extent to which CTA-funded projects and programmes, through its partner NARI, have impacted on the wider community or intended beneficiaries. In assessing the impact of these projects and programmes, the five impact categories highlighted in the CcIPA model have been used as a guide. Time was a serious constraining factor to this study. Given the limited time available to complete this study, it was not possible to trace everyone involved in the ASTI learning, TOT workshops and Web 2.0 and social media for development events. Several of the ASTI learning and TOT workshops were targeted at the regional level. However, those participants from other Pacific Island countries were not The external consultant in consultation with NARI decided to focus the in-depth study on contacted. Hence, the key findings as discussed in this report are confined to the two different types of learning activities: a) ASTI learning and TOT workshops, and b) Web PNG context. 2.0 for development learning events To gather information relating to potential (implemented in 2014). outcomes and impacts at the individual, organisational and final beneficiary levels, Both assessment areas were capacityemails were sent to participants who building and learning events with similar attended the learning and TOT workshops on expected outcomes and impacts at the ASTI systems and Web 2.0 for development. individual, organisational and final Focus group discussions were also held, beneficiary level. However, while ASTI face-to-face or phone interviews were activities included four workshops and three conducted, and site visits were made to two case studies, only a limited number of NARI organisations in Lae - NARI and the PNG staff were involved, many of whom have University of Technology. In these since left NARI. The Web 2.0 workshops, on the other hand, were conducted last year and interactions, respondents were provided with background information on the partnership a considerable number of staff from NARI and other organisations participated in these between CTA and NARI, and the use of the CcIPA model including the impact categories learning events. highlighted for assessing impact. A semistructured questionnaire was used to guide the discussions. Due to time limitations, only a small number of people were consulted. Altogether, 17 people participated in these discussions, 10 from NARI. 36 JOINT IMPACT ASSESSMENT OF CTA’S SUPPORT TO NARI 3.2 Findings: Impact of selected activities 3.2.1 Effects of capacity enhancement in ASTI systems Capacity in ASTI systems of several key actors involved directly and indirectly in PNG’s agricultural sector was enhanced. The actors’ capacities were enhanced through a series of learning and TOT workshops. During the workshops, the ‘systems approach or thinking’ was presented as a sound framework for engaging in the agricultural sector for policy development and implementation to improve the performance of the agricultural sector. Key actors from PNG who attended learning and TOT workshops on ASTI systems included: the University of Goroka, the PNG University of Technology, the University of National Resources and Environment, the National Fisheries College and NARI5. Some of the learning and TOT workshops on ASTI systems were targeted at the regional level. Hence, the capacities of key stakeholders and actors involved, either directly or indirectly, in the agricultural sector in the respective Pacific Island countries were enhanced. For instance, during the 2007 regional TOT workshop on ASTI systems, the following countries were represented: the Cook Islands, Fiji, Kiribati, Samoa and Vanuatu. Two regional organisations were also represented: the Secretariat of the Pacific Community (SPC) and the University of the South Pacific (USP)6. NARI’s capacity in ASTI systems enhanced NARI’s capacity in ASTI systems was enhanced through learning and TOT workshops. The learning and TOT workshops were conducted over a number of years, from 2004 until 2009. When the ASTI concept was first introduced in 2004, only one NARI staff member was involved. During this early phase, two case studies (on rice innovations and farmer innovations) were compiled to demonstrate the application of the new knowledge and skills gained but also to test the relevance and utility of the concept in the PNG agricultural sector. Following these initial learning workshops and the compilation of the case studies, more NARI staff attended the regional ASTI learning and TOT workshops in 2006. Upon the completion of this regional TOT workshop, NARI compiled the third case study. This case study featured the banana crop (banana innovations). Two more learning and TOT workshops on ASTI systems were conducted in 2007 and 2009. However, no case studies where compiled then. 3.2.2 Effects of capacity enhancement in Web 2.0 and social media for development The learning events in Web 2.0 and social media for development were conducted in four different sites throughout the country in 2014. The first session was held in PNG’s capital, Port Moresby. The other sessions were conducted in Kerevant (East New Britain Province), Lae (Morobe Province), and Aiyura (Eastern Highlands Province). These training events were open to the public and people from a range of organisations attended. Participants involved in the trainings included representatives of universities, government departments, research institutions, NGOs, and women’s groups. 5 Workshop reports supplied by NARI (2015) 6 Workshop report (2007) Part A: In-Depth Report 37 “The learning events in Web 2.0 and social media for development were conducted in four different sites throughout the country in 2014.” A total of 91 participants from all four sites attended the learning events. Relatively speaking, NARI sent the highest number of participants to the Web 2.0 and social media for development trainings per organisation. A total of 34 participants from NARI attended the training. • Online mapping (Google Maps); • Voice communication over the internet at no cost (Skype, Google Talk); • Cloud storage (Dropbox) • Online publishing (micro-blogging [twitter] and blogging [WordPress/Blogger]); Participants who attended the Web 2.0 and social media for development training acquired knowledge and skills in the following areas: • Professional/corporate social networking (LinkedIn and Facebook); • Advanced multilingual online searches and information self-service (alerts and RSS feeds); • Web 2.0 self-instruction (introduction to IMARK). • Cyber security, ICT policy (SPC perspective), case studies, youth in ICT, opportunities; and • Remote collaboration (Wikis and Google Docs); Participants at the 2nd ASTI TOT workshop, Holiday Inn, Port Moresby, PNG (2007) 38 JOINT IMPACT ASSESSMENT OF CTA’S SUPPORT TO NARI Director general of NARI, Dr Sergie Bang, presenting a certificate to a participant at the Web 2.0 and social medial for development training (2014) Part A: In-Depth Report 39 3.3 Analysis The AR4D concept was advocated by the Agricultural Research and Development Support Facility (ADRSF), an initiative of the NARI’s capacity in ASTI systems Australian Government Aid Programme. enhanced ARDSF, a five-year programme, was mandated to develop the institutional capacities of the It is acknowledged that NARI’s capacity in National Agricultural Research System (NARS) ASTI systems was enhanced. However, no in PNG. To achieve this outcome, ARDSF evidence exists to demonstrate that the introduced the ‘systems approach’ to doing capacity-building efforts resulted in lasting business in the agricultural sector and strongly changes, both positive and negative, at the promoted the AR4D concept that challenges organisational or final beneficiary level. TOTs agricultural organisations to be responsive to did not utilise their skills to increase capacity of colleagues in the same organisation or other farmer needs. This means going beyond the stakeholders or partners. There were no further mere delivery of scientific excellence and instead, conducting research along the activities to draw on gained knowledge and impact pathway for delivery of skills to address gaps in innovations systems development outcomes. that would result in longer lasting impacts such as wealth creation, income generation, Secondly, the learning and TOT workshops policy development and climate change were informative for participating NARI adaptation. Whilst the intended beneficiaries staff and helped to open up their minds to did not benefit from the ASTI capacity recognise other opportunities. Knowledge enhancement process, individual NARI and information gained during these learning staff benefited in a number of ways, events enabled NARI staff to appreciate the particularly in terms of human and opportunities that existed elsewhere for technical empowerment. which NARI was a potential beneficiary. In this regard, a respondent expressed that In terms of human/technical empowerment, he first became aware of the opportunities a number of benefits were derived by that were available to NARI under the participating NARI staff. Firstly, knowledge International Treaty on Plant Genetics gained from TOT sessions on ASTI systems Resources for Food and Agriculture during in 2007 and 2009 helped those staff to the 2007 TOT workshop. The respondent better appreciate the Agricultural Research indicated further that this is the only lasting for Development (AR4D) concept that was impression on him as far as ASTI learning introduced into the organisation around and TOT workshops are concerned. that time. 3.3.1 Effects of capacity enhancement in ASTI systems 40 JOINT IMPACT ASSESSMENT OF CTA’S SUPPORT TO NARI Thirdly, individual NARI staff benefited from greater confidence, increased visibility and increased opportunities for further career advancements. In one instance, a female employee of NARI who was intently engaged in the ASTI activities advanced her professional career and successfully completed her doctoral studies from Canberra University, Australia. Lack of impact at indended beneficiaries level: possible reasons Apart from success stories within NARI and among NARI staff, why has there been no evidence of any changes at the intended beneficiaries level? A number of factors have contributed to the lack of impact at the wider community level, including: b) Lack of a critical mass within NARI to advocate for the ASTI system of engaging in the agricultural sector Although more than one learning and TOT workshop on ASTI systems was conducted over five years, there was no structured approach within NARI to maximise and build on the learning in ASTI systems. Staff were randomly selected for each of the learning and TOT workshops. There was no consistency in NARI’s staff attendance. In the initial stages, only one staff member from NARI was consistently involved in the activities. In 2006, several more NARI staff attended the TOT workshop on ASTI systems, whilst in 2009 only six out of 45 participants were from NARI7. Some learning and TOT events were regional workshops which targeted other Pacific Island a) Lack of application of knowledge and countries. NARI was only a host in PNG. In skills gained these regional workshops, NARI was The knowledge and skills acquired through the represented by a small number of staff. For learning and TOT workshops were not instance, during the TOT workshop in 2009 (as translated into appropriate interventions that stated above), only six staff members would have the potential to result in changes represented NARI. Hence, a knowledge base at the final beneficiary level, in terms wealth on ASTI systems was not developed in order to creation, income generation, and sustainability be able to pursue it further within NARI (and of the environment. The wider community was the wider community). This also implies that not informed of the significance and the utility there was lack of internal leadership within of the systems approach in the agricultural NARI to advocate for ASTI systems, ensuring sector. The main reason for this was that that it was an integral part of NARI’s core resources were not allocated for the testing activities. and application of new knowledge and skills acquired on ASTI systems. Hence, farmers and 7 Workshop Report, 2009 the wider community did not benefit from the capacity enhancement efforts in the area of ASTI systems. Part A: In-Depth Report 41 c) Mode of engagement not targeting leadership in host institution (NARI) The mode of engagement CTA used to engage with NARI was not holistic, in particular it did not engage NARI’s leadership. In order for a host institution, in this case NARI, to embrace a new concept and confidently apply it, there must be buy-in from top management. When the top management of a host institution is convinced by a new technology or ideology, the chances of acceptance is high. In the case of the TOT sessions on ASTI systems, NARI’s top management were not specifically targeted. No exclusive sessions were held for top management on the concept of ASTI. Hence, there was no meaningful buy-in from NARI leadership. This resulted in no commitment and allocation of resources from NARI for the actual application of the concept. During field consultations, respondents compared the modes of engagement that CTA and ARDSF used to engage with NARI. Respondents concluded that ARDSF’s approach was more appropriate than CTA’s. ARDSF began its institutional capacitybuilding process for NARS organisations by targeting top management first. Once the top management was on board, it was relatively easier for the rest of the staff to join. This was not the case in CTA’s approach to capacitybuilding for NARI. Figure 5: Flyer used by the PNG University of Technology for the training in Web 2.0 for development, planned for 15-19 June 2015 42 JOINT IMPACT ASSESSMENT OF CTA’S SUPPORT TO NARI 3.3.2 Effects of capacity enhancement in Web 2.0 and social media for development The capacity enhancement in Web 2.0 and social media for development has had some positive impact on the wider community. Firstly, knowledge and skills gained through learning events have been utilised to create wealth. In particular, the University of Technology in Lae has adopted the course for education and teaching, successfully transferring knowledge and skills acquired to other people through a series of trainings. Both academics and non-academics within the university, as well as people external to the university, were trained on various aspects of Web 2.0 for development. Building on the experience and successes of the previous trainings, the University of Technology through its Library Department, is now taking this learning to the next level. The university is offering this training to the general public but for a fee. The next training is planned for 15-19 June 2014. So far, 20 people have confirmed their attendance and the training fee alone is PGK 1,200 per person (approximately €400). Organisations participating in this planned training include universities, research institutions, government departments, mining companies and training institutions. The flyer below was generated and used by the university to support of the planned training in Web 2.0 for development. Part A: In-Depth Report 43 Secondly, under the impact category of human/technical empowerment, there have been some notable changes and benefits. Many of the respondents indicated that they benefited in two ways: a) from new knowledge and skills, and b) enhanced existing skills. NARI staff, in particular two females, indicated that they are now able to market themselves professionally using LinkedIn. One other female staff member expressed that she is now familiar with the social media ‘tagging’ tool and appreciates its value, particularly when researching a topic or subject where a huge volume of references are consulted. Respondents from NARI and the University of Technology in Lae said they applied knowledge and skills gained in the learning events to set up new blogs and also improve their existing blogs. NARI staff working in the field indicated that knowledge and skills gained through the training are now being utilised for writing short briefs, including taking pictures and sending these materials immediately to the intended audiences. A senior NARI staffer expressed that there is no need for computers anymore as smartphones are now enough in the field. NARI field officers noted that the main benefits of social media are: timely contact and release of information; instant feedback; and enabling group discussions on a given topic or subject. Box 1 below details the response from a workshop participant in relation to impact at the human/technical level. 44 JOINT IMPACT ASSESSMENT OF CTA’S SUPPORT TO NARI Box 1. Response highlighting successful use of Web 2.0 for development knowledge and skills “Definitely, the training has impacted me positively. Unlike before, I now confidently use the web 2.0 applications I learnt about in the training, such as Google Drive to store my work and reports. This links my monthly and quarterly reports to my superiors, and makes my reporting a lot easier, neater and more professional. I upload YouTube videos of my work and share them with colleagues, making it more fun. I have improved so much in using YouTube, it has enabled me to spend my own time taking online lessons and tutorials on key subjects with relevance to my workplace, such as: using spreadsheets in data analysis; statistics lessons; business; and personal development. I use VOIP too to keep in touch with colleagues and friends living in faraway places. I have improved so much thanks to the Web 2.0 training I received.” Thirdly, under the political environment dimension of impact, the training in Wed 2.0 for development has influenced policy at the organisational level. Within NARI, there is no social media policy in place to regulate the use and application of social media. Following the Web 2.0 and social media for development training, NARI officers are now able to compile the necessary instruments to give effect to this policy. Box 2: Response highlighting challenges in applying Web 2.0 for development tools in the workplace “As a participant, I learnt a lot about the use of Web 2.0 and social media tools, as well as processes and techniques that are often taken for granted. One of the biggest problems in Papua New Guinea is the cost of internet services and very poor internet connectivity and coverage. That is the main set back from my point of view which hinders most of us in making an impact utilising Web 2.0 technologies. It is difficult to use these exciting technologies to our advantage. I could not do much after the training due to lack of reliable internet access and high cost of accessing the available internet services.” Fourth, changes have also occurred in terms of social capital among the wider community. Many of the respondents expressed that they are now able to communicate with a wider range of individuals and organisations. NARI field staff indicated that they can communicate with both agricultural and non-agricultural organisations and colleagues. A NARI staff member, who is also a journalist, indicated that he can use social media tools such as Facebook to send and receive information from non-agricultural sources. In this particular instance, a mining company operating in the country was seeking information on cocoa husbandry and the NARI team have been able to respond to him using social media. Finally, under the impact category of social capital and networking, NARI applies the knowledge and skills gained in Web 2.0 and social media for development to share and disseminate information through an EUfunded project, ‘Generation and Adaption of Improved Agricultural Technologies to Mitigate Climate Change Imposed Risks to Food Production within Vulnerable Smallholder Farming communities in Western Pacific Countries’. This project is aimed at helping communities adapt to the adverse impacts of climate change. This is a regional project targeting three countries in the Pacific region, namely PNG, the Solomon Islands and Vanuatu. NARI posts stories and technologies on the project’s blog. The blog is active (http:// euardproject.wordpress.com). With every success story there are also setbacks and challenges. This is also the case for the Web 2.0 and social media for development learning events. Whilst many of the successes have been highlighted in the preceding paragraphs, there have been some challenges in implementing the knowledge and skills acquired. Some of the respondents mentioned that they were unable to apply the knowledge or skills gained because; a) the cost of internet is so expensive, b) internet connectivity and coverage is very poor/slow, and c) a lack of an enabling environment to implement the knowledge and skills gained. Box 2 below provides an example of some of the challenges experienced by the users of Web 2.0 for development. Part A: In-Depth Report 45 CONCLUSION 4 46 CTA-CAFAN JOINT IMPACT ASSESSMENT The Partnership between CTA and NARI, spanning 10 years, has produced positive results and many organisations and individuals associated with the joint efforts of CTA and NARI have benefited (both directly and indirectly) from these activities. The activities that were supported by CTA ranged from case studies, learning events or workshops to training of trainer sessions. Many organisations and individuals have benefited from the CTA sponsored activities and some of these include: NARI staff, R&D organisations, farmers and farmer groups, policy makers, private sector actors, universities, extension agents, women’s groups and civil society, and regional organisations including R&D organisations. NARI has also assessed it’s organisational capabilities against the five (5) core capabilities (as per CcIPA model) and these assessments reveal that NARI has high to very high capabilities in its ability to relate to its external stakeholders, adapt and self re-new, act and commit, deliver its products and services, and achieve coherence. The expected benefits (both direct and indirect) derived from the CTA supported activities also varies; for instance, immediate enhancement of knowledge and skills in new subject areas for workshop participants, identifying new research priorities and developing new research proposals, and fostering partnerships between different actors. Part A: Conclusion 47 4.1 Where to from here? The following recommendations reflect some of the challenges faced in compiling this report. In addition, these recommendations are not meant to be exhaustive but provide some indication of areas requiring improvement in similar assignments in the future. The points for noting, then, include: • undertake a baseline assessment first, • undertake an assessment to establish CTA’s contribution to the success and sustainability of an organisation, • assess an organisation’s ability to address cross cutting issues such as HIVAIDS, and Gender; and • undertake impact assessments within reasonable timeframes (loss of institutional memory records) “...undertake an assessment to establish CTA’s contribution to the success and sustainability of an organisation.” 48 JOINT IMPACT ASSESSMENT OF CTA’S SUPPORT TO NARI Part A: Conclusion 49 PART B: EXPLANATION OF THE CcIPA MODEL AND PROCESS (THIS SECTION INCLUDES THE LESSONS LEARNED FROM THE NINE CcIPA STUDIES OF ROUND 1) Capacity-centred Impact Pathways Analysis (CcIPA) model: design, testing and use through collaborative case studies Prepared by: Ibrahim Khadar (CTA), Tarikua Woldetsadick (CTA), Jan Brouwers (CDI-WUR) and Eunike Spierings (ECDPM) The Capacity-centred Impact Pathway Analysis Model CHANGES 5 CORE CAPABILITIES PROJECTS & ACTIVITIES OUTCOMES AT BENEFICIARIES LEVEL OUTPUTS / DELIVERABLES PRODUCTS & SERVICES OUTCOMES AT LEVEL OF ORGANISATION OUTCOMES ON WIDER SOCIETY OVERVIEW OF THE CCIPA MODEL 5 52 Section 1: Study area 2: Capability to relate 5.1 Introduction In 2012, CTA’s Learning, Monitoring and Evaluation (LME) Unit initiated a joint impact study of the Centre’s technical and financial support to its long-standing partners, focusing on nine national and regional organisations and networks in Africa (ANAFE, EAFF, FANRPAN, IPACC, KENAFF, RTN and RUFORUM) and the Caribbean region (CaFAN and CARDI). These organisations and networks cover more than 50 countries and they are as diverse as the countries they cover - some operate as a small secretariat with nodes and members in their various constituencies, while others are large organisations with sub-offices in various countries. Some are university networks while others are farmers’ organisations. Their areas of intervention range from ICTs, to forestry education and from research to policy advocacy. The study was limited to collaboration with CTA over the past ten years. Eighteen months after the study was formally launched in a workshop held at CTA’s Headquarters in Wageningen, on 29-30 October 20126, this initiative has produced a number of very interesting results that have benefitted the participating institutions, with some of the results, such as the formulation of the CcIPA model itself, likely to be of interest to the wider development community. The other results include: (i) improved understanding of the impact pathways of CTA’s and its partners’ project interventions, (ii) availability of baseline data for future impact studies, and (iii) strengthened M&E capacities (including CTA’s) of the participating organisations. This paper presents the main highlights of this exciting experience. The next two sections explain the process, starting with the context in which the impact study was conceived and designed, followed by a short description of the different phases of the study, during which the model was formulated and applied. Sections 4 -6 present respectively, the key features of the CcIPA model, some of the immediate benefits of the impact study, and the next steps foreseen in the future development and application of the CcIPA model. Contact details of the experts who contributed to the testing and development of the CcIPA model are listed in Annex 1. These lists are included as an acknowledgement of the fact that, while CTA has provided the technical leadership and financial support for developing CcIPA, the impact study has benefitted extensively from the active involvement of CTA’s partners and local M&E experts, as well as advice given by a number EU-based M&E experts, in particular from three international development centres based in the Netherlands, namely WUR-CDI, ECDPM and MDF. 6 Consultation on 29-30 October 2012 at CTA’s Headquarters in Wageningen, attended by representatives of CTA, ANAFE, KENAFF, FANRPAN, CDI-WUR, ECDPM, MDF and two private/ individual consultants. Part B: Overview of the CCIPA Model 53 5.2 Context in which the CcIPA model was designed Evaluation methodologies are never contextfree, even though as they get more accepted there is a tendency to pay less attention to the context in which they are applied. CTA wanted to carry out an impact assessment of the support it has provided to its longstanding partners, using an evaluation methodology that would genuinely promote joint learning. It was in light of this consideration that the LME Unit sought to set up an evaluation exercise that would meet the following requirements: While a number of authoritative voices in the field of evaluation have subscribed to the view that impact assessment should go beyond programmes and projects, and explore the influences and roles of the implementing organisations, CTA and partners observed during the planning of the impact study that the development community has not yet come up with a satisfactory impact assessment model that fulfils this requirement. It is in order to address this methodological vacuum that CTA brought several partner organisations/ (i) Centred on the partner organisations/ networks rather than focusing exclusively networks and M&E specialists together to on the interventions, products or services identify a suitable tool, which subsequently led to CTA spearheading the development that CTA supported, of the Capacity-centred Impact Pathway (ii) focused on the impact pathways, rather Analysis (CcIPA) model. than only looking for impact, and The decision to focus on ‘impact pathways’, (iii) facilitated collaboration throughout which is expressed in the second requirement the evaluation exercise – leading to above, is acknowledgement of the complex the direct involvement of about thirty nature of the change processes that are participants from Africa, the Caribbean involved in creating and sustaining and Europe. development impact. As such, the evaluation framework should take account of the (iv) Mobilised strong internal support interactions among the key actors (direct from key staff in the participating and indirect beneficiaries, stakeholders), as organisations. well as the complex human and institutional The first requirement poses a serious relationships that may have a crucial bearing challenge methodology-wise. Within the on how impact occurs. development community, the commonly accepted practice in impact assessment is to identify the ‘significant or lasting changes in people’s lives, brought about by a given action or series of action’. Changes in the capacities of the organisations and networks that implement the development actions are not normally considered as impact, which explains why impact studies are usually carried out separately from the evaluation of organisational capacity development. 54 JOINT IMPACT ASSESSMENT OF CTA’S SUPPORT TO NARI The next section explains how the third requirement (i.e. regarding communication among the participants) was handled effectively during the implementation of the impact study. Regarding the fourth requirement, effective communication and careful planning and tack were applied, especially within CTA where the LME Unit needed to secure funding for this unconventional study. 5.3 Implementing the impact study through different phases The joint impact study involved four distinct phases: planning, quick scan, in-depth study and follow-up. The main activities in each phase are listed below (see tables 1a, 1b, 1c and 1d), with an indication of how the responsibilities were shared among the participants (i.e. CTA, partners and consultants). It was also agreed that each partner organisation has responsibility for managing their case study, including the selection of an external M&E expert, based locally, who will work with the staff member appointed to serve as the focal point. The external expert brings the technical support while the staff member brings the data and information. During the planning phase, CTA and partners recognised the need for an effective implementation strategy that would ensure good communication among the participants as well as their full commitment to the study. A key element of the strategy was to implement the study in two distinct phases – a quick scan of approximately three months, followed by in-depth studies. Finally all the organisations were consulted at all the stages of the study: the methodology design, definition of scope, scheduling and budgeting, and mid-term progress review. Table 1a: Planning phase Task Responsibility Preparation of draft Terms of reference for the impact study CTA Selection of partner organisations & networks CTA Negotiation of contracts with EU-based advisers (from CDI-WUR, ECDPM & MDF & UK) CTA 2-day workshops to discuss methodology and develop road map for impact study (hosted by CTA in Wageningen) CTA, partners & EU-based advisers Negotiating roles/responsibilities between CTA & partners CTA & partners Selection of focal points within partner organisations Partners Preparation of guidelines for the quick scan CTA & advisers Part B: Overview of the CCIPA Model 55 “The external expert brings the technical support while the staff member brings the data and information.” Table 1b: Quick scan phase Task Responsibility Appointment of local M&E experts Partners Backstopping of quick scan methodology through face-to-face meetings, emails, Skype, telephone to guide local teams CTA/ CDI-WUR/ ECDPM Mid-term review workshop (Harare, Zimbabwe - June 10-12) to examine the findings of the Quick Scan (hosted by FANRPAN) CTA, partners & EU-based advisers Peer reviewing of quick scan reports Revision and finalisations of quick scan reports & PPTs CTA/ CDI-WUR/ ECDPM Partners/ local consultants Selection of in-depth study topics Partners Finalisation of guidelines for the in-depth studies CTA & CDI-WUR Table 1c: In-depth study phase Task Responsibility Preparation of inception notes for carrying out the in-depth studies Partners/ local consultants Review of inception notes CTA Implementation of in-depth studies (approximately two topics per organisation) & reporting Partners & local consultants Peer review & revision of in-depth study reports CTA & CDI-WUR Finalisation, peer review & revision of in-depth study reports Partners & local consultants Table 1d: Follow-up phase Task Responsibility Dissemination of the findings All participants Promoting organisational learning from the impact study CTA & partners Further refinement of the methodology. CTA 56 JOINT IMPACT ASSESSMENT OF CTA’S SUPPORT TO NARI 5.4 The CcIPA model explained CcIPA is a synthesis model based on the premise that the performance and impact of organisations or networks depend to a large extent on the state of their capabilities. The CcIPA model is built around three main conceptual components: the Five Core Capabilities (5 CCs) model, the Logic Model and a framework for categorising impact indicators. Each of these models or framework is adapted to a certain degree to fit into CcIPA, with the logic model undergoing the most far-reaching modification. a. Conceptualisation of CcIPA During the planning phase, participants agreed that instead of attempting to construct an entirely new model, the team should draw on the existing array of established evaluation frameworks/models.7 In view of the emphasis on the capabilities of organisations, it was decided that the 5 Core Capabilities (5Cs) model8 be given a central place in the CcIPA model. The five core capabilities applied to the impact study can be summarised as followed: • Capability to act and commit: concerns the ability to work properly, including planning, taking decisions and acting on these decisions collectively. • Capability to deliver on development objectives: concerns the organisations’ skill to ensure that it is producing what it is established to do. • Capability to relate to external stakeholders: this is about building and maintaining networks with external actors (including governmental structures, private sector parties, civil society organisations and in the end their constituencies) • Capability to achieve coherence: concerns the strength of an organisations’ identity, self-awareness and discipline. Each of these five categories was further divided into sub-categories referred as domains. A total of fifteen domains were identified, with two or three key questions per domain to help collect the information needed for assessing an organisation’s capability. As already indicated, the participants agreed to complement the 5Cs model by the Logic model9, with the latter being adapted (see Figure 1 below) to reflect the fact that organisations/ networks learn and change as they engage in development processes, which in turn affects their ability to make an impact. 7 The following framework considered: logical framework/DAC criteria, theory of change, logic model, 5 Core capabilities, outcome mapping, results-based management and participatory impact pathway analysis. 8 The 5Cs model was developed by ECDPM and applied in an extensive impact assessment study financed by the Dutch Government in 2009-10. 9 The logic model was chosen because it is less rigid than logical framework and better suited to evaluation exercise. • Capability to adapt and self-renew: concerns the ability of an organisation to learn internally and to adjust to shifting contexts and relevant trends. Part B: Overview of the CCIPA Model 57 The CcIPA framework combines activities from different programmes, projects or services, whereas the conventional use of the logic model is limited to a single project or programme. Outcomes at the level of the implementing organisation/ network are separated from outcomes at the level of beneficiaries, whereas the common practice is to only differentiate outcomes along the time line (short-, medium-, and long-term). Regarding the impact indicators the participants drew on previous work undertaken under the auspices of Oxfam12 to develop a template, referred to as the ‘impact categorisation table, with a similar format to the 5Cs, containing the following heading: • Impact Categories: which are a list of possible general types of impact • Questions: which are a list of questions to help first investigate the impact/finding/ story and then to narrate the story As shown on the opposite page, the five categories of impact included in the CcIPA model are: (i) wealth/ capital, (ii) political empowerment, (iii) human/ technical empowerment, (iv) social capital and (v) natural resources/ environment. Overall a key feature running through the different constituents is the categorisation of indicators aimed at making the information more manageable as well as facilitating communication among the participants. 12 Chris Roche 1999; Impact assessment for development agencies – learning to value change (Oxfam development guidelines) • Dimensions of impact: which are a list of possible changes or specific domains within the general type of impact/impact category • Description of impact: which are examples of possible changes/impact “The CcIPA framework combines activities from different programmes, projects or services, whereas the conventional use of the logic model is limited to a single project or programme.” 58 JOINT IMPACT ASSESSMENT OF CTA’S SUPPORT TO NARI Figure 3: Logic Model adapted for the CcIPA framework Activities Deliverables Outcomes Outcomes Outcomes Involving multiple programmes and actors (including CTA and partner) over a given period (e.g. the last 10 years). Resulting from the grouped activities. Deliverables may be project outputs or direct services not related to any partnership project. (Organisation / network level) • Short-term • Medium-term • Long-term (Direct beneficiaries level) • Short-term • Medium-term • Long-term (Indirect beneficiaries level) • Short-term • Medium-term • Long-term Categories of impact Categories of Impact (fixed) Dimension of Impact (Not fixed, not exhaustive) Descriptive of Impact (Not fixed, not exhaustive) Wealth / Capital Income Increased Revenue for farmers Political empowerment Policy New policy adopted Human / Technical Empowerment New skills Use of new ICT tools Social capital Network New organisations joined Natural / Environment Climate Change New measures taken to mitigate changes Questions Why? What for? Why not? For whom? Unplanned? Etc... Part B: Overview of the CCIPA Model 59 b. Application of CcIPA The Capacity-centred Impact Pathway Analysis (CcIPA)11 framework incorporates three inter-related elements, as shown in the horizontal segments of the model (moving from left to right): (i) Outputs/Deliverables: These are outputs at the level of the partner organisation or network as a direct result of CTA’s partnership with the organisation or network. They are defined in relation to the set of project interventions or products and services. Deliverables may be directed at the partner’s capacity building needs or made up of products/ services aimed at the partner’s beneficiaries. CTA’s support may be only partial but must be considered significant. (ii) Changes in the capabilities: of the CTA partner organisations/ networks. These can be also described as outcomes at level of the organisation or network. (iii) Effects (medium-, long-term outcomes / impact): on the partner’s direct and indirect beneficiaries or on the wider society that can be attributed to the deliverables and/or changes in the partner’s core capabilities. The application of the CcIPA framework will produce a graphical representation of the changes resulting from diverse interventions over time. 11 The development of this synthesis model has been proposed and led by Ibrahim Khadar, in collaboration with the participants at the three preparatory meetings held at CTA’s headquarters in Wageningen, the Netherlands. “Deliverables may be directed at the partner’s capacity building needs or made up of products/ services aimed at the partner’s beneficiaries.” 60 JOINT IMPACT ASSESSMENT OF CTA’S SUPPORT TO NARI Figure 2: The Capacity-centred Impact Pathway Analysis Model CHANGES 5 CORE CAPABILITIES PROJECTS & ACTIVITIES OUTCOMES AT BENEFICIARIES LEVEL OUTPUTS / DELIVERABLES PRODUCTS & SERVICES OUTCOMES ON WIDER SOCIETY OUTCOMES AT LEVEL OF ORGANISATION Part B: Overview of the CCIPA Model 61 LESSONS LEARNED FROM IMPACT STUDY AND NEXT STEPS 6 62 Section 2: Context in which the CCIPA model was designed Concerning the participating organisations Concerning the wider development community Improved understanding of the impact pathways • CcIPA contributes to an emerging new practice which is stronger self-evaluation. So, the utilisation in the evaluation community in terms of methods and quality standards for (assisted) selfevaluation is in my view one of the use results. • Implications of core capabilities (especially capability to relate) • Growth of organisations/ networks • Feedback from beneficiaries • Better appreciation of CTA’s support Baseline information • Nine quick scan reports (finalised) • Nine in-depth study reports (partially completed) Enhanced M&E capabilities • Strengthening of CTA’s M&E framework Joint learning • The evaluation has resulted in new arrangements and procedures how partners collaborate. • Collaboration between ANAFE and RUFORUM (e.g. joint article for the AfrEA conference) • CcIPA is also inspiring the evaluation community in terms of 1) combining different methods, and 2) adapting them to the specific information needs of the partners. In other words, a kind of calibration was done, designing methodologically to the learning needs of the partners. Next steps • Intensify dissemination of findings and CcIPA model; • Carry out a second round of impact studies; • Further clarification of impact pathways. Part B: Next Steps 63 PART C: KEY FINDINGS FROM ALL FIVE REPORTS (THIS SECTION SYNTHESISES THE FINDINGS OF ALL FIVE QUICK SCAN REPORTS) The Capacity-centred Impact Pathway Analysis Model CHANGES 5 CORE CAPABILITIES PROJECTS & ACTIVITIES OUTCOMES AT BENEFICIARIES LEVEL OUTPUTS / DELIVERABLES PRODUCTS & SERVICES OUTCOMES AT LEVEL OF ORGANISATION OUTCOMES ON WIDER SOCIETY BRIEF DESCRIPTION OF THE ORGANISATIONS 7 66 CTA-CAFAN JOINT IMPACT ASSESSMENT AFRACA SPC The African Rural and Agricultural Credit Association (AFRACA), established in 1977, is a regional association of sub-Saharan financial and non-financial institutions involved in promoting rural and agricultural finance. AFRACA’s goal is to build a strong and sustainable regional network of institutions and agencies that provide opportunities for learning amongst members and promotes rural and agricultural finance aimed at empowering rural and agricultural communities in the continent. AFRACA is a member-based organisation with a secretariat based in Nairobi, Kenya. The Secretariat of the Pacific Community (SPC) was founded in 1947 under the Canberra Agreement. SPC is an international development organisation owned and governed by 26 member countries, including 22 Pacific Island countries and territories and four founding member nations. The organisation supports the Pacific’s vision for a secure and prosperous community whose people are educated, healthy and manage their resources in an economically, environmentally and socially sustainable way. SPC helps member countries achieve this vision through programmes encompassing agriculture, forestry and land, fisheries and aquaculture, health, water and sanitation, transport and energy, statistics, mineral resources and geoscience, education, gender, youth and human rights. Members are: American Samoa, Cook Islands, Federated States of Micronesia, Fiji, French Polynesia, Guam, Kiribati, Marshall Islands, Nauru, New Caledonia, Niue, Northern Mariana Islands, Palau, Papua New Guinea, Pitcairn Islands, Samoa, Solomon Islands, Tokelau, Tonga, Tuvalu, Vanuatu, and Wallis and Futuna]. Founding member nations are: Australia, France, New Zealand and the United States of America. NARI The National Agricultural Research Institute (NARI) of Papua New Guinea (PNG) was established in July 1996 as a publiclyfunded, statutory research organisation by an Act of National Parliament. Its purpose is to conduct and foster applied and adaptive research into: any branch of biological, physical and natural sciences related to agriculture; cultural and socioeconomic aspects of the agricultural sector, especially of smallholder agriculture; and, matters relating to rural development and of relevance to PNG. NARI is also responsible for providing technical, analytical, diagnostic and advisory services and up-to-date information to the agriculture sector in PNG. NARI’s headquarters are based in Lae, PNG. Part C: Brief description of the organisations 67 WOUGNET PROPAC The Women of Uganda Network (WOUGNET) is a non-governmental organisation established in May 2000 by several women’s organisations in Uganda. Its aim is to promote and support the use of information and communication technologies (ICTs) as tools to share information and address issues of sustainable national development collectively. WOUGNET currently has over 104 women’s organisations as members, of which the majority are located in urban areas and district towns where there is Internet access; and a few in rural areas. WOUGNET is a service-oriented, network-type organisation working in the area of ICTs for development. Its niche is addressing the gender divide in access to, and use of, ICTs. Its programmes are guided by two pillars - ICTs as tools for development, and gender responsiveness of ICT programmes and policies. Created in 2005, the Regional Platform of Central African Farmers’ Organisations (PROPAC) represents 10 national farmers’ organisations (NFOs) based in 10 member countries in the sub-region. It was set up for the purpose of harmonising the NFOs’ strategies and initiatives to ensure their concerns are reflected in the implementation and assessment of national, regional and international agricultural and rural development policies and strategies. Seeking to improve the living conditions of small-scale farmers and rural communities, the PROPAC strategy is based on four lines of approach: giving farmers’ organisations greater structure, boosting the capacities of small producers, lobbying and advocacy on behalf of family farming and the rural sector, promoting the gender-based approach. PROPAC’s headquarters is located in Yaoundé, Cameroon. 68 JOINT IMPACT ASSESSMENT OF CTA’S SUPPORT TO NARI “WOUGNET is a service-oriented, network-type organisation working in the area of ICTs for development. Its niche is addressing the gender divide in access to, and use of, ICTs.” Part C: Brief description of the organisations 69 SELECTED FINDINGS ON THE CAPACITY OF THE ORGANISATIONS 8 70 CTA-CAFAN JOINT IMPACT ASSESSMENT 8.1 Capability to act and commit AFRACA was rated very strongly on its action-oriented leadership as well as organisational commitment. The leaders actively participate in AFRACAs’ strategies and activities thus enabling the organisation to achieve its mandate and grow despite having a lean secretariat. The levels of commitment to the organisation’s drive is high across the board. Decision-making within the organisation is very participatory. There was a general feeling, however, that the decision-making process is, at times, not based on evidence, leading to uninformed decisions. NARI’s capability to act and commit was rated fairly well. When assessing each of the domains separately, the results reveal that NARI’s leadership is inspiring and actionoriented, and NARI’s organisational culture is committed to change and in agreement on its priorities for achieving organisational outcomes. Whilst NARI’s decisions are evidence-based, they are not necessarily participatory. NARI staff felt that the decision-making process in NARI is less participatory than it should be. SPC has an action-oriented leadership with competent staff committed to successful and efficient management of programmes. SPC also has the ability to involve its staff in decision-making, mostly at project level, which enhances productivity and increases staff confidence. However, decision-making mechanisms and learning structures need to be in place for successful future strategic planning. SPC has strong project management skills backed by highly qualified staff, thus positioning the organisation to attract donor partners. Most staff were project funded, impacting the sustainability of technical service delivery once the project cycle came to an end and leaving a gap until the next phase began, usually a few years later. A narrower focus among new projects with no further increase in core staff funding meant recruitment of staff was limited or necessary skills and knowledge were lacking, creating anxiety amongst the staff and impacting on their output. WOUGNET has a strong action-oriented leadership with a good ability to manage and implement projects, whilst liaising well with staff and stakeholders. Project design and implementation is usually participatory with several staff actively involved in the development of project proposals. WOUGNET is a committed organisation very able to act on collective decisions made. Internaldecision making structures are good, although they could be more participatory. Staff, including junior members, would benefit from more opportunity and encouragement to provide input at appropriate times (and not after decisions have already been made). Part C: Selected findings on the capacity of the organisations 71 8.2. Capability to adapt and self renew PROPAC’s Permanent Secretariat has proven strategic planning and activity implementation skills. PROPAC fulfils its obligations, particularly those entered into with its external partners. Almost all of the activities are initiated by the organisation’s president according to a result-driven approach, while the decisionmaking process is inclusive. However, most of the staff recommendations fail to be taken into consideration, yet there is unanimous approval for the aims being sought and the values being promoted. PROPAC is therefore highly rated as regards this capacity (4/5). AFRACA received a fairly good rating on taking external environmental changes into consideration. This was mainly because of AFRACA’s diverse membership and its continent-wide coverage of activities which the secretariat always have to take into account during planning. The internal environment, however, rated slightly lower compared to the external environment. Many of the staff felt it was still too soon to comment on staff incentives and HR policies since the majority of them were still new. Availability of M&E processes was probably rated the lowest given the lack of M&E structures within the secretariat, which has resulted in poor follow-up of activities. NARI’s capability to adapt and self-renew was rated fairly high although it is worth noting that it was rated the lowest amongst the five core capabilities. NARI’s internal systems and processes are not sufficiently transparent and participatory. The ratings further reveal that NARI’s communication channels are less open and inclusive and that NARI’s M&E system does not allow feedback from its stakeholders, therefore generating minimal support for improving projects’ performance. Although NARI’s capacities in the areas of M&E and internal environment were rated fairly moderately, NARI’s ability to analyse the external environment and incorporate these changes into the internal decision-making process of NARI was rated high. 72 JOINT IMPACT ASSESSMENT OF CTA’S SUPPORT TO NARI “WOUGNET conducts annual self and peer assessments.” SPC’s capability, whilst better than average, has room for improvement. SPC went through some major organisational re-structuring during the period 2009-2013, where functions of two regional organisations (South Pacific Applied Geoscience Commission and the Secretariat of the Pacific Board for Educational Assessment) were integrated into SPC. This critical period also saw a 20% increase in CTA assistance to SPC, which greatly helped build the capacity of staff to improve regional services. SPC’s ability to take changes in the external environment well into account is recognised and thus considered a strength. SPC’s internal environment is evolving, with a whole new management team in place currently trying out new monitoring systems. The new M&E process, which has been in place for a few years now, is helping SPC improve feedback to donors and to national stakeholders. WOUGNET’s secretariat has a strong capability to manage the external environment. It was noted that the external landscape is monitored with information used to guide the organisation to flexibly adapt to change. SWOT analyses conducted during strategic review meetings and annual meetings helps to devise means of minimising threats and weaknesses, as well as taking advantage of opportunities and its strengths for performance improvement. Managing of internal environment scored well. Having adequate incentives for a learning culture (self-assessment, taking action to apply learning) was rated to be strong. WOUGNET conducts annual self and peer assessments. The lessons learnt, best practices, and areas for improvements from the self and peer assessments are shared at the beginning of the new year during feedback meetings. It was however noted that the secretariat need to do more to translate learning by individuals into action for the benefit of the organisation. The internal communication at WOUGNET was noted to be open about constraints and failures in programme implementation. All staff are free to talk about failures, constraints in implementation of activities, and required remedial actions through official emails accessible to all staff, and in staff meetings. However, it was pointed out that staff should always endeavour to provide genuine views. Implementation of the human resource policy was rated to be moderate. It was noted that though the organisation’s policy stipulates human resource issues are shared with staff at the time of joining the organisation, staff do not read it and hence are not familiar with the policy. Due to the size of the secretariat and available resources, it may not be feasible to implement some aspects of the policy hence the need to institutionalise annual staff retreats to reflect on organisational policies, their implementation, and implications for staff under the prevailing contexts. Part C: Selected findings on the capacity of the organisations 73 8.3 Capability to deliver PROPAC’s ability to adapt to the changing external and internal environment is based on an ongoing and inclusive learning process thanks to the many CTA-funded capacitybuilding seminars/workshops. PROPAC employees have therefore developed knowledge and expertise focused on a wide variety of skills and themes. The entity also keeps itself up-to-date with its partners’ activities and attitudes thanks to the reports drawn up at frequent intervals. These reports enable the organisation to make regular readjustments to its strategies in the light of external factors. PROPAC uses a reporting system for its programmes in order to provide information about certain activities but the institution does not have an effectively independent monitoring and evaluation system. However, it does has excellent adaptation and self-renewal capacities, as reflected by its 4/5 rating. AFRACA project-related questions yielded a high rating. This was because staff felt that new recruitments at the secretariat had greatly enhanced AFRACA’s capabilities in implementing and developing new projects. Quality assurance scored fairly well as the majority of staff felt they were aware of quality assurance systems at the secretariat. With regard to infrastructure, staff felt the infrastructure at the secretariat was adequate in the current situation. However, they seem optimistic for improved infrastructure in the near future as more activities are undertaken by the secretariat. NARI’s capability to deliver products and services was rated highly. NARI has a long history of successful design and implementation of projects, both with national and international partners. It is a preferred partner for many funding agencies to implement projects aimed at improving the livelihoods of rural women and men. As a result, this high rating of NARI’s capability to deliver is not a surprise. In fact, this capability could have been rated very high. Yet, NARI has assessed itself moderately in this area of capability. SPC has skilled and competent staff, which indicates a robust HR policy based on merit, ensuring quality and timely delivery of outputs. The staff, however, are inadequately resourced given the infrastructure required to deliver outputs. While the score was better than average, there is room for improvement. SPC has employed highly qualified staff with appropriate skill sets capable of delivering essential technical services that national countries could not themselves deliver. 74 JOINT IMPACT ASSESSMENT OF CTA’S SUPPORT TO NARI “Quality assurance scored fairly well as the majority of staff felt they were aware of quality assurance systems at the secretariat.” This was reflected in a high score for ‘competencies’. The building of a strong senior leadership team over the review period, which is responsive to stakeholder and sector feedback, translates to production and delivery of good quality technical services, where SPC engages with national collaborators and donor partners to increase the value of activities. A moderate scoring on quality assurance is a good indication for SPC to improve how it addresses feedback from stakeholders on its knowledge products and services and put in place feedback mechanisms. LRD responded with the establishment of the email helpdesk, to improve communications with national stakeholders and to receive and respond to queries. This is achieved through use of action sheets and the new template developed to operationalise the Information Communication Management plan. These tools help to capture, track, and document decisions taken during review meetings and the corresponding actions taken to effect the decisions. WOUGNET has a strong capability in quality assurance mechanisms for the organisation’s work. The organisation policy puts in place a quality assurance committee comprising of the executive director and the programme managers. It was noted that top management has to review and endorse all information materials produced by WOUGNET before they are submitted to various audiences. WOUGNET has a strong ability to deliver on products and services and is able to implement activities well. It was noted that staff are competent and have the requisite skills to perform their duties. The well-developed and constantly improving website, as well as a functional e-mail system at the head office, were noted to ease communication, thereby enabling staff to access internal and external knowledge and information. All head office staff have access to a computer and stable internet. However, power black outs, poor internet connectivity experienced by field staff as well as inadequate equipment (notably computers and cameras) were cited as gaps which need to be addressed. All projects undertaken fit into the overall strategy of using ICTs for sustainable development with an emphasis on the livelihoods of women and youth. Decisions regarding project strategies and processes are well documented. PROPAC Boasting fairly high technical skills in terms of providing outputs and services, the staff specify that their capacities in this area have increased twofold since the organisation started working together with the CTA. Conversely, there is still a serious shortage of qualified employees to deal with the size of the workload, hence the trend towards mixed appointments. The projects that are undertaken are consistent with the organisation’s overall aims. However, its lack of a quality management system is attributed to the inadequate quality control applicable to the processes and services on offer. (4/5). Part C: Selected findings on the capacity of the organisations 75 8.4 Capability to relate AFRACA secretariat scored the highest on this capability suggesting a high capability to relate. A common reference point was the just concluded Fin4Ag International Conference, which may have influenced the high ratings. It can therefore be assumed that this may not have been the case perhaps two or three years ago. Being a network, its ability to bring together members and liaise with other financial networks was considered high. This has enabled the organisation to successfully execute learning and experience-sharing exchange tours within and even beyond the continent. NARI received a high rating for this capability, reflected by the organisation’s ongoing commitment to agricultural research and its prominent role as a leader in the agricultural research field in the country (and possibly also in the Pacific region). NARI has no problems developing and maintaining networks and relationships with its external stakeholders. Furthermore, NARI has a high level of influence among many of its national collaborators and takes all necessary precautions to ensure its image and credibility is not diminished in any way. 76 JOINT IMPACT ASSESSMENT OF CTA’S SUPPORT TO NARI SPC scored the strongest here, which is a strong indication of SPC’s niche role as a technical agency in the region, delivering services that are not able to be delivered at the national level. An example is SPC’s genebank facility. SPC is well known in the region. It has a long legacy that goes back to colonial days in the region. With a stronger CTA partnership over this critical period, SPC’s own work to improve how it addresses stakeholder needs, and with senior management putting in place stronger HR policies to improve the work environment, SPC has established a very good relationship with outside donor partners. This translated to an increase in funding to Land and Resources Division from €4.4 million in 2009 to €6.4 million in 2012. WOUGNET has a strong capability to relate and is highly engaged and influential in its partnerships. WOUGNET engages in relevant networks and partnerships which add value to the network’s objectives and members. It has worked with six different categories of organisations including: Uganda government ministries, departments and agencies; international organisations; continental and regional organisations; research organisations; universities; NGOs; and funding agencies. It has collaborated with over 30 individual organisations. WOUGNET has strong social credibility and political legitimacy within its partnerships and networks. This was attributed to WOUGNET serving on a number of committees established by various stakeholder organisations. 8.5 Capability to achieve coherence PROPAC: The CTA has funded several PROPAC activities focused on advocacy, development policies and other key themes. The extensive advocacy /lobbying activities have enabled it to boost and maintain its influence at all levels, allowing the organisation to improve its international, continental and sub-regional positioning. It has over 20 public and private bilateral and multilateral partners at both national and international level. Its enviable reputation has served to consolidate its terms of reference. This is an area in which PROPAC truly excels, as reflected by its top rating (5/5) for its capacity to establish, maintain and build upon its partnerships. AFRACA staff had a high regard on the capabilities of internal management hence the high ratings for governance structure, vision and strategy. Similar to the other domains on HR matters, the rating was relatively fair. The general feeling was that the management is very supportive of staff and invest resources in nurturing and growing its staff. Most past staff have gone on to better jobs with other organisations, which is an indication that their skills grew during their time at AFRACA. The current staff are very clear of the organisation’s strategies and most of their operations are guided by the strategic plan. There is an HR policy which is regularly reviewed. In terms of competencies, there was a feeling that some members of staff lacked adequate competencies to handle some assignments under their responsibilities. NARI’s capability to achieve coherence was considered very high, particularly in light of past work that had been done with the Australian Government’s assistance in the area of corporate planning and institutional capacity development. NARI’s strategic objectives, developed through the lens of ‘science and research for development impact’ as opposed to ‘science for scientific excellence,’ has improved NARI’s visibility and its justification for existence in the country. NARI, subsequent to this paradigm shift, aligned and realigned its internal systems and processes so that its internal processes can enable the achievement of its new strategic vision – science and research for development impact. NARI also realigned its human resources, including their enabling structures, to facilitate the achievement of NARI’s organisational objective. Part C: Selected findings on the capacity of the organisations 77 “Limited core funding has restricted investment in core resourcing.” An integral part of NARI’s HR realignment processes resulted in NARI identifying and developing competency-based terms of reference (TOR) for its staff. However, when assessing NARI’s capability using the 5Cs model, NARI’s performance was rated not very highly. A possible reason for this score is that many of the staff members who participated in this survey were new to NARI and could have been unaware of the processes that NARI has developed since 2007, which are currently in use within NARI, including its M&E system and competency-based TORs. SPC scored moderately for its internal management capabilities. Up to 70% of LRD funding is short-term project funding by a multitude of donors. Each of these projects typically has different areas of coverage, with national, multi-country or regional scopes and reporting requirements, although the common actors (e.g. farmers) are the same. This has created a heavily ‘siloed’ environment. Limited core funding has restricted investment in core resourcing. Core resources such as M&E and finance are often housed within projects, which has posed difficulties in presenting a coherent approach on strategy. The new LRD restructure was a response to this and provides a platform to build on. Key engagement processes with countries also need to be strengthened. SPC has traditionally used joint country strategies to determine country needs. While the process is inclusive, this has also led to unfettered demand for SPC services that have to be delivered with limited resources. 78 JOINT IMPACT ASSESSMENT OF CTA’S SUPPORT TO NARI SPC has an M&E framework in place, reflecting its commitment to achieving sustainable development outcomes through enhancing its results focus in the areas of monitoring, evaluation, learning and reporting. This commitment focuses on learning for excellence as well as accountability to members and donors. WOUGNET has a moderate (moving towards strong) capability to achieve coherence in its mandate, vision and strategy. The internal management support and governance structure scored well. Staff development is facilitated through effective delegation of work, mentoring, and capacitybuilding opportunities accessed through WOUGNET. It was noted that WOUGNET provides space and online infrastructure for those who are proactive in taking advantage of numerous learning opportunities to improve their individual competencies. Staff are well aware of the organisation’s strategy. This was reflected by various staff ably representing WOUGNET and talking about the organisation in various forums, all staff having access to the organisation’s key policy documents, and their participation in the development of proposals to mobilise resources for implementing programme activities. There are challenging issues with the HR policy and implementation, where there is moderate capability. Though staff are recruited on merit basis, it was noted that the organisation does not have wellstipulated procedures for taking on interns and volunteers. WOUGNET does not have a stand-alone HR policy, but HR issues are articulated in the organisation policy. This was cited as a possible cause for limited staff awareness of the HR policy. PROPAC There is room for improvement in terms of the management’s support for the permanent secretariat. However, PROPAC’s development approach and its strategies are adopted on an inclusive basis, so they enjoy the support of all the stakeholders. However, there is some degree of discrepancy between the organisation’s culture, the extent to which the provisions in the procedural manual are applied and the customary practices. The human resources policy is based on the competence criterion but problems are reported in the case of good governance, human resources and financial management and working conditions. The International Fund for Agricultural Development (IFAD) also acknowledges that PROPAC staff wages are below the applicable standards. Moreover, some of the provisions in the procedural manual are not being applied. Consequently, the organisation has received a low rating for this capacity (2/5). Part C: Selected findings on the capacity of the organisations 79 KEY FINDINGS FROM LOGICAL FRAMEWORK ANALYSIS 9 80 CTA-CAFAN JOINT IMPACT ASSESSMENT 9.1 AFRACA Activities CTA-supported activities included conference and study visits, such as: 14th AFRACA General Assembly meeting; study visit on ‘Micro-finance in Africa’ in Kenya; study visit on the ‘Role of Women in the Development of Micro-finance in Africa’; The 4th AFRACA Central Banks Forum; Subregional workshop: Niamey, Niger; AFRACA Southern Africa sub-regional workshop on value chain financing; three day ‘lessons learnt’ forum, Dakar, Senegal; Fin4Ag Conference, Nairobi, Kenya, 2014. CTA-supported knowledge management and research activities, including: publication on experiences in African value chain finance; study on ‘collateral management for smallholder-inclusive value chain finance’; study on ‘mapping best practices on agricultural value chain finance among African financial institutions’; study on ‘opportunities for value chain finance for intra-African food trade’. Outputs/ deliverables Conference report compiled; logistical notes; speaker presentations; feedback reports from sponsored participants; learning and knowledge sharing events; participation of ACP nationals in conferences. Quarterly publication of experiences in AFRACA communication channels; study reports; conference papers; present findings during various AFRACA forums. Outcome at organization level Facilitated and promoted learning among AFRACA member countries; provided a platform for policy dialogue to influence policy in agriculture and rural finance; achievement on shared objective of strengthening smallholder inclusive value chain finance in Africa through improved supply of finance and improved support environment; increased profile and competitiveness of the network manifested through membership growth and new partnerships and collaborations. Generation of new knowledge in agricultural finance; built research capacity within AFRACA; promoted development of innovative rural finance practices and products among AFRACA members. Part C: Key findings from logical framework analysis 81 Outcome at individual level Expected outcomes at individual level were: benefit from sharing experiences with other participants on different agri-finance models, financial innovations, regulatory frameworks and best practices in rural finance; improvement of own institutional operational systems, service delivery methodologies, products offered, and outreach mechanisms; improved ability to disseminate to other members of their organisations, best practices and innovations learnt from the conferences and exchange visits; enrich individual member experiences and enable expanded growth in operations of the institutions so that the benefits can trickledown to rural entrepreneurs. Impact CTA support has enhanced the visibility of AFRACA and enabled it to offer better services to members. By supporting the tours and forums, members have felt the need to join and continue being AFRACA members as they see the value in it. This in turn has attracted more members which is demonstrated by the growth in membership over time. Exchange study tours have built the capacity of AFRACA staff and members in several critical aspects necessary for their operations. Most members are now able to design, develop and run agricultural and rural credit facilities. On the other hand, AFRACA secretariat is able to support their members technically, as their capacity on several aspects of rural finance has been enhanced. 82 JOINT IMPACT ASSESSMENT OF CTA’S SUPPORT TO NARI 9.2 NARI Activities/ Outputs A total of seven activities, ranging from case studies, learning workshops, refresher learning workshops and training of trainer (TOTs) events were supported by CTA within NARI from 2004 until 2014. 55 participants participated in four TOT sessions on agricultural science, technology and innovation (ASTI) systems, and identifying and analysing farmer’s innovations. The application of the knowledge gained in the TOT sessions on ASTI systems resulted in the compilation of several case studies. Three out of the four case studies related directly to the TOT sessions on ASTI systems; undertaken to test/apply the knowledge and skills gained in the TOT sessions. All case studies, except one, were led and managed by NARI. The key deliverable from each of these case studies is the study report. In all instances, a case study report with relevant recommendations has been compiled. These reports are available within NARI. Outcome at organization level Enhanced capacity of NARI staff to undertake case studies in the context of ASTI systems; new knowledge among NARI staff of agricultural innovations systems; new actors identified for partnerships in ASTI systems; improved and strengthened ability for NARI to interface between scientists, farmers and other stakeholders; informed decision making by NARI in linking research to extension; and new information and findings available to NARI for use in policy briefs, new research proposals, and in contributions to development forums. NARI staff who have attended learning workshops or TOTs in ASTI systems acquire the relevant skills and knowledge for conducting TOTs on ASTI systems; NARI improves its vision to undertake demand-led design of research for development programmes and projects; Improved management of NARI’s NAIS with operators identified and established to maintain the NAIS; Increased efficiency of NARI staff in management of NARI’s NAIS; Increased visibility for NARI, particularly among other participating ACP professionals and regional organisations; Improved capacity for NARI to generate and disseminate information; Reduced communication costs; andEnhanced capacity of NARI to build and maintain partnerships and networks. Part C: Key findings from logical framework analysis 83 Outcome at individual level Greater confidence and capacity in application of the AIS concept in analysing and evaluating farmers’ experimentation and innovations; enhanced capacity of three staff in use of ASTI systems; exposure to and awareness of the AIS approach for participating NARI staff; farmers’ own recognition of their innovative capabilities; further debate on improving extension and advisory services; development of improved strategies and policies on extension and advisory services; Improved networking and partnerships; Increased understanding and knowledge of how ST and ICT can contribute to supporting the development process; Insight into proven strategies for research, teaching and innovation in agriculture ; knowledge and awareness of potential opportunities for research and innovation in Pacific agriculture and building on successful initiatives; Improved efficiency in accessing, developing, exchanging and disseminating information; Improved IT knowledge. Impact Apart from within NARI and among NARI staff, there was little evidence of impact at intended beneficiaries level. CTA has enhanced NARI’s capacity throughout their ten-year partnership in a number of areas, including ASTI systems and Web 2.0 for development. The capacities of other individuals and organisations have also been enhanced as a result of CTA’s support. These institutions include universities, government departments, NGOs, research institutions and women’s groups. The impact of CTA and NARI’s joint efforts is particularly notable in one dimension of impact - human and technical empowerment. This is applicable to both ASTI and Web 2.0 for development capacity areas. Individuals who attended the various learning and TOT workshops are the direct and major beneficiaries of CTA’s support. In ASTI systems, no evidence demonstrates any lasting changes in NARI for the other areas of impact, including income generation and policy development. However, the achievements under Web 2.0 and social media for development are remarkable, although small in terms of scale. Positive gains have been made in almost all categories of impact in this area. 84 JOINT IMPACT ASSESSMENT OF CTA’S SUPPORT TO NARI 9.3 SPC Activities/ Outputs Two ASTI noni study reports (for Tuvalu and Samoa); Two TOT sessions. CTA-funded Pacific study tour in floriculture. CTA funding supported SPC activities e.g. Regional Youth and ICT (blog) training, ACP Comp, PYiA Strategy, Social media and web 2.0 in Fiji and Vanuatu, CTA Youth Consultations in the Netherlands; 8 policy briefs; SPC provides content for CTA’s Spore and ICT Update. CTA provides SPC KM updates; Pacific Agricultural and Forestry Policy Network (PAFPNet) mailing list used to disseminate CTA publications and resource materials to 22 PICTs; LRD website showcases CTA publications; portal hosts CTA information/publications; produced 7 policy briefs, 1 promo video, 1 regional strategy, 2 newsletters, and 1 radio interview. Designed and maintained seedling nursery; sold 200,000 seedlings to farmers; small-scale farmer with 8 ha of farmland; feeds into NZ export market Outcome at organization level Skills enhancement in ASTI use; enhanced knowledge on ‘noni’; ASTI relevant to Pacific. Good diagnostic tool; stimulated work in floriculture. Outcome at individual level Skills enhancement in ASTI use; enhanced knowledge on ‘noni’; ASTI relevant to Pacific Skills enhancement; supported resource mobilisation strategies of partners; facilitated establishment of links between SPC and other partners; access to publications and resource materials; implementation of joint projects with CTA as an associate partner; produced IEC materials. Increased visibility and reputation of SPC. Confidence in using social media platforms; improved use of web 2.0 tools effectively; access to publications and resource materials. 3 nursery models adopted by national ministry for other farmers around the country; growth of nurseries signifying enhanced skills and learning by farmers; facilitated training of local farmers funded via AusAID; designed seedling storage saving system – disaster preparedness and during Cyclone Thomas in Fiji saved 29,000 seedlings. Part C: Key findings from logical framework analysis 85 Impact Perhaps the biggest positive impact of CTA-funded work to emerge from the in-depth analyses of two selected activities is their catalytic effect and impact on niche areas for SPC. Today, Web 2.0 training and PAFPNet are core services being delivered by SPC on a programmatic basis. Both areas, initially funded through CTA support, have since received additional donor support. Today, PAFPNet is the only portal in the region that provides for cross-fertilisation of ideas on crop and livestock husbandry, pest management, agribusiness and agricultural research. At the individual level, these training activities have prompted further uptake by youths and individuals who have gone on to develop further knowledge products. Training has also afforded better job opportunities for some individuals. 86 JOINT IMPACT ASSESSMENT OF CTA’S SUPPORT TO NARI 9.4 WOUGNET Activities/ Outputs Skills enhancement of WOUGNET staff E.g. CTA supported the progamme manager for gender, ICT and policy advocacy to participate in a training workshop on media, communications and policy advocacy. 3 WOUGNET staff were trained by CTA on internet-based communication, which included web 2.0, designing websites, and uploading documents. 4 WOUGNET staff participated in training by the United Nations Institute for training and research on Web 2.0 and social media, and an e-learning course on innovative collaboration for development. 15 WOUGNET staff were trained and an M&E framework was developed. 56 people have been trained by WOUGNET in Web 2.0 trainings Development of a 3-year ICM strategy. Access to CTA publications and enhanced access to agricultural information Outcome at organization level Trainings have diversified sources of revenue for WOUGNET. Increased visibility and reputation for providing Web 2.0 training in Uganda has positively impacted on WOUGNETs credibility. WOUGNET staff equipped with knowledge and skills on web 2.0, creating an in-house capacity to continue training of other staff and members. The training on media, communication and advocacy increased staff appreciation of the need for a clear media strategy for effective advocacy. This propelled WOUGNET to produce an outline of a media strategy indicating the various channels to use to reach specific audiences. This also contributed to enhanced capacity in advocacy. WOUGNET’s coordinator and rural projects manager, northern Uganda wrote a report on the outcomes of the regional validation workshop for the information and communication management strategy development facilitator’s guide. WOUGNET staff participated in the Ink4Dev workshops that took place in Namibia (2009), Kampala (2010) and Entebbe (2011). Part C: Key findings from logical framework analysis 87 Outcome at organization level (Continued) Enhanced capacity to support ICT use and develop ICT programmes for greater developmental impact; Tested and proven processes for WOUGNET to use in introducing its member organisations to ICM so that they can appreciate it and also embrace and develop their own ICM strategies at organisational levels; andEnhanced networking and learning from other ACP countries through stories on the ICM wiki. Kubere Information Centre (KIC) was established in Apac town. KIC acts both as an information resource point and supports project implementation and two-way linkages with the women farmers. From the centre, women farmers can access training in agricultural practices, as well as in the use of ICTs, such as mobile phones. Group members were also are also able to share experiences with each other around best practices and new ideas. In addition to the direct work with the rural women’s farming groups, the KIC participated in an e-society programme designed to foster collaboration between local government and civil society in the district through the use of ICTs. Another rural information centre in Amuru district, called Riber-Ber Information Centre, to help farmers access market information using mobile phones. Using a combination of ICTs which included radio, mobile telephones, listening clubs, and face-to-face meetings, women farming communities were equipped with skills to access farming information to improve their farming as well as increase yields. Outcome at individual level Staff delivery of the trainings has improved their confidence and skills as facilitators/trainers in Web 2.0 Improved functioning and performance of roles; and new income streams for participants in Web 2.0 trainings: One woman activist noted that her knowledge on Web 2.0 and social media has enabled her to take her advocacy work to another level and reach many more people. A youth used social media to mobilise funds to support their project’s activities. Opportunities to venture into new income streams: One Web 2.0 trainee claimed that he had trained other people thereby increasing income. 88 JOINT IMPACT ASSESSMENT OF CTA’S SUPPORT TO NARI Outcome at individual level (Continued) Access to information materials such as the ICM facilitator’s guide and the ICM users’ manual developed by CTA guided WOUGNET in drafting the terms of reference for the development of the ICM strategy. WOUGNET staff use the CTA technical guides on various technologies and enterprises to advise farmers through the question and answer services provided by KIC. The farmers also access the technical guides which enable them to acquire knowledge on production of their enterprises. in 2013/14, 123 women and 520 men accessed information both online and from the information centers at KIC, Ribe ber Information Center and the CDTC. Impact The intervention strengthened the capacity of individuals to use ICTs, notably radio, to access agricultural information and services. It also ensured better acquisition of farming skills and contributed to the adoption of modern farming practices and technologies. The increased access to information triggered changes in farmers’ farming knowledge and skills. With respect to social capital, the strengthened groups are recognised by local government authorities. The groups have also been attractive to other development agencies thereby enabling their members to continue to access agricultural advisory services as well as equipment. Groups have facilitated networking and offered a platform for sharing experiences and continued learning. Collective action through the Village Savings and Loan Association, labour exchange, and, in some cases, produce bulking and marketing, have enabled group members to save money, take out loans to stop distress sales, and to access better markets. Part C: Key findings from logical framework analysis 89 Impact (Continued) With regards to impact on wealth, groups acquired productive assets like oxen and ox-ploughs and also enabled their members to own livestock, notably goats and pigs. At the individual level, seven in every ten respondents noted that their incomes had increased. This enabled groups to acquire physical assets, construct permanent households and send their children to better schools. The increased incomes have also contributed to an increased proportion of households reporting owning a radio (86%) and a mobile phone (75%) in 2015, compared with 74% and 2% who reported the same respectively in 2005. The adoption of planting trees (citrus and multi-purpose trees), as well as abandoning the practice of burning garden waste and resorting to trash lines, has had a positive impact on the environment. 360 women farmers from 12 groups were equipped with skills on how to use ICTs to access information to improve their farming. WOUGNET is undertaking a new, connected initiative, Strengthening University – Farming Community Engagement (SUFACE), in partnership with Makerere University, involving women farmers involved in the EAAI project. Access to agricultural information triggered some women farmers to engage in new enterprises including tree planting, and new methods of bee keeping, among others. Members established their own mailing lists in order to target their communications to particular audiences. 90 JOINT IMPACT ASSESSMENT OF CTA’S SUPPORT TO NARI 9.5 PROPAC Activities Since 2005, the CTA has lent its support to 11 event and skill-building activities: Two regional briefings on (i) land issues in Central Africa and (ii) the vision of the future for African farming and the new role of producers’ organisations Two surveys on (i) the consistency of regional policies in Central Africa and (ii) the Economic Partnership Agreements (EPAs) between the European Union and the ACP countries, the analysis of public policies and budgets in Central Africa. A workshop on the take-up and dissemination of the conclusions drawn in the survey of regional agricultural policies in Central Africa Three training programmes on (i) the EPAs, the analysis of public policies and budgets, (ii) information management, (iii) lobbying, negotiating, advocacy by leaders and officials in farmers’ organisations in Central Africa The OHADA Uniform Act: regional farmers’ organisations consultations on cooperative society laws in Central Africa Training and consultations with Central African farmers’ organisations on the EPAs and the WTO agreements Participation in the CTA/ACP partners joint impact planning and assessment workshop Part C: Key findings from logical framework analysis 91 Outputs/ deliverables Workshop reports, surveys and training; communication materials Final workshop declarations; PROPAC declarations on the EPA negotiations in Central Africa Raising staff-awareness about (i) the priorities of the Common Agricultural Policy of the Central African Economic and Monetary Community (CEMAC), (ii) the challenges of the EPAs and the public budget operating systems and (iii) the reformed business laws applicable to the African continent Staff training on the use of (i) ICT techniques and (ii) advocacy/lobbying techniques Declarations by farmers’ organisations approved by the sub-region’s representatives and policymakers Outcomes for the organisation Improving and raising the profile and strategic positioning of PROPAC and its beneficiaries Regular involvement in farmers’ organisations for the purpose of preparing country-specific policies Partnerships concluded with Central African sub-regional integration institutions and their specialist agencies New integrated networks Better land management take-up Reflecting the risks of EPAs in the definition of strategies and advocacy themes Harmonisation of advocacy themes and activities with the new business law provisions 92 JOINT IMPACT ASSESSMENT OF CTA’S SUPPORT TO NARI Outcomes for the organisation (Continued) Adoption of new strategies and working methods More effective dissemination of the conclusions drawn and continuing awareness-raising for those involved in the development process Creation of a unit for information management and communication officer skills improvement More effective dissemination of information on the PROPAC site (reports, articles…) Creation of an exchange platform to allow its beneficiaries opportunities to promote themselves by publishing information on the PROPAC website Regular production of activity reports and other communication materials Impact PROPAC has improved is strategic positioning at several levels: At national level, it is increasingly being consulted for the purpose of developing national development policies. In Cameroon, the organisation has built up an extensive network of relationships with the ministries responsible for the primary sector, research institutes and other Cameroonian NGOs At sub-regional level, it has confirmed its importance in Central Africa via its 10 affiliated NFOs, which regularly seek the organisation’s support in solving conflicts or adding weight to their contacts with the political and administrative authorities in their countries. It has become one of the strategic partners of CEMAC, the Economic Community of Central African States (ECCAS) and their specialist organisations. Part C: Key findings from logical framework analysis 93 Impact (Continued) At regional level, PROPAC interacts with the African Union. It helped set up the Pan African Farmer’s Organisation. Its network is participating in a collaborative platform with the NEPAD. At international level, it is interacting with the CTA, the IFAD, the FAO and other international agencies: more than 20 partners identified at all levels. Its communication strategy has been improved thanks to the creation of an information management and training unit for social networks. It has also succeeded in terms of a better take-up of the CEMAC Common Agricultural Policy. 94 JOINT IMPACT ASSESSMENT OF CTA’S SUPPORT TO NARI RECOMMENDATIONS FOR FUTURE CTA SUPPORT/ ENGAGEMENT 10 CTA-CAFAN JOINT IMPACT ASSESSMENT 95 “ There is need for AFRACA to backstop its member institutions so that AFRACA can offer them technical support where it is lacking.” AFRACA NARI There is a need to develop an M&E framework within AFRACA so as to effectively monitor interventions and evaluate their impacts at various levels and at different stages and timelines. This would not only enhance experiential learning and adoption of key experiences, but also guide the future orientation of strategies envisioned by AFRACA. There is need for AFRACA to backstop its member institutions so that AFRACA can offer them technical support where it is lacking. With most of the innovations being new concepts, AFRACA can be a one-stop-shop for all the information and technical capacity enhancement that these institutions may require so as to effectively innovate, share and adopt these techniques, products and services. In general, CTA’s contribution to AFRACA has been enormous. There is need for future engagements to be well structured so that the specific output, outcome and impact of this support are clearly captured, for the smooth and practical evaluation of the outcomes and impacts at a later date. CTA has enhanced NARI’s capacity throughout their 10-year partnership in a number of areas, including ASTI systems and Web 2.0 for development. The capacities of other individuals and organisations have also been enhanced as a result of CTA’s support. These institutions include universities, government departments, NGOs, research institutions and women’s groups. In the future, CTA and NARI should target their investments directly at improving the living standards of the rural farming communities throughout PNG. CTA should also ensure that future engagement devises smart strategies to engage with leadership and guarantee top management buy-in to ensure the new concept or technology is adapted and applied. The following are some recommendations for improving the relationship between CTA and NARI whilst also promoting a collaborative effort to achieve the desired development outcomes in PNG’s agricultural sector: to improve the performance of CTA-funded initiatives, CTA should maintain a regular presence throughout and in particular during project initiation, the first phase of project implementation, and project closure; longterm investments are better value for CTA than those which focus on the short-term. For example, a five-day learning event will achieve very little on the ground. Finally, CTA’s investments should be flexible, allowing for adaption in accordance with contextual issues, challenges and opportunities. If possible, CTA’s interventions should be context-specific because what works in Africa will not necessarily work in the Pacific. 96 JOINT IMPACT ASSESSMENT OF CTA’S SUPPORT TO NARI SPC Over the last 10 years, CTA has provided around €1.3 million to SPC for a range of information knowledge management (IKM) capacity development activities. The mode of activities included training, workshops, planning sessions and exchange visits to build the capacity of IKM, communications and media in the region – all niche areas for CTA. The key areas of improvement for CTA and SPC are “process-centric”. Programmatic approaches work better for both parties, rather than short, one-off type activities. After 10 years of collaboration, both institutions have signalled intent to establish a more mature relationship through an MOU. Wherever possible, designs should be a joint effort and arranged programmatically so that both parties and countries have a clearer picture of key results going forward. SPC has undertaken an organisational reform process in the last 2 years and LRD – the site for CTA collaboration – has itself revamped its structure and articulated a new strategic framework (2013-17). The recommendations provided under each of the five core organisational capabilities fits well with SPC’s commitment, which focuses on learning for excellence and service delivery. Recommendations for the future are that: SPC strengthens its governance structures to manage CTA or other partner funding, including building evaluation systems at the outset in partnership documents; CTA and SPC share the findings of this study with management and internal M&E teams and discuss overall findings; CTA and SPC publish this report and share it widely with countries; SPC and CTA continue to discuss how the model or elements of the model can be embedded into SPC systems for programming and evaluation; CTA organises a peer learning event to not only share findings of the report but also how the elements of the 5Cs model might be taken forward. Part C: Recommendations for future CTA support/engagement 97 “ The CTA’s support has resulted in a huge improvement in PROPAC’s institutional, technical, operational and geostrategic capacities.” WOUGNET The CTA-supported WOUGNET Enhanced Access to Agricultural Information (EAAI) project triggered outcomes and impacts at individual and group levels. The most significant changes were recorded in the human capital, social capital, wealth impact, political capital and environmental domains, in that order of predominance. The intervention strengthened the capacity of individuals to use ICTs, notably radio, to access agricultural information and services; ensured acquisition of farming skills; and contributed to the adoption of modern farming practices and technologies.. However, limited training on how to send SMS and use of digital cameras, coupled with low literacy levels, have curtailed some women’s use of these functions even when they have access to the phones. Future interventions should allow adequate time for training a critical number of people who can then gradually train other members in the groups. WOUGNET and its partners should also consider engaging in functional adult literacy if the proportion of women with no formal schooling (39%) are to acquire skills to use mobile phones to send, retrieve, read and comprehend information in SMS. Lack of remunerative markets was also cited as a key problem for farmers who planted citrus after listening to the radio programmes and training during the project. WOUGNET could consider interventions to link these farmers with other actors in the citrus fruit value chain and also impart skills in juice and wine making to add value to their fruits and to access better markets. 98 JOINT IMPACT ASSESSMENT OF CTA’S SUPPORT TO NARI Areas identified for further in-depth study include: changes in skills and knowledge of Web 2.0 alumni and their associated outcomes; changes triggered by farmers’ application of ICTs to access agricultural information following the EAAI project; and, changes triggered as a result of the ICT and governance project in northern Uganda. The WOUGNET secretariat should institutionalise staff retreats to ensure dialogue and staff appreciation of the organisational policies, progress and resource implications of their implementation. The secretariat also needs to improve areas where the capability assessment returned low scores. Emphasis should be on strengthening internal decisionmaking structures (building confidence in all staff to genuinely contribute to ideas in a timely manner), utilisation of learning by individual staff for the benefit of the organisation, and human resource management. PROPAC The CTA’s support has resulted in a huge improvement in PROPAC’s institutional, technical, operational and geostrategic capacities. However, PROPAC has to contend with limited financial and material resources. Affecting the organisation and its beneficiaries, the shortfall hampers its performance, while showing the need for capacity-building. The CTA could therefore focus on the long-term programme approach in its future partnership strategies, with a five-year period being regarded as a minimum. Under this heading, the CTA may seek co-funding from other international agencies for the purpose of applying the external experts’ recommendations based on long-term (five-year) programmes. The CTA procedure-related constraints are leading to delays in the process for releasing funds and implementing activities and projects enjoying CTA support. This highlights the need for the procedures to be streamlined. PROPAC and its beneficiaries are enjoying much wider exposure, while their positioning has improved vastly. Nonetheless, a detailed investigation has to be made of the 10 affiliated NFOs in future in order to ensure the five-year capacity-building programme for the ACP partners and its beneficiaries is more effectively designed. ICT training for PROPAC has helped to upgrade its communication systems. However, more resources are required to develop and maintain the NFO communication units. Long-term capacitybuilding projects are needed for this purpose. Part C: Recommendations for future CTA support/engagement 99 ANNEXES 11 100 CTA-CAFAN JOINT IMPACT ASSESSMENT 11.1 List of Participants in the CcIPA impact study (Phase II) ACP-EU Technical Centre for Agricultural and Rural Cooperation (CTA) WOUGNET 1 Dr. Ibrahim Khadar khadar@cta.int 2 Mr Moses Owiny 2 Ms Tarikua Woldetsadick Woldetsadick@cta.int AFRACA 1 Mr John Amimo 2 Mr Saleh Gashiu PROPAC 1 Mrs Dorothy Okello EU-based M & E Specialists Mr. Dick van Blitterswijk MDF, Netherlands Mr. Jan Brouwers CDI-WUR, Netherlands Mrs. Eunike Spierings ECDPM, Netherlands 1 Mr Celestin Nga 2 Mr Gabriel Aime Elat Elat NARI 1 Dr Birte Nass-Komolong 2 Mrs Maxie Makombo Dominic SPC 1 Mr Vili Caniogo 2 Mr Emil Adams Part C: Annexes 101 11.2 Sources References 1. Download from Internet website; http:// www.cta.int/en/; on the 11 November, 2014. 2. National Agricultural Research Institute; NARI Strategic Programme Implementation Plan 2012-2020, NARI Headquarters, Lae, Morobe Province, PNG. 3. CTA Handout, March 2014; Capacitycentred Impact Pathways Analysis (CcIPA) model: design, testing and use through collaborative case studies; CTA, Wageningen, The Netherlands. • ASTI Learning and Workshop Report, 2006 • ASTI Learning and Workshop Report, 2007 • ASTI Learning and Workshop Report, 2009 • NARI Strategic Programme Implementation Plan, 2012-2020 102 JOINT IMPACT ASSESSMENT OF CTA’S SUPPORT TO NARI 11.3 Sample survey Survey opposite: Joint Impact Assessment of CTA Support to NARI: Sample of Survey Form Capabilities Domains 1. Leadership & management ACTING: Capability to act and commit Inspiring, action oriented leadership 2. Committed Organisational ownership, commitment related to culture 3. Decision making structures/ mechanisms Information/ intelligence based, collective, participatory 4. External environment Updating mechanism; staff assigned; what happens to the information 5. Internal environment LEARNING: Capability to adapt and self-renew Checklist/Keywords Open Questions NARI’s ability to manage projects, liaise with staff and stakeholders? What has been your history of implementation of projects? How are staff engaged in the organisation/ network? What are new initiatives? What are the internal mechanisms for regular decision making process? What is new? How is learning linked to external actors or what arrangements are in place? How do you analyse information? Incentives to apply learning, self assessment, culture What is the internal learning structure or what arrangements are in place? Staff culture, HR resources, finances Self assessment, procedures for feedback in place? Internal marketing, communication of issues. Own constraints to implement policies, being open about failure. Openness in internal communication? Feedback from stakeholders, regularity, system, what happens to information; relevance of products/ projects/services What information needs does your M&E cater for? Is a manual, calendar of mgt/ program meetings available? 6. Monitoring Performance appraisal and Evaluation Involvement in baseline assessment Learning from unplanned results and events Answers Are staff appraisals linked to M&E? Are baseline data available? What is new? Part C: Annexes 103 Joint Impact Assessment of CTA Support to NARI: Sample of Survey Form (Continued) Capabilities Domains Checklist/Keywords Open Questions 7. Implementation of activities/ projects Competence of staff; quality of services and products Does the staff have the technical knowledge, skills, and confidence to deliver products and services? Resources Answers Does the staff have the resources to do their work? What is new? PRODUCING: Capability to deliver products and services 8. Project/ activity initiation or phasing out/ termination 9. Quality assurance mechanism Frequency of projects; What is the stage in the source/initiator; number, life cycle of the size, type; reasons for organisation/network? phasing out Strategies (inventory/ policy/plans), process Resources, processes, steps Operational performance appraisal Are decisions documented? What is new? What documented procedures are available? Does NARI have regular performance appraisal of staff? What is new? 10. Level of engagement and influence of organisation in networks, alliances and collaborative efforts RELATING: Capability to relate 11. Mandate 12. Visibility/ credibility - Reputation or status/ visibilitycredibility/ image Networks/alliances/ partnerships; change; reasons In what number and type of networks is the organisation/network involved, and how did these evolve? Acquiring a mandate as an organisation, political legitimacy, social credibility, image of the organisation Incentives, measures taken, why/what triggered this reputation, tools, opportunities 104 JOINT IMPACT ASSESSMENT OF CTA’S SUPPORT TO NARI In what number and type of networks is the organisation/network involved, and how did these evolve? Is the organisation/ partner able to be at the same level playing field? Is the organisation/ partner able to be at the same level playing field? Did the organisation acquire a stronger mandate within the partnerships it is involved in? What is the communication strategy or profile? What is new in the way the organisation/ network presents itself? What documented procedures are available? Joint Impact Assessment of CTA Support to NARI: Sample of Survey Form (Continued) Capabilities COHERING: Capability to achieve coherence Domains Checklist/Keywords 13. Governance structure Internal processes, comprehensiveness, perception of staff of these rules Open Questions Answers To what extent is management steering needed for staff to operate? What is new? 14. Vision and strategy Link between Are vision and strategy operational/ clear for staff? administrative rules and vision, mission, strategy; What is new? making the strategy happen 15. People Human resources qualification, growth, staff turnover etc…, respect for diversity How do staff describe the culture of the organisation/network? What is new? Part C: Annexes 105 11.4 Average Scores and Composite Scores 12.2 NARI’s Self Assessment Scores Against the 5 Core Capabilities: Average Scores and Composite Scores ACTING Questions Average Scores 1 2 3 4 5 6 7 8 9 10 11 Does NARI have an action oriented leadership? 4 4 5 5 4 5 4 3 4 3 4 3 NARI’s culture is committed. 4 3 5 5 4 5 4 4 4 1 5 3 Decisions are made based on information 4 3 5 5 4 4 4 4 4 1 4 4 Decision making happens participatory. 3 3 5 5 3 4 4 3 3 1 4 5 Capability to Act 15 Composite score 3.75 NARI takes external environment changes well into account. 4 4 5 5 3 4 4 3 3 1 4 3 NARI has adequate incentives for a learning culture. 3 4 5 1 3 4 4 2 3 1 2 3 Is NARI’s HR policy well implemented? 3 3 5 5 3 4 4 4 3 2 3 3 Are NARI’s internal communication channels open? 3 2 5 5 4 4 4 2 4 3 4 4 Feedback from stakeholders is taken into account in daily processes. 3 4 5 1 4 3 4 2 3 3 3 3 The M&E process contributes to improving projects. 3 3 4 5 3 4 3 2 3 5 3 3 LEARNING 19 composite score 3.17 Staff is competent for the job it needs to do. 4 4 5 5 3 4 4 3 4 3 4 3 The infrastructure is adequate to perform the job 3 4 4 5 4 5 4 4 3 2 4 4 The type of projects that are undertaken fit in the overall strategy 4 5 5 5 4 5 4 4 3 5 4 4 The quality of NARI’s work is well assured. 4 4 5 5 3 4 4 3 4 5 4 4 106 JOINT IMPACT ASSESSMENT OF CTA’S SUPPORT TO NARI 12 13 14 15 16 3 4 3 4 4 4 5 4 4 4 5 3 2 4 3 4 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 Averages 2.5 3 2 3 3 3 4 5 4 3 4 4 3 5 5 5 3 5 4 3 3 4 3 5 3 3 0 5 3.5 3 3 4 3 5 1 5 2 5 4 3 3 2.5 4 3 4 3 5 4 4 3.5 4 4 5 4 5 1 5 4 5 4 4 3 3 3 2 3 1 3 1 4 3 3 4 3 5 2 1 5 5 2 5 3 3 5 3 4 1 4 2 5 3 3 4 4 3 4 4 5 4 5 5 5 2 5 4 4 3 2 3 3 2.5 3 3 3 3 3 4 5 3 3 5 4 3 5 1 5 3 5 3 4 2 4 3 3 3 1 2 3 2 1 4 2 3 3 4 5 3 5 1 5 3 5 3 4 1 4 3 3 3 3 3 3 3 1 4 4 3 4 4 4 2 5 1 5 2 5 3 4 2 2 4 4 2 3 3 4 2 3 2 5 3 3 4 4 2 5 1 5 4 5 3 4 2 5 3 4 4.5 2 3 4 2 3 4 5 2.7 3 4 4 3 5 0 5 3 5 3 5 3 5 4 4 3.5 3 2 5 3 3 5 5 3 4 4 4 3 5 1 5 2 5 4 5 3 4 4 3 3 3 2 3 1 3 4 3 3 4 3 4 2 5 5 2 1 5 3 4 5 5 3 4 4 3 3 5 3 5 5 5 3.5 4 4 5 3 5 1 5 2 5 4 5 3 5 4 4 4.5 4 3 5 3 3 5 5 3 0 4 2 5 5 5 2 5 4 4 Part C: Annexes 107 12.2 NARI’s Self Assessment Scores Against the 5 Core Capabilities: Average Scores and Composite Scores (Continued) Questions PRODUCING: Capability to deliver … Average Scores 1 2 3 4 5 6 7 8 9 10 11 15 Composite score 3.75 NARI is involved in all relevant networks. 4 4 5 1 3 4 4 5 4 3 4 3 The networks add value to NARI. 4 3 5 5 4 5 4 4 3 3 4 4 4 5 5 5 4 5 4 4 4 5 4 4 ? Does NARI have a strong reputation among its stakeholders? RELATING: Capability to relate COHERING: Capability to Cohere 12 Compsite score 4 The internal management is supportive for staff to operate 4 4 5 5 3 5 4 3 4 3 5 4 Staff are well aware of NARI’s strategy. 4 5 5 5 2.5 4 4 4 3 5 4 4 Is NARI’s HR policy competency 3 based? 4 5 5 3 3 3 3 4 5 3 Composite score 108 JOINT IMPACT ASSESSMENT OF CTA’S SUPPORT TO NARI 11 3.67 4 12 13 14 15 16 3 4 4 5 3 4 5 3 4 3 4 4 5 4 0 3 5 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 Averages 4.5 4 3 5 4 3 4 5 3 4 4 4 3 5 0 4 2 5 4 4 2.5 4 2 5 5 3 5 5 4 4 5 4 5 5 0 5 2 5 4 3 5 3.5 5 3 5 5 3 5 5 4 5 5 5 4 5 0 5 2 5 4 3 3 5 4 4 2 3 2 3 4 4 4 4 3 5 4 5 0 5 2 5 4 5 4 3 5 3 2 3 3 5 5 5 4 5 5 4 5 4 5 5 0 2 5 4 3 5 2 4 4 1 2 3 1 1 3 3 3 4 3 5 3 5 1 0 2 5 3 Part C: Annexes 109 11.5 Details of CTA Funded Projects/ Activities within NARI Project # Project Category Project Name 1 CASE STUDY Support CTA & NARI to build indigenous knowledge systems 2 CASE STUDY Capacity Building for NARI and Stakeholders in ASTI systems 3 CASE STUDY Contract betwenn CTA and NARI of PNG - Regional Training of Trainer Workshop for ACP Experts on Agricutural science, Technology and Innovation systems Port Moresby, PNG 4 WORKSHOP /TRAINING Sub-Regional Training of Trainers’ Workshop on Agricultural, Science, Technology and Innovation (ASTI) systems 5 WORKSHOP /TRAINING National Agricultural Information Systems (NAIS) Workshop -PNG 6 WORKSHOP /TRAINING CTA/UNRE/NARI/USP 2009 Regional S&T Workshop. Science for Impact: Building a Critical Mass of Scientists/ Lecturers Identifying the Value Proposition 7 CASE STUDY REPORT National Case Study Contract, PNG - International Conference on “innovations in Extnesion and Advisory Services: Linking Knowledge to Policy and Action for Food and Livelihoods” 15-18 Nov 2011, Nairobi, Kenya 8 WORKSHOPS /TRAINING Web2 Development and Social Media Learning Opportunities, PNG Host Institutions (4 events) - NARI 110 JOINT IMPACT ASSESSMENT OF CTA’S SUPPORT TO NARI Time Total Cost (EUR) The Activity Deliverables /Outputs 2004 11,500 Undertake case study within the ASTI system and identify key actors Rice was selected as the subsector, and case study on rice completed (report available) 2005 13,109 1. Conduct training 2. Undertake field visits 3. Compile case study Case Study completed on Farmer Experimentaiton & Innovation (report available) 2006 29,912 Workshop or training in TOT - ASTI systems Using Banana System as a Case Study to understand the ASTI systems in PNG: case study has been compiled (draft report is available) 2007 53,678 Conduct workshop or training Workshop was conducted, a draft workshop report is available) Nov-Dec 2009 1,015 1. Conduct refresher /complete training for NAIS operators 2. Carry out SWOT analysis 3. Develop Strategic plan 4. Identify opportunities for collaboratiove initiatives between NAIS partners 5. Enhance commitment of & network by NAIS operators Workshop on National Agricultural Information Systems (NAIS) conducted, total of 13 participants attended, workshop report available Nov-09 139,642 June 2011-Nov 2011 7,000 Conduct case study March 2014 -Sept 2014 11,740 Training in Web2 Conduct workshop for regional researchers / scientists across Pacific region Report on the study of agriculture extension and advisory services in PNG compiled, report is available Part C: Annexes 111 Notes 112 JOINT IMPACT ASSESSMENT OF CTA’S SUPPORT TO NARI Notes 113 114 JOINT IMPACT ASSESSMENT OF CTA’S SUPPORT TO NARI The Technical Centre for Agricultural and Rural Cooperation (CTA) is a joint international institution of the African, Caribbean and Pacific (ACP) Group of States and the European Union (EU). Its mission is to advance food and nutritional security, increase prosperity and encourage sound natural resource management in ACP countries. It provides access to information and knowledge, facilitates policy dialogue and strengthens the capacity of agricultural and rural development institutions and communities. CTA operates under the framework of the Cotonou Agreement and is funded by the EU. For more information on CTA visit, www.cta.int Contact us CTA PO Box 380 6700AJ Wageningen The Netherlands Tel: +31 317 467100 Fax: +31 317 460067 Email: cta@cta.int www.facebook.com/CTApage @CTAflash