Learning and Sharing: P4P data scrutinised by independent experts
Learning and sharing which P4P approaches do and do not work is a key feature of P4P. In order to effectively advocate the adoption of lessons learned through the implementation of the pilot, WFP must show clear evidence of how the pilot has enhanced market opportunities and incomes for smallholder farmers.
In August, WFP’s research partner, the African Economic Research Consortium, hosted a three day workshop in Lusaka, Zambia that brought together a range of experts from universities, NGOs and FAO to study and comment on the results from 3 of the 21 P4P pilot countries. Around 35 practitioners and academics with expertise in rural development and agricultural economics, undertook a close examination of P4P’s impact assessment strategy, including an in-depth discussion of the preliminary results of implementation emerging from Mali, Tanzania and Ethiopia.
A spirit of open and frank exchange characterised the workshop. Participants commended WFP and AERC for the transparent and non-defensive approach to the discussions, which included very careful scrutiny and even some criticism of the approach to data collection followed in some of the pilot countries. Some of the areas of concern pointed out included:
- small sample sizes in some of the pilot countries that were not deemed to be large enough to allow for adequate analysis;
- the eligibility criteria adopted to identify participating farmers’ organizations in some cases violated the principles of the quasi-experimental design and;
- the need to carefully review the extent of comparability between P4P and corresponding control groups. Often, the scientific requirements of randomly assigning organizations to P4P and non-P4P groups were either overtaken by country offices’ implementation priorities, or there were insufficient eligible groups available to work with P4P, let alone randomly assigning them between treatment and control groups. These factors led to the selection criteria being relaxed in some instances, and as such, severely limiting the prospects of undertaking an impact assessment in those countries.
Programme Advisor Clare Mbizule noted: “At a certain point, we inevitably had to make a compromise between the desire to adopt a scientific approach to selecting organizations and the practical challenge of moving implementation ahead”. Implementation constraints at this early stage included instances where there were few eligible farmers’ organizations in the country, to limits to the operational reach of supply side partners in some countries, and within WFP, insufficient M&E human resource and expertise, prior to the technical support partnership with AERC.
While recognising these limitations, the overwhelming consensus of the workshop participants was an acknowledgment of the wealth of value represented by the P4P M&E effort to date. Participants offered valuable recommendations to strengthen the validity of the P4P impact assessment approach, urging WFP and AERC to: (i) Refine and focus the data collection strategy on two to four impact assessment countries to ensure credible attribution of results to P4P implementation; and (ii) to focus on before and after studies with strong qualitative inputs for non-impact assessment countries.
Speaking on behalf of the AERC university network, Dr. Micah Masuku of the University of Swaziland appreciated WFP’s openness and transparency, remarking: “My colleagues and I are happy that you invited us to look at your data and bring in our opinions. I can see exciting opportunities for other studies to emerge from the P4P data, and throughout the AERC network we will have many graduate students coming on board to engage with these data.”
Anne Swindale of USAID commended WFP and AERC for their courage in inviting such an accomplished crowd of external experts, to what she described as being in effect “a mid-term evaluation of P4P’s monitoring and evaluation exercise”. She also emphasised the importance of documenting what P4P is doing and how this is being done in detail.
WFP welcomed the workshop recommendations which closely mirror those offered by the P4P Technical Review Panel and the 2011 Mid-Term evaluation conducted by the Overseas Development Institute. With the support of partners such as the Alliance for a Green Revolution in Africa (AGRA), the AERC network and Cornell University, WFP will consider options to expand the sample size in the impact assessment countries. P4P will also adopt rigorous econometric approaches to modelling the impact that P4P has had on smallholder productivity, marketing capacity of farmers’ organizations, promoting pro-smallholder market structures and ensuring a supportive enabling environment.
In addition, AERC and WFP will launch a data portal in the coming months that will house both the qualitative and quantitative data drawn from the implementation of P4P, and this will enable further interaction with and scrutiny of the P4P outcomes by partners and AERC network members.